WO2024109129A1

WO2024109129A1 - Electronic device and data processing method

Info

Publication number: WO2024109129A1
Application number: PCT/CN2023/108893
Authority: WO
Inventors: 陆世明; 片兆峰; 韩征; 汪静娴; 田鹏升
Original assignee: 海信视像科技股份有限公司
Priority date: 2022-11-23
Filing date: 2023-07-24
Publication date: 2024-05-30

Abstract

The embodiments of the present application relate to an electronic device and a data processing method. The electronic device comprises: a communicator, which is configured to receive a device list acquisition request sent by a HomeKit client after having established a connection with the HomeKit client on the basis of a HomeKit protocol; and a processor, which is configured to determine the device information of an electronic device and the device information of each target device on the basis of the device list acquisition request, and generate target information on the basis of the device information of the electronic device, the device information of each target device and at least one device placeholder, each device placeholder being used for indicating one unknown device for access, and the device information corresponding to each device placeholder being null. The communicator is further configured to send to the HomeKit client the target information, so that the HomeKit client displays a device list on the basis of the target information.

Description

Electronic device and data processing method

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the priority of the application number 202211477456.8 filed with the China Patent Office on November 23, 2022, and the application number 202211658353.1 filed with the China Patent Office on December 22, 2022, and the application number 202211714328.0 filed with the China Patent Office on December 29, 2022, the entire contents of which are incorporated by reference into this application.

Technical Field

The embodiments of the present application relate to the field of terminal technology, and in particular, to an electronic device and a data processing method for the electronic device.

Background technique

With the development of smart home technology, users can control all HomeKit-supported devices in their home, including lights, locks, thermostats, smart plugs and other devices, through the HomeKit app on the device.

Electronic devices equipped with the HomeKit protocol can also be discovered and controlled by the HomeKit app. For example, after electronic device A is discovered and connected by the HomeKit app, the user can control electronic device A through the HomeKit app. The HomeKit app can not only control electronic device A, but also control electronic device B connected to the electronic device, and even control electronic device C connected to electronic device B. In this way, users can control multiple electronic devices through the HomeKit app, which provides great convenience for users.

Among them, according to the HomeKit protocol, HomeKit app is the control end, and the device list of controlled devices displayed on HomeKit app is obtained when HomeKit app is initialized after connecting with electronic device A. However, once the device list is initialized, other devices that are later connected to electronic device A cannot be added to the device list. In other words, after HomeKit app connects with electronic device A and initializes, it is impossible to dynamically update the device list displayed on HomeKit app.

Summary of the invention

In a first aspect, according to an embodiment of the present application, an electronic device includes: a communicator, configured to: after establishing a connection with a HomeKit client based on a HomeKit protocol, receive a device list acquisition request sent by the HomeKit client, the device list acquisition request being used to indicate acquisition of device information of each target device connected to the electronic device, the device information including an identification of the device; a processor, configured to: based on the device list acquisition request, determine the device information of the electronic device and the device information of each target device, the device information including an identification of the corresponding device; generate target information based on the device information of the electronic device, the device information of each target device and at least one device placeholder, each device placeholder being used to indicate an unknown device to be connected, and the device information corresponding to each device placeholder being empty; the communicator is further configured to: send the target information to the HomeKit client so that the HomeKit client displays a device list based on the target information, the device list including the identification of the electronic device and the identification of each target device.

In a second aspect, according to an embodiment of the present application, a data processing method is applied to an electronic device, comprising: after establishing a connection with a HomeKit client based on the HomeKit protocol, receiving a device list acquisition request sent by the HomeKit client, the device list acquisition request is used to indicate the acquisition of device information of each target device connected to the electronic device, the device information including an identification of the device; based on the device list acquisition request, determining the device information of the electronic device and the device information of each target device, the device information including an identification of the corresponding device; generating target information based on the device information of the electronic device, the device information of each target device and at least one device placeholder, each device placeholder being used to indicate an unknown device to be connected, and the device information corresponding to each device placeholder is empty; sending the target information to the HomeKit client so that the HomeKit client displays a device list based on the target information, the device list including the identification of the electronic device and the identification of each target device.

In a third aspect, according to an embodiment of the present application, a computer-readable non-volatile storage medium is provided, comprising: a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, the device list shown in the second aspect is implemented. Table acquisition method.

In a fourth aspect, according to an embodiment of the present application, a computer program product is provided, including: when the computer program product is run on a computer, the computer implements the device list acquisition method shown in the second aspect.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG1 is one of the operation scenarios between a control device and an electronic device according to some embodiments;

FIG2 is a block diagram of a hardware configuration of an electronic device 200 according to some embodiments;

FIG3 is a second operation scenario between a control device and an electronic device according to some embodiments;

FIG4 is a third operation scenario between a control device and an electronic device according to some embodiments;

FIG5 is a flowchart of a method for obtaining a device list according to some embodiments;

FIG6 is a second flow chart of a method for acquiring a device list according to some embodiments;

FIG7 is a third flow chart of a method for acquiring a device list according to some embodiments;

FIG8 is a block diagram of a structure between a mobile phone and a television according to some embodiments;

FIG9 is a fourth flow chart of a method for acquiring a device list according to some embodiments;

FIG10 is a fifth flow chart of a method for obtaining a device list according to some embodiments;

FIG11 is an operation scenario between a display device and a control device according to some embodiments;

FIG12 is a block diagram of a hardware configuration of a control device 001 according to some embodiments;

FIG13 is a schematic diagram of a screen projection connection interface according to some embodiments;

FIG14 is a second schematic diagram of a screen projection connection interface according to some embodiments;

FIG15 is a flowchart of a screen projection processing method according to some embodiments;

FIG16 is a schematic diagram of a screen projection data flow according to some embodiments;

FIG17 is a schematic diagram of a screen projection data flow according to some embodiments;

FIG18 is a second flow chart of a screen projection processing method according to some embodiments;

FIG19 is a third flowchart of a screen projection processing method according to some embodiments;

FIG20 is a fourth flowchart of a screen projection processing method according to some embodiments;

FIG21 is a fifth flowchart of a screen projection processing method according to some embodiments;

FIG22 is a sixth flowchart of a screen projection processing method according to some embodiments;

FIG23 is a schematic diagram of software configuration in a display device according to some embodiments;

FIG24 is a schematic diagram showing an icon control interface of an application in a display device according to some embodiments;

FIG25 is a system framework diagram for performing screen projection processing according to some embodiments;

FIG26 is a schematic diagram of a flow chart of a screen projection processing method according to some embodiments;

FIG27 is a schematic diagram of an interface of a display when the current playback state of the display device is in a target mode according to some embodiments;

FIG28 is a schematic diagram of an interface of a display when a display switch of the display is in an on state and the display displays non-audio data according to some embodiments;

FIG29 is a flowchart of another screen projection processing method according to some embodiments;

FIG30 is a schematic diagram of an interface when a display displays prompt information according to some embodiments;

FIG31 is a schematic diagram of an interface of a display when a display device enters an input page according to some embodiments;

FIG32 is a schematic diagram of a screen projection process of a display device according to some embodiments;

Figure 33 is a schematic diagram of the overall process of a screen projection processing method according to some embodiments.

Detailed ways

In order to make the purpose and implementation method of the present application clearer, the exemplary implementation method of the present application will be clearly and completely described below in conjunction with the drawings in the exemplary embodiments of the present application. Obviously, the described exemplary embodiments are only part of the embodiments of the present application, rather than all the embodiments.

It should be noted that the brief description of terms in this application is only for the convenience of understanding the embodiments described below, and is not intended to limit the embodiments of this application. Unless otherwise specified, these terms should be used in their ordinary and common Normal understanding of the meaning.

The electronic device provided in the embodiments of the present application can have various implementation forms, for example, it can be a television, a smart TV, a laser projection device, a display (monitor), an electronic whiteboard (electronic bulletin board), an electronic desktop (electronic table), a mobile phone, a tablet computer, a laptop computer, a PDA, a car-mounted electronic device, etc.

Figure 1 is a schematic diagram of the operation scenario between the control device and the electronic device according to the embodiment, wherein the HomeKit app is installed in the control device 100, and the electronic device 200 supports the HomeKit protocol. Therefore, after the control device 100 establishes a connection with the electronic device 200 through the HomeKit app, the user can control the electronic device 200 that supports the HomeKit protocol and other devices directly or indirectly connected to the electronic device 200 through the HomeKit app installed in the control device 100.

In some embodiments, the control device 100 may be an iOS device with an iOS operating system installed with the HomeKit app, such as a handheld device such as an iPhone or iPad, or a device with other operating systems installed with the HomeKit app. The control device 100 may establish a connection with the electronic device 200 by means of device discovery after connecting to the same wired or wireless network. The user may input user commands through voice input, control panel input, gesture input, etc. to control the device 100.

As shown in FIG2 , the electronic device 200 includes at least one of a tuner 210 , a communicator 220 , a detector 230 , an external device interface 240 , a processor 250 , a display 260 , an audio output interface 270 , a user interface 280 , an external memory, and a power supply.

In some embodiments, the processor includes a processor, a video processor, an audio processor, a graphics processor, a RAM, a ROM, and a first interface to an nth interface for input/output.

The display 260 includes a display screen component for presenting images, and a driving component for driving image display, a component for receiving image signals output from a processor, and a component for displaying video content, image content, and a menu control interface and a user control UI interface.

The display 260 may be a liquid crystal display, an OLED display, or a projection display, and may also be a projection device and a projection screen.

The communicator 220 is a component for communicating with an external device or server according to various communication protocol types. For example, the communicator may include at least one of a Wifi module, a Bluetooth module, a wired Ethernet module, and other network communication protocol chips or near field communication protocol chips, and an infrared receiver. The electronic device 200 can establish transmission and reception of control signals and data signals with the external control device 100 or the server 400 through the communicator 220.

The user interface 280 can be used to receive control signals from the control device 100 (such as an infrared remote controller, etc.). It can also be used to directly receive user input operation instructions and convert the operation instructions into instructions that the electronic device 200 can recognize and respond to. In this case, it can be called a user input interface.

The detector 230 is used to collect signals from the external environment or the external interaction. For example, the detector 230 includes a light receiver, a sensor for collecting the intensity of ambient light; or, the detector 230 includes an image collector, such as a camera, which can be used to collect external environment scenes, user attributes or user interaction gestures; or, the detector 230 includes a sound collector, such as a microphone, etc., for receiving external sounds.

The external device interface 240 may include, but is not limited to, any one or more of the following interfaces: a high-definition multimedia interface (HDMI), an analog or digital high-definition component input interface (component), a composite video input interface (CVBS), a USB input interface (USB), an RGB port, etc. It may also be a composite input/output interface formed by the above multiple interfaces.

The tuner-demodulator 210 receives broadcast television signals via wired or wireless reception, and demodulates audio and video signals, such as EPG data signals, from a plurality of wireless or wired broadcast television signals.

In some embodiments, the processor 250 and the tuner-demodulator 210 may be located in different separate devices, that is, the tuner-demodulator 210 may also be located in an external device of the main device where the processor 250 is located, such as an external set-top box.

The processor 250 controls the operation of the electronic device and responds to the user's operation through various software control programs stored in the memory (internal memory or external memory). The processor 250 controls the overall operation of the electronic device 200. For example, in response to receiving a user command for selecting a UI object to be displayed on the display 260, the processor 250 can perform an operation related to the object selected by the user command.

In some embodiments, the processor includes a central processing unit (CPU), a video processor, an audio processor, a graphics processing unit (GPU), and at least one of a random access memory (RAM), a read-only memory (ROM), a first interface to an nth interface for input/output, a communication bus (Bus), etc.

Among them, RAM, also called main memory, is an internal memory that directly exchanges data with the processor. It can be read and written at any time (except when refreshed), and the speed is very fast. It is usually used as a temporary data storage medium for operating systems or other running programs. The biggest difference between it and ROM is the volatility of data, that is, once the power is cut off, the stored data will be lost. RAM is used in computers and digital systems to temporarily store programs, data and intermediate results. ROM works in a non-destructive read-out mode and can only read but not write information. Once the information is written, it is fixed, and even if the power is cut off, the information will not be lost, so it is also called fixed memory.

The user may input a user command through a graphical user interface (GUI) displayed on the display 260, and the user input interface receives the user input command through the graphical user interface (GUI). Alternatively, the user may input a user command through a specific sound or gesture, and the user input interface recognizes the sound or gesture through a sensor to receive the user input command.

With the development of smart home technology, users can control all HomeKit-supported devices in their home, including lights, locks, thermostats, smart plugs and other devices, through the HomeKit app on iOS devices.

Electronic devices equipped with the HomeKit protocol can also be discovered and controlled by the HomeKit app on iOS devices. For example, after electronic device A is discovered and connected by the HomeKit app, the user can control electronic device A through the HomeKit app on the iOS device. The HomeKit app can not only control electronic device A, but also control electronic device B connected to the electronic device, and even control electronic device C connected to electronic device B. In this way, users can control multiple electronic devices through the HomeKit app on iOS devices, which provides great convenience for users.

Among them, according to the HomeKit protocol, the iOS device side is the control side, and the device list of controlled devices displayed on the iOS device is obtained when the iOS device is initialized after connecting with electronic device A. However, once the device list is initialized, other devices that are later directly or indirectly connected to electronic device A cannot be added to the device list. In other words, after the iOS device is connected to electronic device A and initialized, it is impossible to dynamically update the device list displayed on the iOS device.

In order to solve the above technical problems, according to an embodiment of the present application, a method for obtaining a device list is provided, specifically a method for dynamically updating a device list by presetting device placeholders based on the HomeKit protocol, specifically, after establishing a connection with a HomeKit client based on the HomeKit protocol, receiving a device list acquisition request sent by the HomeKit client, the device list acquisition request being used to indicate obtaining device information of each target device connected to the electronic device, the device information including a device identifier; based on the device list acquisition request, determining the device information of the electronic device and the device information of each target device, the device information including a corresponding device identifier; generating target information based on the device information of the electronic device, the device information of each target device and at least one device placeholder, each device placeholder being used to indicate an unknown device to be connected, and the device information corresponding to each device placeholder being empty; sending the target information to the HomeKit client so that the HomeKit client displays a device list based on the target information, the device list including the identifier of the electronic device and the identifier of each target device. In this way, by adding device placeholders in the target information, since each device placeholder is used to indicate an unknown device to be connected, and the device information corresponding to each device placeholder is empty, not only will the user not be misled because the device list generated by the HomeKit client based on the target information includes the identifiers of some devices that have not been connected, but a position is reserved in the device list for the unknown devices to be connected. Therefore, when other devices subsequently establish a connection with the electronic device, a device placeholder in the target information can be updated with the device information of the other device to update the target information, and then the HomeKit client updates the device list based on the updated target information to dynamically update the device list displayed on the HomeKit client.

According to an embodiment of the present application, an electronic device and a device list acquisition method are provided, wherein the electronic device can implement the device list acquisition method according to the embodiment of the present application or the functional module or functional entity in the electronic device can implement the device list acquisition method according to the embodiment of the present application. The electronic device includes: a communicator and a processor, which correspond to the communicator 220 and the processor 250 in Figure 2 above, respectively. The HomeKit client can be the control device 100 in the figure, or it can be the HomeKit app installed in the control device 100, which can be determined according to actual conditions.

In some embodiments, in order to enable the HomeKit client to dynamically update the device list of devices controlled by the electronic device, after establishing a connection with the HomeKit client based on the HomeKit protocol, the communicator receives a device list acquisition request sent by the HomeKit client to instruct the acquisition of device information (the device information includes the identification of the device) of each target device connected to the electronic device; the processor determines the device information of the electronic device and the device information of each target device based on the device list acquisition request, and the device information includes the identification of the corresponding device; then generates target information based on the device information of the electronic device, the device information of each target device and at least one device placeholder; the communicator sends the target information to the HomeKit client, so that the HomeKit client displays the device list based on the target information, and the device list includes the identification of the electronic device and the identification of each target device. In an embodiment of the present application, by adding device placeholders to the target information, since each device placeholder is used to indicate an unknown device to be connected, and the device information corresponding to each device placeholder is empty, not only will the user not be misled because the device list generated by the HomeKit client based on the target information includes the identification of some unconnected devices, but also a position is reserved in the device list for the unknown devices to be connected. Therefore, when other devices subsequently establish a connection with the electronic device, a device placeholder in the target information can be updated with the device information of the other device to update the target information, and then the HomeKit client updates the device list based on the updated target information to dynamically update the device list displayed on the HomeKit client.

Each device placeholder is used to indicate an unknown device to be connected. It can be understood that each device placeholder corresponds to a position reserved for the device information of a device to be connected in the target information. In this way, a position is reserved for the device to be connected in the device list generated by the HomeKit client based on the target information. The device to be connected can be any device that is directly or indirectly connected to the electronic device.

Among them, the HomeKit client device is used to discover and control electronic devices and various other devices connected to the electronic devices (directly or indirectly). After the HomeKit client and the electronic device are connected to the same wired or wireless network, the HomeKit client performs device discovery. After the HomeKit client discovers the electronic device, it establishes a connection with the electronic device. The specific process can refer to the existing related technology and will not be repeated here.

Among them, after the HomeKit client establishes a connection with the electronic device, it sends a device list acquisition request to the electronic device, and then receives the target information sent by the electronic device. The HomeKit client generates a device list based on the target information, specifically, based on the identifier of the electronic device in the target information and the identifier of each target device. Since the device information corresponding to the device placeholder is empty, the device list does not include the identifier of the unknown device to be connected indicated by each device placeholder. In other words, the HomeKit client displays the identifier of the device whose device information in the target information is not empty in the device list, ignores the device placeholder whose device information in the target information is empty, and does not display it in the device list.

Among them, the direct connection between the electronic device and the device A means that the electronic device and the device A are directly connected by wire or wirelessly, without being connected through other devices. The indirect connection between the electronic device and the device B means that the electronic device is connected to the device B through the device C, which can also be understood as the electronic device is directly connected to the device C, and the device C is directly connected to the device B. Among them, the device A and the device C can be called the inherent connection of the electronic device, and the device B can be called the extended connection of the electronic device.

In some embodiments, when the electronic device detects that a connection (direct connection or indirect connection) is established with another device, the identifier of the other device can be dynamically updated to the device list so that the HomeKit client can control the newly connected other device.

In some embodiments, after sending the target information to the HomeKit client, the processor detects that the first device has established a connection with the electronic device; then updates a device placeholder in the target information to the device information of the first device to update the target information; the communicator sends the updated target information to the HomeKit client so that the HomeKit client updates the device list based on the updated target information, and the updated device list includes the identifier of the electronic device, the identifiers of each target device, and the identifier of the first device.

In the embodiment of the present application, when the first device establishes a connection with the electronic device, a device placeholder in the target information is replaced with the device information of the first device to obtain updated target information, and then the updated target information is sent to the HomeKit client, so that the HomeKit client adds the identifier of the first device to the device list based on the information of the first device in the updated target information, thereby realizing dynamic update when a new device is connected to the electronic device. The device list displayed by the HomeKit client does not require the electronic device to disconnect from the HomeKit client and then reconnect. Instead, the device list displayed by the HomeKit client is dynamically updated while the electronic device and the HomeKit client remain connected.

In an embodiment of the present application, when an electronic device detects that it is disconnected from a target device (directly or indirectly), the identifier of the target device can be dynamically deleted from the device list to prevent the user from controlling the disconnected target device through the HomeKit client.

In some embodiments, after sending the target information to the HomeKit client, the processor detects that a second device is disconnected from the electronic device, the second device being any one of the target devices; then the device information of the second device in the target information is updated to a placeholder for the device to update the target information; then the communicator sends the updated target information to the HomeKit client, so that the HomeKit client updates the device list based on the updated target information, the updated device list including the identifier of the electronic device and the identifiers of each device among the target devices except the second device.

In some embodiments, after sending the target information to the HomeKit client, the processor detects that a second device is disconnected from the electronic device, the second device being any one of the target devices; then the device information of the second device is deleted from the target information to update the target information; then the communicator sends the updated target information to the HomeKit client, so that the HomeKit client updates the device list based on the updated target information, the updated device list including the identifier of the electronic device and the identifiers of each device among the target devices except the second device.

In an embodiment of the present application, when the second device is disconnected from the electronic device, the device information of the second device in the target information can be replaced by a device placeholder, or the device information of the second device can be deleted from the target information to obtain updated target information, and then the updated target information is sent to the HomeKit client, so that the HomeKit client deletes the identifier of the second device from the device list based on the updated target information that does not include the device information of the second device, thereby dynamically updating the device list displayed by the HomeKit client after an already connected device connects to the electronic device. In this process, there is no need for the electronic device to disconnect from the HomeKit client and then reconnect, but the dynamic update of the device list displayed by the HomeKit client is achieved while the electronic device remains connected to the HomeKit client.

In the embodiment of the present application, the number of devices directly connected to the electronic device in each target device is not limited, nor is the number of devices indirectly connected to the electronic device. The specific number can be determined based on actual conditions and is not limited here.

In some embodiments, each target device includes: at least one third device, each third device is directly connected to the electronic device, that is, each target device is a third device, each target device is a device directly connected to the electronic device.

In some embodiments, as shown in Figure 3, the HomeKit client is a mobile phone 301 with the HomeKit app installed, the electronic device is a TV 201 that supports the HomeKit protocol, and the at least one third device is three third devices, each of which is an HDMI device 302 directly connected to the TV via an HDMI interface.

In some embodiments, each target device includes: at least one third device and at least one fourth device; wherein each third device is directly connected to the electronic device, and each fourth device is connected to the electronic device through a corresponding third device. That is, a target device is a third device or a fourth device. The at least one fourth device can be connected to the electronic device through the same third device or through different third devices, which can be determined according to actual conditions.

In some embodiments, as shown in FIG. 4 , the HomeKit client is a mobile phone 301 with the HomeKit app installed, the electronic device is a TV 302 that supports the HomeKit protocol, the at least one third device is two third devices, the at least one fourth device is two fourth devices, each third device is an HDMI device 303 directly connected to the TV via an HDMI interface, and each fourth device is an HDMI device 304 directly connected to a third device via an HDMI interface.

In the embodiments of the present application, the connection method between the target device directly connected to the electronic device and the electronic device is not limited, nor is the connection method between the target device indirectly connected to the electronic device and the electronic device limited. It can be a wired interface connection, such as a High Definition Multimedia Interface (HDMI) connection, a USB interface connection, or a USB interface connection. It can also be a wireless connection, such as a Bluetooth connection, a WiFi connection, etc. The specific connection can be determined according to actual conditions.

In some embodiments, the electronic device is connected to each of the third devices via an HDMI interface; when each of the target devices includes the at least one fourth device, each of the third devices is connected to the corresponding third device via the HDMI interface.

Among them, the HDMI interface is a digital video/audio interface technology and a dedicated digital interface suitable for image transmission. It can transmit audio and image signals at the same time, with a maximum data transmission speed of 2.25GB/s, and does not require digital/analog or analog/digital conversion before signal transmission. HDMI can be used with High-bandwidth Digital Content Protection (HDCP) to prevent unauthorized copying of copyrighted audio and video content. The extra space provided by HDMI can be used in future upgraded audio and video formats.

In the embodiment of the present application, when the electronic device and each target device are devices with display screens, and when the transmitted data is audio and video data, connecting through an HDMI interface can improve the data transmission efficiency, protect the security of the data during the transmission process, and support audio and video formats that will be upgraded in the future.

In some embodiments, the device information also includes the type of the corresponding device; the device list also includes the type of the electronic device and the type of each target device.

In some embodiments, when the electronic device is connected to and initialized with the HomeKit client, the device list includes the identification and type of the electronic device, and the identification and type of each target device; when it is detected that the electronic device has established a connection with the first device, the device list is updated to include the identification and type of the electronic device, the identification and type of each target device, and the identification and type of the first device; when it is detected that the electronic device is disconnected from the second device, the device list is updated to include the identification and type of the electronic device, and the identification and type of each target device except the second device.

In an embodiment of the present application, the device information includes the type of device, and the device identifier and type may be displayed in the device list. This allows the user to better understand the identifier and type of each device based on the device list displayed by the HomeKit client, thereby better controlling each device according to user needs.

In some embodiments, the sum of the number of each target device and the number of the at least one device placeholder is equal to a first preset value; or, the number of the at least one device placeholder is equal to a second preset value.

Among them, the first preset value and the second preset value can be determined according to actual conditions.

In some embodiments, the first preset value is greater than the second preset value.

In an embodiment of the present application, the total number of devices controlled by the HomeKit client based on the connection with the electronic device (i.e., the total number of electronic devices, each target device, and at least one device placeholder, i.e., the sum of the first preset value and 1) can be set according to the properties of the electronic device, the total number of devices controlled by the HomeKit client input by the user, and other information. The number of at least one device placeholder (i.e., the second preset value) can also be set according to the properties of the electronic device, the total number of devices controlled by the HomeKit client input by the user, and other information. In this way, the number of device placeholders to be added in the target information can be set according to actual needs, which can improve the management efficiency of the device list.

In order to explain the present solution in more detail, the following will be explained in an exemplary manner in conjunction with Figures 5 to 10. It can be understood that the steps involved in Figures 5 to 10 may include more steps, or fewer steps in actual implementation, and the order of these steps may also be different, so as to be able to implement the device list acquisition method provided in the embodiment of the present application. The execution subject of the device list acquisition method can be an electronic device, or it can be a functional module or functional entity in the electronic device that can implement the device list acquisition method. Moreover, the specific description of the device list acquisition method according to the embodiment of the present application can refer to the relevant description of the above-mentioned electronic device, and the same or similar technical effects can be achieved, which will not be repeated here.

FIG5 is a flowchart of the steps of implementing a method for acquiring a device list according to one or more embodiments of the present application. The method for acquiring a device list may include:

S501. After establishing a connection with a HomeKit client based on the HomeKit protocol, receiving a device list acquisition request sent by the HomeKit client.

The device list acquisition request is used to instruct to acquire device information of each target device connected to the electronic device, and the device information includes an identification of the device.

S502: Determine the device information of the electronic device and the device information of each target device based on the device list acquisition request. Backup information.

The device information includes the identifier of the corresponding device.

S503: Generate target information based on the device information of the electronic device, the device information of each target device, and at least one device placeholder.

Each device placeholder is used to indicate an unknown device to be connected, and the device information corresponding to each device placeholder is empty.

S504: Send the target information to the HomeKit client, so that the HomeKit client displays a device list based on the target information.

The device list includes the identifier of the electronic device and the identifiers of each target device.

In an embodiment of the present application, by adding device placeholders to the target information, since each device placeholder is used to indicate an unknown device to be connected, and the device information corresponding to each device placeholder is empty, not only will the user not be misled because the device list generated by the HomeKit client based on the target information includes the identification of some unconnected devices, but also a position is reserved in the device list for the unknown devices to be connected. Therefore, when other devices subsequently establish a connection with the electronic device, a device placeholder in the target information can be updated with the device information of the other device to update the target information, and then the HomeKit client updates the device list based on the updated target information to dynamically update the device list displayed on the HomeKit client.

In some embodiments, in combination with FIG. 5 , as shown in FIG. 6 , after the above S504 , the device list acquisition method according to the embodiment of the present application may further include the following S505 to S507 .

S505: Detect that the first device establishes a connection with the electronic device.

S506: Update a device placeholder in the target information to the device information of the first device to update the target information.

S507: Send the updated target information to the HomeKit client, so that the HomeKit client updates the device list based on the updated target information.

The updated device list includes the identifier of the electronic device, the identifiers of each target device, and the identifier of the first device.

In some embodiments, in combination with FIG. 5 , as shown in FIG. 7 , after the above S504 , the device list acquisition method according to the embodiment of the present application may further include the following S508 to S510 .

S508: Detect that the second device is disconnected from the electronic device.

The second device is any one of the target devices.

S509: Update the device information of the second device in the target information to a placeholder for the device, so as to update the target information.

S510. Send the updated target information to the HomeKit client, so that the HomeKit client updates the device list based on the updated target information.

The updated device list includes the identifier of the electronic device and the identifiers of each device among the target devices except the second device.

In some embodiments, the respective target devices include: at least one third device; or, the respective target devices include: at least one third device and at least one fourth device; wherein each third device is directly connected to the electronic device, and each fourth device is connected to the electronic device through a corresponding third device.

The iOS device is a mobile phone, the electronic device is a TV, and the electronic device is connected to other devices via an HDMI interface. For example, according to the HomeKit protocol, the device list displayed on the mobile phone is initialized after the mobile phone is connected to the TV. Once the list is initialized, new devices cannot be added. Although the number of HDMI interfaces on the TV is fixed, it can be solidified after the mobile phone is connected to the TV. However, the device connected to the HDMI interface of the external device can also be extended to connect other HDMI devices, and this part of the data is changing. Under the premise that the mobile phone and the TV have established a link, this part of the data cannot be dynamically added to the mobile phone, so the mobile phone cannot dynamically display the devices connected later.

In some embodiments, as shown in FIG8 , a mobile phone is used as a control terminal, on which a HomeKit app is installed, and a television is used as a controlled terminal supporting the HomeKit protocol, which is composed of a HomeKit proxy module 61, a data processing module 62, and a device management module 63. Among them, the HomeKit Proxy module 61 is responsible for establishing a network connection and data communication with the mobile phone. Receive control instructions and data (including device list acquisition request) sent by the mobile phone; Data processing module 62: responsible for parsing control instructions and data, receiving the device list acquisition request forwarded by HomeKit Proxy, forwarding the parsed device list acquisition request to the device management module 63, and receiving the device information of the TV returned by the device management module, as well as the device information of each device directly or indirectly connected to the TV. Then the data processing module is also used to add at least one device placeholder based on the device information of the TV and the device information of each device, reorganize the data, generate target information, and return the target information to the HomeKit Proxy module 61, so that the HomeKit Proxy module 61 returns it to the mobile phone; The device management module 63 is used to obtain the device information of the TV, the device information of each device connected to the TV, and determine the number of devices connected to the TV, etc., and report the obtained information to the data processing module 62.

In the following description, the inherent device list information includes the device information of the TV and the device information of the device directly connected to the TV; the extended device list information includes the device information of the device indirectly connected to the TV. When the TV is directly connected or indirectly connected to the target device through the HDMI interface, the device management module 63 is used to identify the connection information corresponding to each HDMI interface of the TV, determine the third device connected to the inherent HDMI interface of the TV, and obtain the device information of the corresponding third device, and then determine the inherent device list information based on the device information of the TV and the device information of each third device; then, through the HDMI interface, obtain the fourth device connected to the third device through the HDMI interface from the third device directly connected to the TV, and obtain the device information of the fourth device, and then obtain the extended device list based on the device information of each fourth device.

In some embodiments, as shown in FIG. 9 , the device list acquisition method according to an embodiment of the present application includes the following S701 to S710 .

S701. After the mobile phone and TV are connected to the same network, use the HomeKit app to discover the device and establish a connection with the TV.

S702: After the connection between the mobile phone and the TV is established, the mobile phone sends a device list acquisition request to the TV.

S703. After receiving the device list acquisition request, the HomeKit Proxy module of the TV sends the device list acquisition request to the data processing module.

S704: The data processing module of the television sends a device list acquisition request to the device management module.

After receiving the request to obtain the device list, the TV's HomeKit Proxy module calls the TV's data processing module and requests the device management module for the device list.

S705. The device management module obtains inherent device list information and extended device list information.

It can be understood that the device management module obtains the number of local HDMI interfaces of the TV and the device information of the TV, as well as the device information of the device connected to the HDMI interface, to obtain the inherent device list information, and at the same time obtains whether the device connected to the HDMI interface is connected to an extension device, and obtains the device information of the extension device connected to the device connected to the HDMI interface, to obtain the extension device list information.

S706: The device management module returns the inherent device list information and the extended device list information to the data processing module.

S707. The data processing module adds N device placeholders based on the inherent device list information and the extended device list information, and sets the device name to be empty to generate target information.

S708. The data processing module returns the target information to the HomeKit Proxy module.

S709, HomeKit Proxy module sends the target information to the mobile phone.

S710. After receiving the target information, the mobile phone displays a device list based on the target information.

It can be understood that the mobile phone displays the identification of each inherent device or the identification and type of each inherent device based on the device information of each device in the inherent device list information according to the target information; and displays the identification of each extended device or the identification and type of each extended device based on the device information of each device in the extended device list information, and ignores each device placeholder. It can also be understood that the mobile phone displays the device information of the device whose device information in the target information is not empty, and ignores the device placeholder whose device information is empty. In this way, the initialization of the device list is completed, and a position is reserved in the initialized device list for the device to be connected.

In some embodiments, after the initialization of the device list is completed, during the use of the TV, if the user newly connects an expansion device, it is necessary to notify the HomeKit app on the mobile phone to update the list, as described in Figure 10. The specific process includes the following S801 to S808.

S801. The HomeKit Proxy module sends a monitoring command to the data processing module to monitor whether there is a device connected to the TV or a device connected to the TV.

S802: The data processing module forwards the monitoring instruction to the device management module.

S803: The device management module responds to the monitoring instruction, and when monitoring that a new expansion device is connected to the device connected to the television, obtains the device information of the newly connected expansion device.

S804: The device management module returns the device information of the newly connected expansion device to the data processing module.

The device management module may update the device information of the newly connected expansion device into the expansion device list information and then send it to the data processing module, or may send the device information of the newly connected expansion device to the data processing module separately.

S805: The data processing module receives the device information of the newly connected extended device, and replaces a device placeholder in the target information with the newly connected extended device information to update the target information.

S806. The data processing module sends the updated target information to the HomeKit Proxy module.

S807, HomeKit Proxy module sends the updated target information to the mobile phone.

S808. After receiving the updated target information, the mobile phone displays an updated device list based on the updated target information.

It can be understood that after the device management module recognizes that a new expansion device is connected, it returns the device information of the newly connected expansion device to the data processing module. The data processing module updates the device information of the newly connected expansion device to a device placeholder in the target information, and sets the name, type and other information of the newly connected expansion device. In this way, the device placeholder becomes a real device; the updated target information is sent to the mobile phone through the HomeKit Proxy module. After the mobile phone receives the updated target information, it refreshes the device placeholder in the list. Since the placeholder has real information such as name and type, it can be refreshed and displayed on the device list displayed on the mobile phone.

In addition to the above-mentioned embodiments for solving the technical problem that after the iOS device and the electronic device A are connected and initialized, the device list displayed on the iOS device cannot be dynamically updated, the present application also provides examples of some embodiments for further solving other technical problems when the electronic device is used as a display device, which are as follows:

The pseudo standby state refers to a state where the operating system of the display device is in operation, but the screen of the display device is powered off and in a dark state, and the output port stops outputting. The power-on state refers to a state where the operating system of the display device is in operation, and the screen of the display device is powered on and in a bright state.

The Android system is constantly iterating and updating. In order to protect user privacy, the permissions of the entire device application have been tightened to prevent the abuse of permissions, which will cause the performance of the entire device to be depleted and user complaints. The current security measures are just merged by Google, and many applications have not yet been adapted, resulting in the failure to wake up the TV when pushed in the fake standby state, which is a poor user experience.

In order to solve the above technical problems, the electronic device described in the embodiment of the present application, when used as a display device, receives a screen projection request sent by a smart device; in response to the screen projection request, when the display device is in a pseudo standby state, controls the display device to switch from the pseudo standby state to the power-on state based on the AirPlay2 application; executes the screen projection operation corresponding to the screen projection request based on the AirPlay2 application, so that when the screen projection request sent by the smart device is received in the pseudo standby state, the display device can be first woken up based on the AirPlay2 application, and then the corresponding screen projection operation is executed, so that screen projection can be realized in the pseudo standby state, thereby reducing the user's operating cost from the entire link, improving the efficiency of screen projection processing, and enhancing the user's operating experience.

FIG. 11 is a schematic diagram of an operation scenario between a display device and a control device according to an embodiment, wherein the control device includes As shown in FIG11 , the user can operate the display device 200 through the smart device 300 or the control device 001 . The smart device 300 and the control device 001 can both be regarded as the control device 100 .

In some embodiments, the control device 001 may be a remote controller, and the communication between the remote controller and the display device includes infrared protocol communication or Bluetooth protocol communication, and other short-range communication methods, and the display device 200 is controlled wirelessly or wired. The user can input user commands through buttons on the remote controller, voice input, control panel input, etc. to control the display device 200.

In some embodiments, a smart device 300 (such as a mobile terminal, a tablet computer, a computer, a laptop computer, etc.) may also be used to control the display device 200. For example, the display device 200 is controlled using an application running on the smart device.

In some embodiments, the display device may not use the above-mentioned smart device or control device to receive instructions, but may receive user control through touch or gestures.

In some embodiments, the display device 200 can also be controlled in a manner other than the control device 001 and the smart device 300. For example, the user's voice command control can be directly received through a module for obtaining voice commands configured inside the display device 200, or the user's voice command control can be received through a voice control device set outside the display device 200.

In some embodiments, the display device 200 also communicates data with the server 400. The display device 200 may be allowed to communicate via a local area network (LAN), a wireless local area network (WLAN), and other networks. The server 400 may provide the display device 200 with various content and interactions.

FIG12 is a block diagram of a configuration of a control device 001 according to an exemplary embodiment. As shown in FIG12 , the control device 001 includes a processor 110, a communication interface 130, a user input/output interface 140, an external memory, and a power supply. The control device 001 can receive an input operation instruction from a user, and convert the operation instruction into an instruction that can be recognized and responded to by the display device 200, playing the role of an interactive intermediary between the user and the display device 200.

According to an embodiment of the present application, a display device and a screen projection processing method are provided, wherein the display device can implement the screen projection processing method according to the embodiment of the present application or the functional module or functional entity in the display device can implement the screen projection processing method according to the embodiment of the present application. The display device includes: a communicator and a processor, which correspond to the communicator 220 and the processor 250 in Figure 2 above, respectively. The smart device may be the smart device 300 in Figure 11, and the smart device may be a device that supports the AirPlay2 protocol and can perform screen mirroring (screen projection connection) to the display device, such as an iOS device such as an iPhone or iPad.

In some embodiments, the communicator receives a screen projection request sent by a smart device; the processor responds to the screen projection request and, when the display device is in a pseudo standby state, controls the display device to switch from a pseudo standby state to a power-on state based on the AirPlay2 application; and executes the screen projection operation corresponding to the screen projection request based on the AirPlay2 application.

In some embodiments, the display device is controlled to switch from a pseudo-standby state to a power-on state based on the AirPlay2 application. It can be understood that the standard interface for waking up the screen is called based on the AirPlay2 application to power on the screen of the display device, so that the screen changes from a dark screen state to a bright screen state, thereby waking up the screen.

In some embodiments, the screen projection request can be a screen projection connection request, and the corresponding screen projection operation is to establish a screen projection connection; the screen projection request can be a screen projection push request, and the corresponding screen projection operation is to play the projection content, and the projection content can be pictures, audio or video, etc.; the specific details can be determined based on actual conditions and are not limited here.

Among them, the AirPlay2 application is an application built based on the AirPlay2 protocol that can realize the AirPlay2 screen projection function between smart devices and display devices. The specific name of the AirPlay2 application is not limited in the embodiment of the present application and can be determined according to actual conditions.

In an embodiment of the present application, when a screen projection request is received from a smart device in a pseudo-standby state, the display device can be first woken up based on the AirPlay2 application, and then the corresponding screen projection operation can be performed, so that screen projection can be achieved in the pseudo-standby state, thereby reducing the user's operating cost in the entire link, improving the efficiency of screen projection processing, and enhancing the user's operating experience.

In some embodiments, the screen projection request is a screen projection connection request, and the screen projection operation is to establish a screen projection connection; the processor is configured to perform the screen projection operation corresponding to the screen projection request based on the AirPlay2 application, specifically: to establish a screen projection connection between the display device and the smart device based on the AirPlay2 application. It can be understood that the communicator receives the message sent by the smart device. a screen projection connection request; the processor responds to the screen projection connection request, and when the display device is in a pseudo-standby state, controls the display device to switch from the pseudo-standby state to the power-on state based on the AirPlay2 application; and establishes a screen projection connection between the display device and the smart device based on the AirPlay2 application.

In an embodiment of the present application, when a screen projection connection request is received from a smart device in a pseudo-standby state, the display device can be first woken up based on the AirPlay2 application, and then the corresponding screen projection connection operation can be performed, so that the screen projection connection can be successfully established in the pseudo-standby state, thereby reducing the user's operating cost in the entire link, improving the efficiency of establishing the screen projection connection, and enhancing the user's operating experience.

In some embodiments, the screen projection request is a screen projection push request, and the screen projection operation is to play the screen projection content; the processor is also configured to: before the communicator receives the screen projection request sent by the smart device, establish a screen projection connection between the display device and the smart device based on the AirPlay2 application; when the target condition is met, control the display device to switch from the power-on state to the pseudo-standby state; the processor is configured to perform the screen projection operation corresponding to the screen projection request based on the AirPlay2 application, specifically: play the screen projection content corresponding to the screen projection push request based on the AirPlay2 application. It can be understood that the processor establishes a screen projection connection between the display device and the smart device based on the AirPlay2 application; when the target condition is met, control the display device to switch from the power-on state to the pseudo-standby state; the communicator receives the screen projection connection request sent by the smart device; the processor responds to the screen projection connection request, and when the display device is in the pseudo-standby state, controls the display device to switch from the pseudo-standby state to the power-on state based on the AirPlay2 application; and plays the screen projection content corresponding to the screen projection push request based on the AirPlay2 application.

In some embodiments, when the display device is connected to a WiFi network and is powered on, pressing the power button will cause the device to enter a pseudo-standby state. When the display device is powered on and has no audio or video output (e.g., in a paused playback state), and has been left idle for a certain period of time (e.g., 1 hour, 24 hours, etc.) without any key operation, the device will also enter a pseudo-standby state.

When a user uses AirPlay2 to cast the screen to watch videos or audio and leaves the screen for too long or presses the power button, the Android TV will enter a pseudo-standby state. When the user comes back and wants to cast the screen to watch the program again, he or she can operate the smart device to send a casting push request to the display device. The display device will respond to the casting push request, wake up the screen, and play the corresponding video or audio.

In an embodiment of the present application, when a screen projection push request is received from a smart device in a pseudo-standby state, the display device can be first woken up based on the AirPlay2 application, and then the projection content corresponding to the screen projection push request can be played based on the AirPlay2 application, so that the corresponding projection content can be played in response to the screen projection push in the pseudo-standby state, thereby reducing the user's operating cost in the entire link, improving the efficiency of screen projection push processing, and enhancing the user's operating experience.

In some embodiments, the target condition includes any one of the following: the playback content of the display device does not change within a preset time period; a user input for entering a pseudo standby state is received.

In some embodiments, in the power-on state, the playback content of the display device does not change within a preset time period, that is, the display device does not play any content or the played content does not change within the preset time period (paused playback state), and the display device switches from the power-on state to the pseudo-standby state.

In some embodiments, the user input for entering the pseudo standby state may be a user clicking a power button or a button for entering the pseudo standby state, or the display device may display a prompt message indicating whether to enter the pseudo standby state, and the user may select an option to enter the pseudo standby state. When the user is in the power-on state, upon receiving the user input for entering the pseudo standby state, the display device switches from the power-on state to the pseudo standby state.

In some embodiments, the user can set the AirPlay2 application to have a wake-up function in the pseudo standby state, that is, a function to wake up the display device screen, so that the display device switches from the pseudo standby state to the power-on state. The processor is also configured to: before the communicator receives the screen projection request sent by the smart device, in response to the received target user input authorizing the AirPlay2 application to have the wake-up function, determine that the AirPlay2 application has the wake-up function; wherein the wake-up function is used to control the display device to switch from the pseudo standby state to the power-on state.

In some embodiments, the target user input can be the input of setting the AirPlay2 application to have a wake-up function in the interface corresponding to the application permission setting item under the system setting interface; the target user input can also be the input of selecting authorize the AirPlay2 application to have a wake-up function when the AirPlay2 application is turned on for the first time or for a preset number of times, and when an authorization prompt message is displayed and prompts whether to authorize the AirPlay2 application to have a wake-up function, the specific input can be determined based on actual conditions and is not limited here.

In an embodiment of the present application, the AirPlay2 application is set to have a wake-up function through the target user input, so that when the display device receives a screen projection request sent by a smart device in a pseudo-standby state, it can control the display device from the pseudo-standby state to the power-on state based on the AirPlay2 application, and then perform the corresponding screen projection operation, thereby reducing the user's operating cost in the entire link, improving the efficiency of screen projection push processing, and enhancing the user's operating experience.

In some embodiments, when the AirPlay2 application is started, the authorization prompt information is displayed to set the AirPlay2 application to have a wake-up function. The processor is further configured to: display the authorization prompt information, the authorization prompt information includes a first option and a second option, the first option is used to indicate that the AirPlay2 application is authorized to have a wake-up function, and the second option is used to indicate that the AirPlay2 application is not authorized to have a wake-up function; the target user input is an input for selecting the first option.

In some embodiments, when the user selects the second option in response to the authorization prompt information, the AirPlay2 application does not have a wake-up function. When the display device receives a screen projection request sent by a smart device in a pseudo standby state, the AirPlay2 application cannot control the display device to switch from the pseudo standby state to the power-on state, and the display device remains in the pseudo standby state, and is unable to perform the corresponding screen projection operation.

In the embodiment of the present application, whether the AirPlay2 application has the wake-up function is set by authorization prompt information, which is simple to operate. Moreover, by interacting with the user through the authorization prompt information, the user can set whether the AirPlay2 application has the wake-up function according to their own needs, which can improve the user experience.

In some embodiments, the AirPlay2 application can be prefabricated in the system directory, and then the AirPlay2 application has a wake-up function without user authorization. Specifically, the AirPlay2 application is installed in the system directory of the display device, and any application in the system directory has a wake-up function; wherein the wake-up function is used to control the display device to switch from the pseudo standby state to the power-on state.

In some embodiments, the system directory can be the system/app directory, the system_ext/priv_app directory, or other system directories. In this case, user authorization is not required, that is, the TV can be woken up through Airplay2 screen casting in pseudo-standby state to perform screen casting demonstration.

The smart device is an iPhone, and the display device is a TV. Screen mirroring operation is performed on the iPhone, and an interface as shown in Figure 13 is displayed. The interface includes the device identifiers of various devices that can be used for Airplay2 screen projection with the iPhone (which can be device names, device models, device numbers, etc.), including multiple identifiers such as "Living Room TV", "Bedroom TV", and "Smart TV". Among them, the user selects the "Smart TV" identifier corresponding to the TV, and a screen projection connection is established between the iPhone and the TV. The TV displays the interface shown in Figure 14, indicating that the screen projection connection between the iPhone and the TV is successfully established.

The display device includes the Airplay daemon (AirPlayDaemon), the Airplay system service (AirPlaySystemService), and the Airplay2 application (AirPlayApk). Among them, AirPlayDaemon is used to parse the screen projection request. AirPlaySystemService is used to switch to the AirPlay channel and pull up AirPlayApk according to the screen projection request parsed by AirPlayDaemon; AirPlaySystemService is also used to forward the messages between AirPlayDaemon and AirPlayApk to establish a screen projection connection and screen projection push. AirPlayDaemon corresponds to the iOS system language, AirPlayApk corresponds to the Android system language, and AirPlaySystemService is used to convert between the iOS system language and the Android system language. As shown in Figure 15, the business process of establishing a screen projection connection and completing the screen projection push in the display device includes the following steps.

S601. AirPlayDaemon receives a screen projection connection request sent by a smart device.

AirPlayDaemon parses the screen casting connection request to obtain the screen casting connection control instruction, and sends a socket message to AirPlaySystemService. The socket message carries the screen casting connection control instruction, which is used to notify AirPlaySystemService to pull up AirPlayApk, and then let AirPlayApk complete the screen casting connection initialization, realize the screen casting connection, and display the screen casting (Curtain) page. The screen casting page is used to indicate that the connection is successfully established. The screen casting page can be a pairing page on the smart device.

S602. AirPlayDaemon sends a socket message to AirPlaySystemService.

S603, AirPlaySystemService executes JNI to trigger Java callback AirPlayApk based on the socket message.

JNI (Java Native Interface) is a programming framework used for Java programs in a Java virtual machine to call local applications or libraries. Local applications generally refer to programs written in languages such as C, C++, or assembly and compiled into programs for native hardware and operating systems.

JNI triggers the Java callback AirPlayApk, that is, JNI calls the logic of starting the application to switch to the AirPlay channel and launch the AirPlay application.

S604, AirPlayApk broadcasts the first request message for requesting the initialization startup parameters of the screen projection connection.

S605. AirPlayApk broadcasts a second request message for requesting drawing parameters of the projection screen page.

The first request message and the second request message may be the same request message or two independent request messages.

S606, AirPlaySystemService broadcasts the projection connection initialization startup parameters and the drawing parameters of the projection screen page.

AirPlaySystemService can broadcast the projection connection initialization startup parameters and the projection screen page drawing parameters at the same time through one broadcast message, or it can broadcast the projection connection initialization startup parameters and the projection screen page drawing parameters separately through two broadcast messages.

After receiving the first request message and the second request message, AirPlaySystemService can forward the first request message and the second request message to AirPlayDaemon. AirPlayDaemon sends the projection connection initialization startup parameters and the drawing parameters of the projection screen page to AirPlaySystemService. AirPlaySystemService then sends the projection connection initialization startup parameters and the drawing parameters of the projection screen page to AirPlayApk.

S607. AirPlayApk initializes the screen projection based on the screen projection connection initialization startup parameters to complete the screen projection connection; AirPlayApk draws the screen projection page based on the drawing parameters of the screen projection page and displays the screen projection page.

When the TvInputService onCreate cycle function in AirPlayApk is started, the broadcast AirPlay.check.breakin (first request message) is sent to inform that the channel switching is successful and request the initialization startup parameters of the screen projection connection. When the InputService.Session.onCreateOverLay function is triggered, the broadcast Request.CurtainPage.TIS (second request message) is sent to request the drawing parameters of the projection screen page. After obtaining the drawing parameters of the projection screen page, the projection screen page is drawn and displayed.

S608, AirPlayDaemon receives the screen projection push request sent by the smart device.

AirPlayDaemon parses the screen casting push request. If the screen casting data corresponding to the screen casting push request is audio data, the screen casting push request is parsed to obtain the audio data and the screen casting push control instruction. AirPlayDaemon processes the audio data and sends the screen casting push control instruction to AirPlaySystemService; if the screen casting data corresponding to the screen casting push request is a video uniform resource locator (Uniform Resource Locator, URL), the screen casting push request is parsed to obtain the video URL and the screen casting push control instruction. AirPlayDaemon sends the video URL and the screen casting push control instruction to AirPlaySystemService, and AirPlaySystemService downloads the video data according to the video URL.

S609, AirPlaySystemService receives the screen projection push control instruction sent by AirPlayDaemon.

AirPlaySystemService receives the screen projection push control command of video or audio, and sends the broadcast AirPlay.from.breakin (screen projection push control command) to notify AirPlayApk to play.

S610, AirPlaySystemService broadcasts screen projection and pushes control instructions.

S611. AirPlayApk receives the screen projection push control instruction and broadcasts a third request message for requesting the screen projection playback startup parameters.

AirPlayApk receives the screen projection push control instruction, creates a session, and sends a broadcast AirPlay.reqbreakin.parameter (the third request message) to AirPlaySystemService to request startup parameters.

S612, AirPlaySystemService broadcasts screen casting and playback startup parameters.

AirPlaySystemService sends the startup parameters to AirPlayApk using the broadcast airplay.tis.action.

After receiving the third request message, AirPlaySystemService can forward the third request message to AirPlayDaemon. AirPlayDaemon sends the screen projection playback startup parameters to AirPlaySystemService, and AirPlaySystemService then sends the screen projection playback startup parameters to AirPlayApk.

S613, AirPlayApk performs screen casting based on the screen casting startup parameters.

AirPlay calls launchAirPlayAppTIS to inform the underlying screen casting that the playback has started and is playing.

Figure 16 shows the audio data push process based on AirPlay2 application. The sender device is a smart device, AirPlay Daemon and Luna are display devices, and Luna is a part of AirPlayApk. When the display device receives the audio push from the sender device, the audio cache data is directly sent to AirPlay Daemon. The (metadata) data is forwarded to Luna in the AirPlay App for upper-level UI drawing. The audio synchronization between the Sender device and the AirPlay Daemon node is performed through Clock sync to achieve simultaneous playback.

Metadata data, also known as intermediary data or relay data, is data about data. It mainly describes the properties of data and is used to support functions such as indicating storage location, historical data, resource search, and file records.

Figure 17 shows the video data push process based on AirPlay2 application. When the display device receives the video push from the sending device, it sends the video URL to AirPlay Daemon, which is then forwarded to the Luna engine in the AirPlay App. Luna completes the media resource download, video playback, and upper-layer UI drawing.

In order to explain the present solution in more detail, the following will be explained in an exemplary manner in conjunction with Figures 18 to 22. It can be understood that the steps involved in Figures 18 to 22 may include more steps, or fewer steps in actual implementation, and the order of these steps may also be different, so as to implement the screen projection processing method provided in the embodiment of the present application. The executor of the screen projection processing method can be a display device, or it can be a functional module or functional entity in the display device that can implement the screen projection processing method, which is not limited here. Moreover, the specific description of the screen projection processing method according to the embodiment of the present application can refer to the relevant description of the above-mentioned display device, and the same or similar technical effects can be achieved, which will not be repeated here.

FIG. 18 is a flowchart of the steps of implementing a screen projection processing method according to one or more embodiments of the present application. The screen projection processing method may include:

S901. Receive a screen projection request sent by a smart device.

S902. In response to the screen projection request, when the display device is in a pseudo-standby state, control the display device to switch from the pseudo-standby state to the power-on state based on the AirPlay2 application.

S903. Execute the screen projection operation corresponding to the screen projection request based on the AirPlay2 application.

In some embodiments, the screen projection request is a screen projection connection request, and the screen projection operation is to establish a screen projection connection; in combination with FIG. 18 , as shown in FIG. 19 , the above S903 can be specifically implemented through the following S903a.

S903a. Establish a screen projection connection between the display device and the smart device based on the AirPlay2 application.

In some embodiments, the screen projection request is a screen projection push request, and the screen projection operation is to play the projection content; in combination with Figure 18, as shown in Figure 20, before the above S901, the screen projection processing method according to the embodiment of the present application may also include the following S904 and S905, and the above S903 can be specifically implemented by the following S903b.

S904. Establish a screen projection connection between the display device and the smart device based on the AirPlay2 application.

S905: When the target condition is met, control the display device to switch from the power-on state to the pseudo-standby state.

S903b. Play the projection content corresponding to the projection push request based on the AirPlay2 application.

In some embodiments, in combination with FIG. 18 , as shown in FIG. 21 , before the above-mentioned S901 , the screen projection processing method according to the embodiment of the present application may further include the following S906 .

S906: In response to the received target user input authorizing the AirPlay2 application to have the wake-up function, determine that the AirPlay2 application has the wake-up function.

The wake-up function is used to control the display device to switch from a pseudo-standby state to a power-on state.

In some embodiments, in combination with Figure 21, as shown in Figure 22, before the above-mentioned S901, the screen projection processing method according to the embodiment of the present application may also include the following S907.

S907: Display authorization prompt information.

Among them, the authorization prompt information includes a first option and a second option, the first option is used to indicate that the AirPlay2 application is authorized to have a wake-up function, and the second option is used to indicate that the AirPlay2 application is not authorized to have a wake-up function; the target user input is the input for selecting the first option.

In some embodiments, the AirPlay2 application is installed in the system directory of the display device, and any application in the system directory has a wake-up function; wherein the wake-up function is used to control the display device to switch from a pseudo-standby state to a power-on state.

In addition to the above-mentioned embodiments for solving the technical problem that after the iOS device is connected to the electronic device A and initialized, the device list displayed on the iOS device cannot be dynamically updated, and the embodiments for realizing screen projection in the pseudo standby state, reducing the user's operating cost from the entire link, improving the screen projection processing efficiency, and enhancing the user's operating experience, the present application also provides examples of some embodiments that can further solve other technical problems when the electronic device is used as a display device, as follows:

In recent years, with the rapid development of the Internet and the popularization of artificial intelligence (AI) technology, display devices such as smart TVs have gradually become smart terminals in people's daily lives, and they are also the first choice for living room entertainment. Especially during family gatherings, the large-screen advantage of smart TVs will be highlighted, becoming the mainstream entertainment method for people.

At the same time, as user needs gradually increase, the requirements for large screens are also higher. For example, users hope to share real-time game images on a large screen; for another example, users hope to play their favorite videos on a large screen. In order to meet user needs, the screen projection function can solve this problem well. Current display devices can support multiple screen projection protocols at the same time, such as Miracast protocol, AirPlay (air playback) protocol, etc.

When playing a data stream on a display device through screen projection, if the user only wants to hear the sound of the data stream, the display device can be operated to enter target mode. In target mode, the display device only plays audio data, and the display switch of the display is in the off state. However, if the user casts the screen again to play other data streams, such as playing a video, the display device will still be in target mode, and the display switch of the display will still be in the off state, resulting in the user being unable to watch the content of the video, and the user experience is relatively poor.

In response to the above-mentioned problem, when the above-mentioned electronic device is used as a display device, an embodiment of the present application further proposes a screen projection processing method, wherein the processor of the display device first obtains the type of the currently received projection data stream and the current playback status of the display device, and then determines that the type of the currently received projection data stream is non-audio data, and determines that the current playback status of the display device is in the target mode, controls the display switch of the display to be in the on state, and controls the display to display the non-audio data, wherein, when the playback status is in the target mode, the display switch of the display is in the off state. In the above-mentioned technical scheme, the display device can automatically control the display switch of the display to be in the on state when determining that the type of the currently received projection data stream is non-audio data, and determining that the current playback status of the display device is in the target mode, and controls the display to display the non-audio data, thereby avoiding the situation where the display switch of the display is in the off state when the non-audio data is currently being played due to the target mode, and the user cannot view the content of the non-audio data, thereby ensuring that the user can normally view the projection content during the screen projection process, thereby improving the user experience.

FIG23 is a schematic diagram of software configuration in a display device according to one or more embodiments of the present application. As shown in FIG23 , the system is divided into three layers, from top to bottom, namely, the application layer (Applications layer) (referred to as the "application layer"), the application framework layer (Application Framework layer) (referred to as the "framework layer"), and the operating system layer.

In some embodiments, at least one application is running in the application layer, and these applications may be window programs, system settings programs, clock programs, etc. that come with the operating system; they may also be applications developed by third-party developers. In specific implementations, the applications in the application layer include but are not limited to the above examples. In this embodiment, the application layer may include TV Input (the input module of the TV, also known as the channel list module) and the AirPlay Android Application Package (AirPlay APK), where the AirPlay APK is mainly used to obtain the properties of the entire machine (display device), such as obtaining the volume of the entire machine, etc., which is not limited here.

In some embodiments, the framework layer mainly includes: switching channels (cutting Input, cutting TIS), AirPlay service and audio and video middleware. Among them, the AirPlay service can interact with the AirPlay process and AirPlay APK, and control the life cycle of AirPlay APK, and can interact with the switching channel and audio and video middleware through interface calls; the AirPlay service interacts with the AirPlay APK through broadcasting; the switching channel interacts with the TV Input through the Java Native Interface (JNI).

In some embodiments, the operating system layer mainly includes: the AirPlay process and the implementation of mirroring, video and audio function modules. The operating system here can be Linux. The AirPlay process is mainly a resident native process, including the AirPlay protocol stack, which can call the AirPlay specific playback function module (mirror, video or audio, etc.) to implement the corresponding function; the AirPlay process and the AirPlay service realize the interaction between the two through network communication. The network communication may be implemented based on a network communication protocol, such as a socket protocol or other protocols, which is not limited in this embodiment.

In some embodiments, when a user operates a mobile device to cast a screen, the AirPlay protocol stack can receive the casting information or casting request sent by the mobile device, and send the casting information or casting request to the AirPlay service. After receiving the casting information or casting request (including casting parameters), the AirPlay service can determine the type of the current casting data stream (such as video, audio or mirror, etc.), and can also determine whether it is the first casting. If it is the first casting, the interface in the Android TV input framework (TV Input Framework, TIF) is called to switch the channel of the TV Input, and switch the current channel of the display device to the channel corresponding to AirPlay; if it is not the first casting, the AirPlay service will obtain the type of the currently received casting data stream, and call the audio and video middleware interface to obtain the current playback status of the display device. If the type of the currently received casting data stream is non-audio data, and it is determined that the current playback status of the display device is in target mode, then AirPlay will call the audio and video middleware interface to turn off the switch corresponding to the Audio Only function to control the display switch of the display to be in the on state, at which time the screen of the display will light up, and the display can display non-audio data.

It should be noted that FIG. 23 illustrates the software configuration in the display device by taking the display device using the Android platform as an example, and is not intended to limit it.

Figure 24 is a schematic diagram of the icon control interface display of an application in a display device according to one or more embodiments of the present application. As shown in Figure 24, the application layer includes at least one application that can display a corresponding icon control in the display, such as: a live TV application icon control, a video on demand application icon control, a media center application icon control, an application center icon control, a game application icon control, etc. The live TV application can provide live TV through different signal sources. The video on demand application can provide videos from different storage sources. Unlike the live TV application, the video on demand provides video display from certain storage sources. The media center application can provide applications for playing various multimedia content. The application center can provide storage for various applications.

In some embodiments, the display device is a terminal device with a display function, such as a television, a mobile phone, a computer, a learning machine, etc. In the display device:

An output interface (display 260 and/or audio output interface 270 ) configured to output user interaction information;

A communicator 220, used for communicating with a server;

The processor 250 is configured to execute computer instructions in the memory to cause the device to perform:

Obtain the type of the currently received projection data stream and the current playback status of the display device;

Determine that the type of the currently received projection data stream is non-audio data, and determine that the current playback state of the display device is in target mode, control the display switch of the display to be in an on state, and control the display to display the non-audio data, wherein when the playback state is in the target mode, the display switch of the display is in a off state.

In some embodiments, the processor 250 is further configured to:

After determining that the type of the currently received projection data stream is non-audio data and determining that the current playback state of the display device is in the target mode, controlling the display to display prompt information, wherein the prompt information is used to prompt the user whether to turn on the display switch of the display;

When an operation of turning on a display switch of the display is received, the display switch of the display is controlled to be in an on state, and the display is controlled to display the non-audio data.

In some embodiments, the processor 250 is further configured to:

In response to the current screen projection request, determining channel information of the display device;

When it is determined according to the channel information that the channel of the display device is the channel corresponding to the airplay, and the current screen projection request is a non-mirror screen projection request, the type of the currently received screen projection data stream and the current playback status of the display device are obtained.

In some embodiments, the processor 250 is further configured to:

Call the audio and video middleware interface to obtain the switch status corresponding to the Audio Only function;

When the state is the on state, it is determined that the current playback state of the display device is in the target mode.

In some embodiments, the processor 250 is further configured to:

Call the audio and video middleware interface and turn off the switch corresponding to the Audio Only function to control the display switch of the display to be in the on state.

In some embodiments, the processor 250 is further configured to:

Determine that the current screen projection request is a first screen projection request, establish a screen projection connection between the display device and the sender of the current screen projection request, and switch the channel of the display device to the channel corresponding to the airplay;

Based on the screen projection connection and the channel corresponding to the airplay, the data stream sent by the sender is received and the data stream is played.

In some embodiments, the screen projection protocol between the display device and the sender of the current screen projection request is: AirPlay protocol.

To summarize, the embodiment of the present application executes the above-mentioned screen projection processing method on the display device. The processor of the display device first obtains the type of the currently received screen projection data stream and the current playback status of the display device, and then determines that the type of the currently received screen projection data stream is non-audio data, and determines that the current playback status of the display device is in the target mode, controls the display switch of the display to be in the on state, and controls the display to display the non-audio data, wherein, when the playback status is in the target mode, the display switch of the display is in the off state. In the above-mentioned technical scheme, the display device can automatically control the display switch of the display to be in the on state and control the display to display the non-audio data when determining that the type of the currently received screen projection data stream is non-audio data and determining that the current playback status of the display device is in the target mode, so as to avoid the situation where the display switch of the display is in the off state when the non-audio data is currently being played due to the target mode, and the user cannot view the content of the non-audio data, thereby ensuring that the user can normally view the projection content during the screen projection process and improving the user experience.

FIG25 is a system framework diagram for performing screen projection processing according to one or more embodiments of the present application. As shown in FIG25, the system may include an acquisition module 401, a determination module 402, and a control module 403. The system first acquires the type of the currently received screen projection data stream and the current playback state of the display device through the acquisition module 401, and then the determination module 402 determines that the type of the currently received screen projection data stream is non-audio data, and determines that the current playback state of the display device is in the target mode. Finally, the control module 403 controls the display switch of the display to be in an open state, and controls the display to display non-audio data, wherein, when the playback state is in the target mode, the display switch of the display is in a closed state. In the above technical solution, the display device can automatically control the display switch of the display to be in an open state when determining that the type of the currently received screen projection data stream is non-audio data, and when determining that the current playback state of the display device is in the target mode, and control the display to display non-audio data, without the need for manual operation by the user, to avoid the display switch of the display being in a closed state when the non-audio data is currently being played due to the target mode, and the user cannot view the content of the non-audio data, ensuring that the user can normally view the projection content during the screen projection process, and improving the user's experience.

In order to explain the screen projection processing scheme in the embodiment of the present application in more detail, the following will be explained in an exemplary manner in combination with Figure 26. It can be understood that the steps involved in Figure 26 may include more steps, or fewer steps in actual implementation, and the order of these steps may also be different, so as to implement the screen projection processing method provided in the embodiment of the present application. The embodiment of the present application is not limited.

FIG26 is a flow chart of a screen projection processing method according to an embodiment of the present application. This embodiment can be applied to prevent the display switch of the display from being in the off state during the screen projection process, resulting in the inability to view the projected content. As shown in FIG26, the screen projection processing method specifically includes the following steps:

S51, obtain the type of the currently received screen projection data stream and the current playback status of the display device.

Among them, the currently received screen projection data stream can be understood as the screen projection data stream currently received by the display device and sent by the mobile device. It can be understood that: the user can operate the mobile device to cast the screen. During the screen projection process, the user can perform multiple screen projections, for example, first cast audio data, then cast video data, and then cast image data, etc. Correspondingly, the mobile device will send multiple screen projection data streams to the display device. In this embodiment, there is no specific limitation on the mobile device that sends the screen projection data stream to the display device during the screen projection process. It can be the same mobile device or different mobile devices. Mobile devices may include client devices that support the Airplay protocol, such as mobile phones, iPads, or macbooks. The types of screen projection data streams may include audio data and non-audio data, among which audio data is sound-related data, such as music, recordings, etc. The non-audio data may include video streams, live streams, or broadcast streams, etc. The current playback status of the display device may include: the display device is currently playing only audio, the display device is currently playing only picture content, or the display device is currently playing both audio and picture content.

In order to solve the problem that the user cannot view the projected content because the display switch of the display is in the off state during the screen projection process, the processor of the display device obtains the type of the currently received projection data stream and the current playback status of the display device, so as to determine whether the type of the currently received projection data stream is non-audio data, and determine whether the current playback status of the display device is in target mode, thereby facilitating the execution of subsequent steps.

S52, determine that the type of the currently received projection data stream is non-audio data, and determine that the current playback state of the display device is in target mode, control the display switch of the display device to be in the on state, and control the display to display non-audio data.

When the playback state is in the target mode, the display switch of the display is in the off state, that is, in the target mode, the display is in a black screen state, and the user cannot view the screen content. The display switch of the display can be understood as a switch that controls the display to be in a black screen state or a bright screen state.

The processor of the display device determines that the type of the currently received projection data stream is non-audio data, and determines that the current playback state of the display device is in target mode, based on the type of the currently received projection data stream and the current playback state of the display device, indicating that the display device is now projecting and playing non-audio data, but the display switch of the display is in the off state, and the screen content of the non-audio data cannot be played normally. Therefore, the processor of the display device controls the display switch of the display device to be in the on state, and controls the display to display non-audio data (screen content), so as to achieve the effect of normal playback of non-audio data.

In this embodiment, the processor of the display device first obtains the type of the currently received projection data stream and the current playback status of the display device, and then determines that the type of the currently received projection data stream is non-audio data, and determines that the current playback status of the display device is in the target mode, controls the display switch of the display to be in the on state, and controls the display to display the non-audio data, wherein, when the playback status is in the target mode, the display switch of the display is in the off state. In the above technical scheme, the display device can automatically control the display switch of the display to be in the on state and control the display to display the non-audio data when determining that the type of the currently received projection data stream is non-audio data and determining that the current playback status of the display device is in the target mode, without the need for manual operation by the user, thereby avoiding the situation where the display switch of the display is in the off state when the non-audio data is currently being played due to the target mode, and the user cannot view the content of the non-audio data, thereby ensuring that the user can normally view the projection content during the projection process, thereby improving the user experience.

In some embodiments, determining that the current playback state of the display device is in the target mode may specifically include:

Among them, the Audio Only function is a function provided by the display device to play only sound and turn off the display device screen. Turning off the display device screen means: controlling the display switch of the display to be in the off state. The switch corresponding to the Audio Only function can be understood as a module used to control turning this function on or off. The state of the switch corresponding to the Audio Only function can be controlled by the user operating the remote control or the mobile device connected to the display device. For example, by clicking the Menu button of the remote control (it can also be other buttons, which is not limited in this embodiment), the switch corresponding to the Audio Only function can be called up to control its state. The audio and video middleware interface can be understood as an interface for recording and controlling the state of the switch corresponding to the Audio Only function.

Specifically, the processor of the display device calls the audio and video middleware interface to obtain the status of the switch corresponding to the Audio Only function. When the status of the switch corresponding to the Audio Only function is on, it can be determined that the current playback status of the display device is in the target mode.

In this embodiment, determining that the current playback state of the display device is in the target mode by the above method is simple, fast and efficient.

In some embodiments, controlling the display switch of the display to be in an on state may specifically include:

The audio and video middleware interface is called to turn off the switch corresponding to the Audio Only function to control the display switch of the display to be in the on state.

Specifically, the processor of the display device calls the audio and video middleware interface to turn off the switch corresponding to the Audio Only function, that is, the display device only plays sound and the function of turning off the display device screen is turned off. After turning off the switch corresponding to the Audio Only function, the display switch of the display can be controlled to be in the on state, and the display can display the picture normally.

In this embodiment, the above method can automatically control the display switch of the display to be in the on state, which is simple and efficient, and does not require manual operation by the user, thereby avoiding the situation where the display cannot display the picture normally because the display is in a black screen state when playing non-audio data.

Exemplarily, FIG27 is a schematic diagram of the interface of the display when the current playback state of the display device is in the target mode according to an embodiment of the present application. As shown in FIG27, when the current playback state of the display device is in the target mode (i.e., the state of the switch corresponding to the Audio Only function is not turned on), the display switch of the display is in the off state, and at this time, the interface of the display is a black screen interface.

Exemplarily, Figure 28 is a schematic diagram of the interface of a display when a display switch of a display according to an embodiment of the present application is in an on state and the display displays non-audio data. As shown in Figure 28, when the display switch of the display is in an on state and the display displays non-audio data, the display normally displays the screen content corresponding to the non-video data.

It should be noted that the screen content shown in FIG. 28 is only for exemplary description, and the screen content can be determined according to the playback progress corresponding to the non-video data, which is not limited in this embodiment.

Figure 29 is a flow chart of another screen projection processing method according to an embodiment of the present application. This embodiment is optimized on the basis of the above embodiment. This embodiment can be applied to the process of determining that the type of the currently received screen projection data stream is non-audio data, and determining that the current playback state of the display device is in the target mode. As shown in Figure 29, the screen projection processing method specifically includes the following steps:

S291, obtain the type of the currently received screen projection data stream and the current playback status of the display device.

S292, determine that the type of the currently received projection data stream is non-audio data, and determine that the current playback state of the display device is in target mode, and control the display to display prompt information.

The prompt information is used to prompt the user whether to turn on the display switch of the display.

The processor of the display device determines that the type of the currently received projection data stream is non-audio data, and determines that the current playback state of the display device is in target mode, and controls the display to display prompt information, through which the user can be prompted whether to turn on the display switch of the display.

S293, receiving an operation of turning on a display switch of the display, controlling the display switch of the display to be in an on state, and controlling the display to display video data.

When the processor of the display device receives an operation of turning on a display switch of the display triggered by the user, the processor controls the display switch of the display to be in an on state, and controls the display to display video data.

In this embodiment, the processor of the display device first obtains the type of the currently received projection data stream and the current playback status of the display device, then determines that the type of the currently received projection data stream is non-audio data, and determines that the current playback status of the display device is in target mode, controls the display to display prompt information, and finally receives an operation to turn on the display switch of the display, controls the display switch of the display to be in an on state, and controls the display to display video data. By displaying the prompt information on the display, the user can be prompted to know that the display switch of the display was turned off during the previous screen projection, which can increase the human-computer interaction experience. At the same time, after receiving the operation to turn on the display switch of the display, the display switch of the display is controlled to be in an on state, and controls the display to display video data, avoiding the situation where the display switch of the display is in a closed state (i.e., a black screen state) when non-audio data is currently playing due to the target mode, and the user cannot view the content of the non-audio data, thereby ensuring that the user can normally watch the projection content during the screen projection process and improving the user's usage experience.

Exemplarily, FIG30 is a schematic diagram of an interface when a display displays prompt information according to an embodiment of the present application. As shown in FIG30, after the processor of the display device determines that the type of the currently received projection data stream is non-audio data and that the current playback state of the display device is in target mode, it can control the display to display prompt information. The prompt information may include a countdown prompt. Specifically, the processor of the display device can start a 5-minute countdown, and a prompt page as shown in FIG30 will pop up 15 seconds before the countdown ends. The page includes: whether to turn on the display switch of the display. Off, the operable options include "Yes" and "No". Among them, Yes is used to turn on the display switch of the display, and No is used to turn off the display switch of the display. The 15s in the upper right corner is a time prompt, and its value will continue to decrease as time changes until it reaches 0.

The prompt page shown in FIG30 can prompt the user whether to turn on the display switch of the display to avoid the display being in a black screen state. If no user operation is received, the processor of the display device automatically closes the prompt page after 15 seconds by default and automatically turns on the display switch of the display.

It is understandable that 5 minutes and 15 seconds are both user-defined values and may be other values, which are not limited in this embodiment.

In some embodiments, obtaining the type of the currently received screen projection data stream and the current playback status of the display device may specifically include:

The current screen projection request can be understood as: the screen projection request currently received by the display device and sent by the mobile device. The screen projection request can carry relevant screen projection parameters, such as mirroring, screen projection video or screen projection audio information.

Specifically, after the processor of the display device receives the current screen projection request sent by the mobile device, it determines the channel information of the display device in response to the current screen projection request, wherein the communication information may be the identification information of the channel currently corresponding to the display device. When the processor of the display device determines that the channel of the display device is the channel corresponding to the airplay according to the channel information (indicating that the current screen projection request is not the first screen projection request, that is, it is not the first screen projection), and the current screen projection request is a non-mirror screen projection request, the type of the currently received screen projection data stream and the current playback status of the display device are obtained.

It should be noted that: if the current screen projection request is the first screen projection request or a mirrored screen projection request, the display device will execute the logic of switching channels, that is, switch the channel of the display device to the channel corresponding to the airplay. After the display device executes the logic of switching channels, the display will not have a black screen. When the current screen projection request is: not the first screen projection request and not a mirrored screen projection request, the display device will not execute the logic of switching channels. In this case, if the current playback state of the display device is in target mode (the target mode can be set during the previous screen projection), the display will remain in a black screen state (that is, the display switch of the display is in the off state).

In this embodiment, under the above conditions, obtaining the type of the currently received projection data stream and the current playback status of the display device is conducive to the subsequent determination of whether to execute the step of controlling the display switch of the display to be in the on state, thereby preventing the situation where the non-audio data screen content cannot be played due to the display switch of the display being in the off state.

It should be noted that: if the current screen projection request is the first screen projection request, the display device will execute the logic of switching channels. There is no restriction on the type of screen projection data stream corresponding to the first screen projection request, that is: regardless of whether the screen projection data stream is audio data or non-audio data, the display device will execute the logic of switching channels.

Exemplarily, FIG31 is a schematic diagram of the interface of a display when a display device enters an input page according to an embodiment of the present application. As shown in FIG31, a user can operate the input button of a remote control or operate other mobile devices connected to the display device to enter the input page, which may include multiple options, such as: Home, TV, AirPlay, HDMI (High Definition Multimedia Interface) 1 and HDMI2, etc. When switching from one option in the page to another, the display device will execute the logic of switching channels.

In some embodiments, when the AirPlay portal is in the settings and the AirPlay option is selected, the display device will also execute the logic of switching channels.

In some embodiments, the method may further specifically include:

Among them, the sender of the current screen projection request can be any mobile device, and this embodiment does not limit this.

Specifically, after the processor of the display device receives the current screen projection request sent by the mobile device, it determines the channel information of the display device in response to the current screen projection request. When it is determined according to the channel information that the channel of the display device is not the channel corresponding to the airplay, it can be determined that the current screen projection request is the first screen projection request. At this time, the processor of the display device establishes a screen projection connection between the display device and the sender of the current screen projection request based on the first screen projection request, and switches the channel of the display device to the channel corresponding to the airplay. At the same time, since the current screen projection request is the first screen projection request, the display device will execute the logic of switching channels. After the display device executes the logic of switching channels, the display will not have a black screen. Therefore, the processor of the display device can play the data stream normally after receiving the data stream sent by the sender based on the screen projection connection and the channel corresponding to the airplay.

In this embodiment, through the above process, it can be ensured that the corresponding data stream is played normally when the screen is projected for the first time.

Among them, the AirPlay protocol is a playback technology in the iOS operating system, which can transfer files (including videos, photos and mirrors) on iOS or later versions to devices that support airplay.

Specifically, there is a logic of switching channels in the AirPlay protocol. Therefore, the screen projection protocol between the display device and the sender of the current screen projection request in this embodiment is: AirPlay protocol.

For example, FIG32 is a schematic diagram of a screen projection process of a display device according to an embodiment of the present application. As shown in FIG32, the screen projection process of the display device may include the following steps:

Step 1: When a user casts a screen through a mobile device, the screen casting driver module of the display device can receive the screen casting parameters sent by the mobile device, such as mirroring, screen casting video or screen casting audio and other information, and determine whether the current screen casting is the first screen casting;

Step 2: If the current screen projection is the first screen projection, since the display device is not currently in the AirPlay channel when the screen is projected for the first time, the screen projection driver sends a channel switching message to the channel switching module, so that the channel switching module switches the channel of the display device to the channel corresponding to AirPlay, and then calls the AirPlay function module (such as audio, video or mirroring, etc.) to implement the corresponding function; if the current screen projection is mirror projection, regardless of whether it is the first screen projection, the screen projection driver will send a channel switching message to the channel switching module, and implement mirror projection through the mirroring function module, so that the display device synchronously plays the screen content of the mobile device;

Step 3. If the current screen projection is not the first screen projection and is not a mirror projection, the screen projection driver defaults the channel of the display device to the channel corresponding to AirPlay. Then, when the type of the currently received screen projection data stream is non-audio data, the screen projection driver calls the audio and video middleware interface to obtain the status of the switch corresponding to the Audio Only function. If the status of the switch corresponding to the Audio Only function is on, the status of the switch corresponding to the Audio Only function is controlled to be off, so as to control the display switch of the monitor to be on, and control the monitor to display non-audio data; if the status of the switch corresponding to the Audio Only function is off, the monitor is controlled to display non-audio data.

FIG33 is a schematic diagram of the overall process of a screen projection processing method according to an embodiment of the present application. As shown in FIG33 , the overall process may specifically include the following steps:

S331, the user selects any music player software in the mobile device to cast the audio (Audio) screen (i.e., the user casts the screen to play the audio data), and then the TV will display the Audio related information in the mobile device and start playing the Audio;

S332, the user is on the Audio page of the display device, and clicks the "Audio Only" option in the menu by operating the menu button of the remote control (or other methods), that is, turning on the switch corresponding to the Audio Only function, at which time the display device only plays sound and turns off the display of the display device;

S333. When the user selects non-audio data, such as video data, for screen projection playback in the mobile device, because the display device has played audio before, the logic of switching channels will not be executed when playing the video at this time, and it is considered that the channel of the display device has not sent changes. Since the switch corresponding to the Audio Only function is in the on state, the display will not light up the screen at this time and will be in a black screen state;

S334: The user plays non-audio data on the screen;

S335. The display device determines whether the type of the currently received projection data stream is non-audio data, and whether the state of the switch corresponding to the Audio Only function is on; if yes, that is, the type of the currently received projection data stream is non-audio data, and the state of the switch corresponding to the Audio Only function is on, then execute step S336; otherwise, the type of the currently received projection data stream is audio data, or the state of the switch corresponding to the Audio Only function is not on (i.e., off), then execute step S338;

S336. If the display device determines that the type of the currently received projection data stream is non-audio data, and the state of the switch corresponding to the Audio Only function is on, then turn off the switch corresponding to the Audio Only function;

S337, the display switch of the display is in the on state;

S338. Control the display device to play the current non-audio data normally.

In some embodiments, the state change of screen projection can be determined in the following manner, and at this time, it can be identified that the current state of Airplay is Audio/Video/Screen.

The code looks like this:

In the above code file, the parameter ui is the current Airplay status, which can be Screen (equivalent to the Airplay homepage)/Video/Audio. Whenever the user casts the screen, the notifyUIPage method will be called. For example, if the user casts the screen to play video data, then ui = video; if the user casts the screen to play audio data, then ui = audio.

Among them, the implementation logic of quitAudioOnlyWhenAudioToVideo(ui) is as follows:

First, get the last Airplay status. When it is Audio and the current Airplay status is Video, determine whether the switch status corresponding to the Audio Only function is on. Turn off the Audio Only function and the display screen will light up.

The code looks like this:

To summarize, the embodiment of the present application executes the above-mentioned screen projection processing method on the display device. The processor of the display device first obtains the type of the currently received screen projection data stream and the current playback status of the display device, and then determines that the type of the currently received screen projection data stream is non-audio data, and determines that the current playback status of the display device is in the target mode, controls the display switch of the display to be in the on state, and controls the display to display the non-audio data, wherein, when the playback status is in the target mode, the display switch of the display is in the off state. In the above-mentioned technical scheme, the display device can automatically control the display switch of the display to be in the on state when determining that the type of the currently received screen projection data stream is non-audio data, and determining that the current playback status of the display device is in the target mode, and controls the display switch of the display to be in the on state, and controls the display to display the non-audio data, so as to avoid the situation that the display switch of the display is in the off state when the non-audio data is currently being played due to the target mode, and the user cannot view the content of the non-audio data, thereby ensuring that the user can normally view the projection content during the screen projection process, thereby improving the user's experience.

According to an embodiment of the present application, a computer-readable non-volatile storage medium is also provided, on which a computer program is stored. When the computer program is executed by a processor, the various processes of the above-mentioned device list acquisition method are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

Among them, the computer-readable non-volatile storage medium can be a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

The embodiment of the present application also provides a computer program product, including: when the computer program product is run on a computer, the computer is enabled to implement the above method.

For the convenience of explanation, the above description has been made in conjunction with specific embodiments. However, the above exemplary discussion is not intended to be exhaustive or limit the embodiments to the specific forms disclosed above. Based on the above teachings, various modifications and variations can be obtained. The selection and description of the above embodiments are to better explain the principles and practical applications, so that those skilled in the art can better use the embodiments and various different variations of the embodiments suitable for specific use considerations.

Claims

An electronic device, comprising:

The communicator is configured to: after establishing a connection with a HomeKit client based on the HomeKit protocol, receive a device list acquisition request sent by the HomeKit client, wherein the device list acquisition request is used to instruct to acquire device information of each target device connected to the electronic device, wherein the device information includes an identifier of the device;

A processor configured to: determine device information of the electronic device and device information of each target device based on the device list acquisition request, wherein the device information includes an identifier of a corresponding device;

Generate target information based on the device information of the electronic device, the device information of each target device and at least one device placeholder, each device placeholder is used to indicate an unknown device to be connected, and the device information corresponding to each device placeholder is empty;

The communicator is further configured to: send the target information to the HomeKit client, so that the HomeKit client displays a device list based on the target information, wherein the device list includes an identifier of the electronic device and an identifier of each target device.
The electronic device according to claim 1,

The processor is further configured to: after sending the target information to the HomeKit client, detect that a first device establishes a connection with the electronic device;

updating one of the device placeholders in the target information with the device information of the first device to update the target information;

The communicator is further configured to: send the updated target information to the HomeKit client, so that the HomeKit client updates the device list based on the updated target information, and the updated device list includes the identifier of the electronic device, the identifiers of each target device and the identifier of the first device.
The electronic device according to claim 1,

The processor is further configured to: after sending the target information to the HomeKit client, detect that a second device is disconnected from the electronic device, the second device being any one of the target devices;

Updating the device information of the second device in the target information to a device placeholder to update the target information;

The communicator is further configured to: send the updated target information to the HomeKit client so that the HomeKit client updates the device list based on the updated target information, and the updated device list includes the identifier of the electronic device and the identifier of each device among the target devices except the second device.
The electronic device according to claim 1, wherein each target device comprises: at least one third device;

or,

The target devices include: at least one third device and at least one fourth device;

Each of the third devices is directly connected to the electronic device, and each of the fourth devices is connected to the electronic device through the corresponding third device.
The electronic device according to claim 4, wherein the electronic device is connected to each of the third devices via an HDMI interface;

In the case that each of the target devices includes the at least one fourth device, each of the third devices is connected to the corresponding third device via an HDMI interface.
According to the electronic device according to claim 1, the device information also includes the type of the corresponding device; the device list also includes the type of the electronic device and the type of each target device.
The electronic device according to any one of claims 1 to 6, wherein the sum of the number of each target device and the number of the at least one device placeholder is equal to a first preset value;

or,

The number of the at least one device placeholder is equal to a second preset value.
A data processing method for an electronic device, the method comprising:

After establishing a connection with a HomeKit client based on the HomeKit protocol, receiving a device list acquisition request sent by the HomeKit client, the device list acquisition request is used to instruct to acquire device information of each target device connected to the electronic device, the device information including an identifier of the device;

Based on the device list acquisition request, determine the device information of the electronic device and the device information of each target device, wherein the device information includes an identification of a corresponding device;

Generate target information based on the device information of the electronic device, the device information of each target device and at least one device placeholder, each device placeholder is used to indicate an unknown device to be connected, and the device information corresponding to each device placeholder is empty;

The target information is sent to the HomeKit client, so that the HomeKit client displays a device list based on the target information, wherein the device list includes an identifier of the electronic device and an identifier of each target device.
According to the method of claim 8, after sending the target information to the HomeKit client, the method further comprises:

Detecting that a first device establishes a connection with the electronic device;

updating one of the device placeholders in the target information with the device information of the first device to update the target information;

The updated target information is sent to the HomeKit client so that the HomeKit client updates the device list based on the updated target information, wherein the updated device list includes an identifier of the electronic device, identifiers of each target device, and an identifier of the first device.
According to the method of claim 8, after sending the target information to the HomeKit client, the method further comprises:

detecting that a second device is disconnected from the electronic device, the second device being any one of the target devices;

Updating the device information of the second device in the target information to a device placeholder to update the target information;

Sending the updated target information to the HomeKit client so that the HomeKit client updates the device list based on the updated target information, wherein the updated device list includes an identifier of the electronic device and an identifier of each device among the target devices except the second device.
According to the method of claim 8, the device information also includes the type of the corresponding device; the device list also includes the type of the electronic device and the type of each target device.
The method according to any one of claims 8 to 11, wherein the sum of the number of each target device and the number of the at least one device placeholder is equal to a first preset value;

or,

The number of the at least one device placeholder is equal to a second preset value.
A computer-readable non-volatile storage medium, comprising: a computer program stored on the computer-readable storage medium, and when the computer program is executed by a processor, the data processing method according to any one of claims 8 to 12 is implemented.