CN117478682A

CN117478682A - Method, equipment and cooperative work system for establishing point-to-point channel

Info

Publication number: CN117478682A
Application number: CN202210859323.0A
Authority: CN
Inventors: 何明祺
Original assignee: Honor Device Co Ltd
Current assignee: Honor Device Co Ltd
Priority date: 2022-07-21
Filing date: 2022-07-21
Publication date: 2024-01-30

Abstract

The application provides a method, equipment and a cooperative work system for establishing a point-to-point channel. According to the method, the traditional P2P channel establishment standard is omitted, the second electronic device initiating the request directly designates own role information when the P2P channel is established, and identifiers for identifying the identities of the two parties are exchanged in an authentication stage, so that after the first electronic device receives a P2P channel establishment request message sent by the second electronic device, the identifier carried in the P2P channel establishment request message is directly checked according to the identifier obtained in the authentication stage, whether a hot spot is established or directly accessed is determined according to the role information after the verification is passed, and finally connection between the two is realized according to the hot spot information, such as a password and an account number, the process of establishing the P2P channel is greatly simplified, and the P2P channel between the two is established in a wireless access point mode based on the hot spot information, so that the success rate of establishing the P2P channel is greatly improved.

Description

Method, equipment and cooperative work system for establishing point-to-point channel

Technical Field

The present disclosure relates to the field of communications technologies, and in particular, to a method and an apparatus for establishing a peer-to-peer channel, and a cooperative system.

Background

With the development of internet technology and internet of things (Internet of Things, IOT) technology, various IOT devices are layered endlessly. The IOT devices can work alone or be matched with other electronic devices, such as mobile phones and tablet computers (hereinafter simply referred to as tablet computers), and are suitable for more use scenes. Taking IOT devices as examples of desk lamps with camera shooting function and audio frequency function (hereinafter referred to as "desk lamps"), the desk lamps currently establish Peer-to-Peer (P2P) channels with electronic devices with powerful processing capability such as mobile phones and tablets, so that collected image data and audio data can be transmitted to the electronic devices, and the electronic devices can carry out subsequent processing.

However, the current process of establishing the P2P channel involves more flows, complicated operations, and redundant flows, and the IOT device is usually a thin device (system resources and processing capacity are limited), so the current manner of establishing the P2P channel is limited by the capability of the IOT device, and the establishment process is slow. This is a significant resource and time consumption for IOT devices with less resources, and therefore a need exists to provide a fast and efficient method of establishing P2P channels.

Disclosure of Invention

In order to solve the technical problems, the application provides a method, equipment and a collaborative work system for establishing a point-to-point channel, which aim to simplify the establishment flow of a P2P channel, thereby reducing the consumption of IOT equipment resources and time and achieving the purpose of quickly and efficiently establishing the P2P channel between IOT equipment and electronic equipment.

In a first aspect, the present application provides a method of establishing a point-to-point channel. The method is applied to the first electronic equipment, the first electronic equipment and the second electronic equipment exchange identifiers for identifying the first electronic equipment in an authentication stage, and the method comprises the following steps: receiving a point-to-point P2P channel establishment request message sent by a second electronic device, wherein the P2P channel establishment request message comprises an identifier of the second electronic device, role information and authentication information corresponding to the role, and the role information is used for indicating that the second electronic device is one party for creating a hot spot or one party for accessing the hot spot; verifying the identifier included in the P2P channel establishment request message according to the identifier of the second electronic device obtained in the authentication stage; when the verification is successful and the role information indicates that the second electronic equipment is one party accessing the hot spot, creating the hot spot; and sending a P2P channel establishment response message to the second electronic equipment, wherein the P2P channel establishment response message comprises an identifier of the first electronic equipment and first hotspot information of the created hotspot.

The first electronic device is, for example, an IOT device.

In some implementations, the IOT device may be any device capable of collecting data, establishing a P2P channel with the second electronic device, and transmitting the data, for example. In particular, in the present application, IOT devices are exemplified by a desk lamp, a camera, etc. having an image capturing function and an audio function.

The second electronic device is, for example, an electronic device with a powerful processing capability such as a mobile phone or a tablet.

The first electronic device starts a wireless access point (softtap) through software, so that the second electronic device can access the softtap, and the first electronic device and the second electronic device are in the same local area network. The first electronic device and the second electronic device may then send an identifier capable of identifying the device's uniqueness to each other based on a restricted application protocol (The Constrained Application Protocol, COAP), i.e. the first electronic device sends an identifier identifying the first electronic device to the second electronic device based on the COAP protocol, and the second electronic device sends an identifier identifying the second electronic device to the first electronic device based on the COAP protocol.

In the application, the establishment of the P2P channel is performed after the first electronic device and the second electronic device have completed the operation of exchanging identifiers in the authentication stage, so that the user does not need to manually input authentication information to perform authentication when the P2P channel is established, thereby simplifying the user operation.

In addition, even if the first electronic device and the second electronic device do not authenticate when a request for establishing the P2P channel is initiated, in the application, the first electronic device and the second electronic device complete authentication according to the authentication process given above, and because the user does not need to manually input authentication information in the authentication stage, the scheme for establishing the P2P channel provided by the application is intersected with the existing mode, and the establishment process is still concise.

Wherein the party creating the hotspot, such as GO, hereinafter; a party accessing the hotspot, such as the GC described below.

The authentication information corresponding to the roles is different according to different angles. Illustratively, when the role is GO, the authentication information is, for example, hotspot information, and when the role is GC, the authentication information is, for example, a personal identification code (Personal identification number, PIN code).

For example, in the present application, the hotspot information specifically includes an account number and a password corresponding to the hotspot.

In addition, in order to facilitate distinguishing between the hotspot information of the hotspot created by the first electronic device and the hotspot information of the hotspot created by the second electronic device, in this application, the hotspot information of the hotspot created by the first electronic device is referred to as first hotspot information, and the hotspot information of the hotspot created by the second electronic device is referred to as second hotspot information.

Therefore, the method for establishing the P2P channel omits the traditional P2P channel establishment standard, namely, the processes of scanning MAC addresses, sending broadcasting, negotiating roles, connecting a P2P network port, inputting authentication information by a user and the like are skipped, the second electronic equipment initiating a request directly designates own role information when establishing the P2P channel, and identifiers for identifying the identities of the two parties are exchanged in the authentication stage, so that the first electronic equipment directly checks the identifiers carried in the P2P channel establishment request message according to the identifiers obtained in the authentication stage after receiving the P2P channel establishment request message sent by the second electronic equipment, determines whether to establish a hot spot or directly Access the hot spot according to the role information after the verification is passed, finally realizes the connection between the two according to the hot spot information, such as a password and an account number, thereby greatly simplifying the process of establishing the P2P channel, and greatly improving the success rate of establishing the P2P channel by a wireless Access Point (AP) mode based on the hot spot information.

That is, the method for establishing the P2P channel can effectively reduce the consumption of the IOT device resources and time, thereby achieving the purpose of quickly and efficiently establishing the P2P channel between the IOT device and the electronic device, and being better suitable for the interaction scene with various thin devices.

In addition, it should be understood that the method for establishing a P2P channel provided in the present application is equally applicable to non-IOT devices, that is, any two second electronic devices may also establish a P2P channel based on this manner. For this scenario, the electronic device sending the request message for establishing the P2P channel is the nominal second electronic device, and the electronic device receiving the request message is the first electronic device.

In other implementations, the method for establishing a P2P channel provided in the present application may also be used by any two first electronic devices to establish a P2P channel. For this scenario, the electronic device sending the request message for establishing the P2P channel is the nominal second electronic device, and the electronic device receiving the request message is the first electronic device.

That is, the first electronic device and the second electronic device in the present application are not limited to specific types of electronic devices, but are merely for distinguishing roles.

According to the first aspect, when the role information indicates that the second electronic device is a party for creating a hotspot, the authentication information is second hotspot information of the hotspot created by the second electronic device; the method further comprises the steps of: when the verification is successful and the role information indicates that the second electronic equipment is a party for creating the hot spot, a P2P channel between the second electronic equipment and the second electronic equipment is established according to the second hot spot information; and sending a P2P channel establishment response message to the second electronic equipment, wherein the P2P channel establishment response message comprises an identifier, role information and authentication information corresponding to the role of the first electronic equipment.

In this way, when the second electronic device is a party for creating the hot spot, the first electronic device directly establishes connection with the second electronic device in an AP manner according to the second hot spot information carried in the P2P channel establishment request message, so that the P2P channel can be established without other additional operations, which is fast and concise.

According to a first aspect, or any implementation manner of the first aspect, after creating the hotspot, the method further includes: distributing an Internet Protocol (IP) address for the second electronic equipment, and establishing a corresponding relation between the IP address and an identifier of the second electronic equipment; according to the correspondence, the IP address and the identifier of the second electronic device are added to the IP address/identifier management table.

Therefore, the first electronic device can realize one-to-many connection management by dividing the IP address for the second electronic device with which the P2P channel needs to be established and establishing the corresponding relation between the allocated IP address and the identifier of the second electronic device, so that the second electronic device can conveniently know which IP addresses are used for establishing the P2P channel with the second electronic device, and the subsequent management such as data transmission, disconnection of the P2P channel and the like is convenient.

According to the first aspect, or any implementation manner of the first aspect, the method further includes: after a P2P channel between the second electronic device and the second electronic device is established, acquiring state information of the P2P channel, wherein the state information indicates that the P2P channel is opened or disconnected; the status information is added to the IP address/identifier management table, and a correspondence is established between the identifier of the second electronic device and the IP address.

In this way, the state information of the successfully established P2P channel is obtained in real time, and the state information is updated to the corresponding identifier and IP address in the IP address/identifier management table, so as to be used as the calibration of the P2P channel corresponding to the identifier and the IP address, thereby facilitating subsequent management, such as data transmission, disconnection of the P2P channel, and the like, according to the state information.

According to the first aspect, or any implementation manner of the first aspect, after the establishing a P2P channel with the second electronic device, the method further includes: receiving a P2P channel disconnection notification message sent by the second electronic device, wherein the P2P channel disconnection notification message comprises an identifier of the second electronic device; and disconnecting the P2P channel between the second electronic device identified by the identifier in the P2P channel disconnection notification message.

In this way, according to the identifier carried in the P2P channel disconnection notification message, it can be quickly determined which second electronic device needs to disconnect the P2P channel.

According to the first aspect, or any implementation manner of the first aspect, the method further includes: and modifying the state information corresponding to the identifier of the second electronic equipment in the IP address/identifier management table to be disconnected.

In this way, when the P2P channel is disconnected, the corresponding state information is modified instead of being deleted directly, so that the IP address is not required to be reassigned when the disconnected P2P channel is reconnected immediately, and the corresponding relationship is established, and the method can be better suitable for the scene that the P2P channel is disconnected immediately after Ma Chong is disconnected immediately due to network reasons or other anomalies.

According to the first aspect, or any implementation manner of the first aspect, the method further includes: and deleting the identifier of the second electronic equipment recorded in the IP address/identifier management table, and the state information and the IP address corresponding to the identifier of the second electronic equipment.

Thus, redundant information in the IP address/identifier management table can be reduced, and occupation of equipment resources and space can be reduced.

According to a first aspect, or any implementation of the first aspect above, the identifier of the first electronic device is a media access control, MAC, address. I.e. the MAC address to which the first electronic device corresponds.

It should be noted that, this manner of using the MAC address as the first electronic device identifier is generally applicable to a usage scenario in which the MAC address is fixed.

According to the first aspect, or any implementation of the first aspect above, the identifier of the first electronic device identifies the BSSID for the basic service group.

Since the MAC address corresponding to the P2P portal of the partially-structured chip may not be fixed, by adopting the fixed and unique BSSID as the identifier of the first electronic device, it can be ensured that the first electronic device identified by each identifier is unique and fixed.

According to the first aspect, or any implementation manner of the first aspect, the role information of the second electronic device is determined according to the product characteristics of the second electronic device.

For example, when the second electronic device is a mobile phone, a tablet or the like that needs to process data, the role information of the second electronic device may be determined as GC; and for a second electronic device that can access a plurality of electronic devices and transmit data to the accessed electronic device, its role information may be determined as GO.

Similarly, the role information of the first electronic device may also be determined based on the product characteristics, and a specific determining manner is similar to that of the second electronic device, which is not described herein.

Because the characteristics of the product can be determined according to the current use scene and the historical use scene, the role information can be directly determined, so that role negotiation is not required when a P2P channel is established, and a great amount of time is saved.

In a second aspect, the present application provides a method of establishing a point-to-point channel. The method is applied to the second electronic equipment, the second electronic equipment exchanges an identifier for identifying the second electronic equipment with the first electronic equipment in an authentication stage, and the method comprises the following steps: sending a point-to-point P2P channel establishment request message to the first electronic equipment, wherein the P2P channel establishment request message comprises an identifier of the second electronic equipment, role information and authentication information corresponding to the role, and the role information is used for indicating that the second electronic equipment is one party for creating a hot spot or one party for accessing the hot spot; when the role information indicates that the second electronic equipment is one party accessing the hot spot, receiving a P2P channel establishment response message sent by the first electronic equipment, wherein the P2P channel establishment response message comprises an identifier of the first electronic equipment and first hot spot information of the created hot spot; verifying the identifier included in the P2P channel establishment response message according to the identifier of the first electronic device obtained in the authentication stage; and when the verification is successful, a P2P channel between the second electronic equipment and the first electronic equipment is established according to the first hot spot information.

According to a second aspect, the method further comprises: and sending a P2P channel disconnection notification message to the first electronic device, wherein the P2P channel disconnection notification message comprises an identifier of the second electronic device.

According to a second aspect, or any implementation of the second aspect above, the identifier of the second electronic device is a media access control, MAC, address. I.e. the MAC address to which the second electronic device corresponds.

It should be noted that, this manner of using the MAC address as the second electronic device identifier is generally applicable to a usage scenario in which the MAC address is fixed.

According to a second aspect, or any implementation of the second aspect above, the second electronic device identifies the BSSID for the basic service group.

Because the MAC address corresponding to the P2P network port of the chip with the partial framework is possibly unfixed, by adopting the fixed and unique BSSID as the identifier of the second electronic equipment, the second electronic equipment identified by each identifier can be ensured to be unique and fixed, and further the unique and fixed second electronic equipment, the state information of the P2P channel between the second electronic equipment and the IP address distributed for the second electronic equipment can be inquired in the IP address/identifier management table recorded at the first electronic equipment side according to the identifier of the second electronic equipment, so that the subsequent management and maintenance are facilitated.

Any implementation manner of the second aspect and the second aspect corresponds to any implementation manner of the first aspect and the first aspect, respectively. The technical effects corresponding to the second aspect and any implementation manner of the second aspect may be referred to the technical effects corresponding to the first aspect and any implementation manner of the first aspect, which are not described herein.

In a third aspect, the present application provides an electronic device. The electronic device includes: a memory and a processor, the memory and the processor coupled; the memory stores program instructions that, when executed by the processor, cause the electronic device to perform the instructions of the first aspect and the methods in any possible implementation manner of the first aspect, or instructions for performing the second aspect and the methods in any possible implementation manner of the second aspect.

By way of example, in some implementations, the electronic device may be, for example, a first electronic device as described above, or a second electronic device.

For example, in other implementations, the electronic devices may perform role replacement according to an actual usage scenario, i.e., the same electronic device may be used as both the first electronic device and the second electronic device.

Illustratively, when the electronic device is the first electronic device described above, any implementation manner of the third aspect and any implementation manner of the third aspect correspond to any implementation manner of the first aspect and the first aspect, respectively. The technical effects corresponding to the third aspect and any implementation manner of the third aspect may be referred to the technical effects corresponding to the first aspect and any implementation manner of the first aspect, which are not described herein.

Illustratively, when the electronic device is the second electronic device described above, any implementation manner of the third aspect and any implementation manner of the third aspect correspond to any implementation manner of the second aspect and the second aspect, respectively. The technical effects corresponding to the third aspect and any implementation manner of the third aspect may be referred to the technical effects corresponding to the second aspect and any implementation manner of the second aspect, which are not described herein.

In a fourth aspect, the present application provides a co-operating system comprising a first electronic device for performing any one of the implementations of the first aspect and the first aspect, and a second electronic device for performing any one of the implementations of the second aspect and the second aspect.

Any implementation manner of the fourth aspect and any implementation manner of the fourth aspect correspond to any implementation manner of the first aspect and any implementation manner of the first aspect, or any implementation manner of the second aspect and any implementation manner of the second aspect, respectively. Technical effects corresponding to any implementation manner of the fourth aspect may be referred to any implementation manner of the first aspect and the first aspect, or technical effects corresponding to any implementation manner of the second aspect and the second aspect, which are not described herein.

In a fifth aspect, the present application provides a computer readable medium storing a computer program comprising instructions for performing the method of the first aspect and any one of the possible implementations of the first aspect, and instructions for performing the method of the second aspect and any one of the possible implementations of the second aspect.

Any implementation manner of the fifth aspect and any implementation manner of the fifth aspect correspond to any implementation manner of the first aspect and any implementation manner of the first aspect, or any implementation manner of the second aspect and any implementation manner of the second aspect, respectively. Technical effects corresponding to any implementation manner of the fifth aspect may be referred to any implementation manner of the first aspect and the first aspect, or technical effects corresponding to any implementation manner of the second aspect and the second aspect, which are not described herein.

In a sixth aspect, the present application provides a computer program comprising instructions for performing the method of the first aspect and any one of the possible implementations of the first aspect, and instructions for performing the method of the second aspect and any one of the possible implementations of the second aspect.

Any implementation manner of the sixth aspect and any implementation manner of the sixth aspect correspond to any implementation manner of the first aspect and any implementation manner of the first aspect, or any implementation manner of the second aspect and any implementation manner of the second aspect, respectively. Technical effects corresponding to any implementation manner of the sixth aspect may be referred to any implementation manner of the first aspect and the first aspect, or technical effects corresponding to any implementation manner of the second aspect and the second aspect, which are not described herein.

In a seventh aspect, the present application provides a chip comprising processing circuitry, a transceiver pin. Wherein the transceiver pin and the processing circuit communicate with each other via an internal connection path, the processing circuit performing the method of the first aspect and any one of the possible implementations of the first aspect, or performing the method of the second aspect and any one of the possible implementations of the second aspect, to control the receiving pin to receive signals, to control the transmitting pin to transmit signals.

Any implementation manner of the seventh aspect and any implementation manner of the seventh aspect correspond to any implementation manner of the first aspect and any implementation manner of the first aspect, or any implementation manner of the second aspect and any implementation manner of the second aspect, respectively. Technical effects corresponding to any implementation manner of the seventh aspect may be referred to any implementation manner of the first aspect and the first aspect, or technical effects corresponding to any implementation manner of the second aspect and the second aspect, which are not described herein.

Drawings

FIGS. 1 a-1 b are exemplary illustrations of an application scenario;

fig. 2a is a schematic diagram of a hardware structure of an exemplary electronic device;

FIG. 2b is a schematic diagram of a software architecture of an exemplary electronic device;

fig. 3a is a schematic diagram of a hardware structure of an exemplary illustrated internet of things device;

fig. 3b is a schematic software architecture diagram of an exemplary illustrated internet of things device;

FIG. 4a is a schematic diagram of module interaction according to an embodiment of the present disclosure;

FIG. 4b is a schematic diagram of module interaction according to an embodiment of the present disclosure;

FIGS. 5 a-5 b are exemplary illustrations of an application scenario;

FIG. 6a is a schematic diagram of module interaction according to an embodiment of the present disclosure;

FIG. 6b is a schematic diagram of module interaction according to an embodiment of the present disclosure;

7 a-7 b are exemplary illustrations of one application scenario;

fig. 8 is a schematic diagram of module interaction provided in an embodiment of the present application;

9 a-9 c are exemplary illustrations of one application scenario;

FIG. 10 is a schematic diagram of module interaction according to an embodiment of the present application

FIGS. 11 a-11 b are exemplary illustrations of an application scenario;

FIG. 12 is a functional block diagram illustrating the electronic device and IOT device involved in establishing a P2P channel;

Fig. 13 is a schematic diagram illustrating one example of establishing a P2P channel;

fig. 14 is a schematic diagram illustrating yet another exemplary P2P channel establishment;

FIG. 15 is a schematic diagram of an exemplary illustrated disconnection of a P2P channel;

FIG. 16 is an interactive schematic diagram of functional modules involved in an exemplary illustrated authentication phase;

FIG. 17 is an interactive schematic diagram illustrating functional modules involved in a P2P channel setup phase;

fig. 18 is an interaction diagram of the functional modules involved in the exemplary illustrated P2P channel disconnection phase.

Detailed Description

The following description of the embodiments of the present application will be made clearly and fully with reference to the accompanying drawings, in which it is evident that the embodiments described are some, but not all, of the embodiments of the present application. All other embodiments, which can be made by one of ordinary skill in the art based on the embodiments herein without making any inventive effort, are intended to be within the scope of the present application.

The term "and/or" is herein merely an association relationship describing an associated object, meaning that there may be three relationships, e.g., a and/or B, may represent: a exists alone, A and B exist together, and B exists alone.

The terms first and second and the like in the description and in the claims of embodiments of the present application are used for distinguishing between different objects and not necessarily for describing a particular sequential order of objects. For example, the first target object and the second target object, etc., are used to distinguish between different target objects, and are not used to describe a particular order of target objects.

In the embodiments of the present application, words such as "exemplary" or "such as" are used to mean serving as examples, illustrations, or descriptions. Any embodiment or design described herein as "exemplary" or "for example" should not be construed as preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplary" or "such as" is intended to present related concepts in a concrete fashion.

In the description of the embodiments of the present application, unless otherwise indicated, the meaning of "a plurality" means two or more. For example, the plurality of processing units refers to two or more processing units; the plurality of systems means two or more systems.

With the development of the internet, online education is popular with more and more people, and users (such as students) have an increasing demand for online education. In some application scenarios, when a new word is encountered, students can perform online word searching to obtain relevant interpretation; in some application scenarios, online reading of book content is more convenient for students to learn knowledge and pronunciation; in some application scenarios, students need to submit their jobs online. Therefore, how to meet the online education needs of users based on intelligent devices is a problem to be solved.

At present, aiming at an online education scene, a user usually uses an intelligent learning machine with a shooting function and a display function, and the intelligent learning machine needs a camera or a reflecting mirror at a special position to shoot books, so that the universality and the usability are not strong. In addition, the equipment with the shooting function and the display function needs stronger hardware and system support, and the equipment cost is higher. Furthermore, how to provide better online education experience for users based on intelligent equipment, improves universality and usability, reduces online education cost and is a problem to be solved.

The collaborative work system provided by the embodiment of the application can be applied to an online education scene. The system working system comprises electronic equipment and a desk lamp which are in communication connection, wherein a camera is arranged on the desk lamp and can be used for shooting books downwards. The electronic equipment invokes the camera of the desk lamp to collect images and combines the platform online education resources to meet the online education requirements of users. The electronic device may be a tablet computer or a mobile phone. In addition to the online educational scenario, the electronic device and desk lamp may also serve users based on their respective basic functions (i.e., communication function and lighting function). Therefore, the system working system can create better online education experience for the user based on two intelligent devices commonly used by the user, and has strong universality and usability. In addition, as the tablet personal computer or the mobile phone and the like are necessary products of the household, the cost display of the desk lamp equipment with the shooting function is lower than that of the equipment with the shooting function and the display function, and the online education cost of the user is greatly reduced.

The technical scheme provided by the application is explained below by taking electronic equipment as a flat plate as an example.

Fig. 1a shows an exemplary application scenario. As shown in fig. 1a, the co-operating system comprises a tablet 100 and a desk lamp 200 which establish a communication connection. The desk lamp 200 includes a camera 201 for capturing images downwards, for example, for capturing text or picture contents in a book downwards. An education APP (Application) is installed in the tablet 100, and the education APP may call the camera 201 of the desk lamp 200 to collect images, and provide various online education functions, such as online word checking, online reading, online submission, etc., for the user according to the images collected by the camera 201 of the desk lamp 200.

Although the tablet is also provided with the front camera and the rear camera, no matter which camera is used for shooting books, the user is required to hold the tablet by hand to aim the cameras at the books, so that the shooting is unstable, the finger reading and the point reading operations executed by the user by hands are affected, and better online education experience cannot be provided for the user. As shown in fig. 1a, the tablet 100 and the desk lamp 200 can be placed at fixed positions, the tablet 100 uses the camera 201 of the desk lamp 200 to shoot a book, and the shot images are stable, so that the success rate of content identification is high, and a user can flexibly perform finger reading and point reading operations in the book. Therefore, the linkage of the tablet computer and the table lamp can provide a better online education experience for the user.

As shown in fig. 1b, the tablet 100 and the desk lamp 200 may perform near field communication or far field communication. The near field communication can complete information interaction among devices through the router and other devices, and the far field communication can complete information interaction among devices through the cloud server. Illustratively, the tablet 100 and the desk lamp 200 may implement near field communication based on Wi-Fi (wireless fidelity ) network protocols or the like.

Fig. 2a is a schematic structural diagram of the electronic device 100. Alternatively, the electronic device 100 may be a terminal, which may also be referred to as a terminal device, and the terminal may be a cellular phone (cellular phone) or a tablet computer (pad), which is not limited in this application. It should be noted that the schematic structural diagram of the electronic device 100 may be applied to the flat panel in fig. 1a to 1 b. It should be understood that the electronic device 100 shown in fig. 2a is only one example of an electronic device, and that the electronic device 100 may have more or fewer components than shown in the figures, may combine two or more components, or may have a different configuration of components. The various components shown in fig. 2a may be implemented in hardware, software, or a combination of hardware and software, including one or more signal processing and/or application specific integrated circuits.

The electronic device 100 may include: processor 110, external memory interface 120, internal memory 121, universal serial bus (universal serial bus, USB) interface 130, charge management module 140, power management module 141, battery 142, antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, audio module 170, speaker 170A, receiver 170B, microphone 170C, headset interface 170D, sensor module 180, keys 190, motor 191, indicator 192, camera 193, display 194, and subscriber identity module (subscriber identification module, SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor, a gyroscope sensor, an acceleration sensor, a temperature sensor, a motion sensor, a barometric sensor, a magnetic sensor, a distance sensor, a proximity sensor, a fingerprint sensor, a touch sensor, an ambient light sensor, a bone conduction sensor, etc.

The processor 110 may include one or more processing units, such as: the processor 110 may include an application processor (application processor, AP), a modem processor, a graphics processor (graphics processing unit, GPU), an image signal processor (image signal processor, ISP), a controller, a memory, a video codec, a digital signal processor (digital signal processor, DSP), a baseband processor, and/or a neural network processor (neural-network processing unit, NPU), etc. Wherein the different processing units may be separate devices or may be integrated in one or more processors.

The controller may be a neural hub and a command center of the electronic device 100, among others. The controller can generate operation control signals according to the instruction operation codes and the time sequence signals to finish the control of instruction fetching and instruction execution.

A memory may also be provided in the processor 110 for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory.

The USB interface 130 is an interface conforming to the USB standard specification, and may specifically be a Mini USB interface, a Micro USB interface, a USB Type C interface, or the like. The USB interface 130 may be used to connect a charger to charge the electronic device 100, and may also be used to transfer data between the electronic device 100 and a peripheral device. And can also be used for connecting with a headset, and playing audio through the headset. The interface may also be used to connect other electronic devices, such as AR devices, etc.

The charge management module 140 is configured to receive a charge input from a charger. The charger can be a wireless charger or a wired charger. In some wired charging embodiments, the charge management module 140 may receive a charging input of a wired charger through the USB interface 130. In some wireless charging embodiments, the charge management module 140 may receive wireless charging input through a wireless charging coil of the electronic device 100. The charging management module 140 may also supply power to the electronic device through the power management module 141 while charging the battery 142.

The power management module 141 is used for connecting the battery 142, and the charge management module 140 and the processor 110. The power management module 141 receives input from the battery 142 and/or the charge management module 140 and provides power to the processor 110, the internal memory 121, the external memory, the display 194, the camera 193, the wireless communication module 160, and the like.

The wireless communication function of the electronic device 100 may be implemented by the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, a modem processor, a baseband processor, and the like.

The antennas 1 and 2 are used for transmitting and receiving electromagnetic wave signals. Each antenna in the electronic device 100 may be used to cover a single or multiple communication bands. Different antennas may also be multiplexed to improve the utilization of the antennas. For example: the antenna 1 may be multiplexed into a diversity antenna of a wireless local area network. In other embodiments, the antenna may be used in conjunction with a tuning switch.

The mobile communication module 150 may provide a solution for wireless communication including 2G/3G/4G/5G, etc., applied to the electronic device 100. The mobile communication module 150 may include at least one filter, switch, power amplifier, low noise amplifier (low noise amplifier, LNA), etc.

The wireless communication module 160 may provide solutions for wireless communication including wireless local area network (wireless local area networks, WLAN) (e.g., wi-Fi network), bluetooth (BT), global navigation satellite system (global navigation satellite system, GNSS), frequency modulation (frequency modulation, FM), near field wireless communication technology (near field communication, NFC), infrared technology (IR), etc., as applied on the electronic device 100.

In some embodiments, antenna 1 and mobile communication module 150 of electronic device 100 are coupled, and antenna 2 and wireless communication module 160 are coupled, such that electronic device 100 may communicate with a network and other devices through wireless communication techniques.

The electronic device 100 implements display functions through a GPU, a display screen 194, an application processor, and the like. The GPU is a microprocessor for image processing, and is connected to the display 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. Processor 110 may include one or more GPUs that execute program instructions to generate or change display information.

The display screen 194 is used to display images, videos, and the like. The display 194 includes a display panel. In some embodiments, the electronic device 100 may include 1 or N display screens 194, N being a positive integer greater than 1.

In the embodiment of the present application, the display screen 194 may display a photographing preview interface, a photographing image interface, and the like. It should be noted that, in the embodiment of the present application, the shooting preview interface refers to an interface where a user can view an image collected by the desk lamp camera in real time through the display screen 194.

The external memory interface 120 may be used to connect an external memory card, such as a Micro SD card, to enable expansion of the memory capabilities of the electronic device 100. The external memory card communicates with the processor 110 through an external memory interface 120 to implement data storage functions.

The internal memory 121 may be used to store computer executable program code including instructions. The processor 110 executes various functional applications of the electronic device 100 and data processing by executing instructions stored in the internal memory 121, for example, to cause the electronic device 100 to implement a cooperative method in the embodiments of the present application. The internal memory 121 may include a storage program area and a storage data area. The storage program area may store an application program (such as a sound playing function, an image playing function, etc.) required for at least one function of the operating system, etc. The storage data area may store data created during use of the electronic device 100 (e.g., audio data, phonebook, etc.), and so on. In addition, the internal memory 121 may include a high-speed random access memory, and may further include a nonvolatile memory such as at least one magnetic disk storage device, a flash memory device, a universal flash memory (universal flash storage, UFS), and the like.

The electronic device 100 may implement audio functions through an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, an application processor, and the like. Such as music playing, recording, etc.

The audio module 170 is used to convert digital audio information into an analog audio signal output and also to convert an analog audio input into a digital audio signal. The audio module 170 may also be used to encode and decode audio signals. In some embodiments, the audio module 170 may be disposed in the processor 110, or a portion of the functional modules of the audio module 170 may be disposed in the processor 110.

The speaker 170A, also referred to as a "horn," is used to convert audio electrical signals into sound signals. The electronic device 100 may listen to music, or to hands-free conversations, through the speaker 170A. In some embodiments, the electronic device 100 may be provided with a plurality of speakers 170A.

A receiver 170B, also referred to as a "earpiece", is used to convert the audio electrical signal into a sound signal. When electronic device 100 is answering a telephone call or voice message, voice may be received by placing receiver 170B in close proximity to the human ear.

Microphone 170C, also referred to as a "microphone" or "microphone", is used to convert sound signals into electrical signals. When making a call or transmitting voice information, the user can sound near the microphone 170C through the mouth, inputting a sound signal to the microphone 170C. The electronic device 100 may be provided with at least one microphone 170C. In other embodiments, the electronic device 100 may be provided with two microphones 170C, and may implement a noise reduction function in addition to collecting sound signals. In other embodiments, the electronic device 100 may also be provided with three, four, or more microphones 170C to enable collection of sound signals, noise reduction, identification of sound sources, directional recording functions, etc.

The earphone interface 170D is used to connect a wired earphone. The headset interface 170D may be a USB interface 130 or a 3.5mm open mobile electronic device platform (open mobile terminal platform, OMTP) standard interface, a american cellular telecommunications industry association (cellular telecommunications industry association of the USA, CTIA) standard interface.

The pressure sensor is used for sensing a pressure signal and can convert the pressure signal into an electric signal. In some embodiments, the pressure sensor may be provided on the display screen 194. The electronic device 100 may also calculate the location of the touch based on the detection signal of the pressure sensor.

Touch sensors, also known as "touch panels". The touch sensor may be disposed on the display screen 194, and the touch sensor and the display screen 194 form a touch screen, which is also referred to as a "touch screen". The touch sensor is used to detect a touch operation acting on or near it. The touch sensor may communicate the detected touch operation to the application processor to determine the touch event type.

The keys 190 include a power-on key (or power key), a volume key, etc. The keys 190 may be mechanical keys or touch keys. The electronic device 100 may receive key inputs, generating key signal inputs related to user settings and function controls of the electronic device 100.

The motor 191 may generate a vibration cue. The motor 191 may be used for incoming call vibration alerting as well as for touch vibration feedback. For example, touch operations acting on different applications (e.g., photographing, audio playing, etc.) may correspond to different vibration feedback effects.

The indicator 192 may be an indicator light, may be used to indicate a state of charge, a change in charge, a message indicating a missed call, a notification, etc.

The software system of the electronic device 100 may employ a layered architecture, an event driven architecture, a microkernel architecture, a microservice architecture, or a cloud architecture. In this embodiment, taking an Android system with a layered architecture as an example, a software structure of the electronic device 100 is illustrated.

Fig. 2b is a software architecture block diagram of the electronic device 100 according to an embodiment of the present application.

The layered architecture of the electronic device 100 divides the software into several layers, each with a distinct role and division of labor. The layers communicate with each other through a software interface. In some embodiments, the Android system is divided into four layers, from top to bottom, an application layer, an application framework layer, a system layer, a HAL layer (Hardware Abstract Layer, hardware abstraction layer), and a kernel layer, respectively.

The application layer may include a series of application packages.

As shown in fig. 2b, the application packages may include conversations, video, bluetooth, camera, WLAN, educational applications, device manager applications, and the like. The application packages may also include calendar, map, navigation, music, short messages, etc. applications.

Among other things, the educational application may be used to provide online educational functions for the user, such as online word recognition, online reading aloud, online submission jobs, and the like.

In some examples, a device manager application may be used to bind IOT (Internet of Things ) devices such as desk lamps. In some examples, the educational application may enable binding of IOT (Internet of Things ) devices such as desk lamps.

The application framework layer provides an application programming interface (application programming interface, API) and programming framework for application programs of the application layer. The application framework layer includes a number of predefined functions.

As shown in fig. 2b, the application framework layer may include a camera service, an authentication service, a hardware virtualization service, a device management service, a transmission management service, and the like.

Among other things, a camera service (camera service) may be used to invoke a camera (including a front-facing camera and/or a rear-facing camera) in response to a request of an application.

In the embodiment of the application, the camera service may be used for calling the virtual camera at the electronic device side, that is, calling the camera in the IOT device, in response to a request of the application.

Authentication services are used to provide secure rights management capabilities.

A hardware virtualization service may be used to establish a logical channel between the electronic device side (i.e., the center device side) and the IOT device side, providing the ability to virtualize a camera.

The device management service can be used for discovering and managing the IOT devices and providing far-field (i.e. cloud) IOT device information and near-field (i.e. near-field connectable) IOT device information for application programs such as education applications.

The transmission management service can be used for establishing a physical transmission channel and providing data transmission capability.

In addition, a window manager, a content provider, a view system, a phone manager, a resource manager, a notification manager, etc. may be included.

The window manager is used for managing window programs. The window manager can acquire the size of the display screen, judge whether a status bar exists, lock the screen, intercept the screen and the like.

The content provider is used to store and retrieve data and make such data accessible to applications. The data may include video, images, audio, calls made and received, browsing history and bookmarks, phonebooks, etc. The view system includes visual controls, such as controls to display text, controls to display pictures, and the like. The view system may be used to build applications. The display interface may be composed of one or more views. For example, a display interface including a text message notification icon may include a view displaying text and a view displaying a picture. The telephony manager is used to provide the communication functions of the electronic device 100. Such as the management of call status (including on, hung-up, etc.). The resource manager provides various resources for the application program, such as localization strings, icons, pictures, layout files, video files, and the like. The notification manager allows the application to display notification information in a status bar, can be used to communicate notification type messages, can automatically disappear after a short dwell, and does not require user interaction.

The system library and Runtime layer (i.e., system layer) includes a system library and Android Runtime (Android run time).

Android run time (Android Runtime) includes a core library and virtual machines. Android run is responsible for scheduling and management of the Android system.

The core library consists of two parts: one part is a function which needs to be called by java language, and the other part is a core library of android. The application layer and the application framework layer run in a virtual machine. The virtual machine executes java files of the application program layer and the application program framework layer as binary files. The virtual machine is used for executing the functions of object life cycle management, stack management, thread management, security and exception management, garbage collection and the like.

In the embodiment of the application, the Android run time further comprises a virtual camera adaptation layer, and the virtual camera registration capability is provided.

The system library in the system layer may include a plurality of functional modules. For example: multimedia platform, graphic image processing library, codec, etc.

The multimedia platform can be used for managing multimedia and supporting various common audio, video format playback and recording, still image files and the like. The multimedia platform may support a variety of audio and video coding formats, such as: MPEG4, h.264, MP3, AAC, AMR, JPG, PNG, etc.

Graphics image processing libraries may be used to implement graphics drawing, image rendering, compositing, and layer processing, among others.

The codec may be used to implement codec operations on audio data, video data.

The HAL layer is an interface layer between the operating system kernel and the hardware circuitry. HAL layers include, but are not limited to: audio HAL, sensor HAL, modem HAL, camera HAL, virtual camera HAL.

Wherein the audio HAL is used for processing the audio stream, for example, noise reduction, directional enhancement, etc. of the audio stream. The camera HAL is used for processing the image stream corresponding to the camera at the electronic equipment side, and the virtual camera HAL is used for processing the image stream corresponding to the virtual camera registered at the electronic equipment side, namely, the image stream acquired by the camera at the IOT equipment side.

The kernel layer is a layer between hardware and software. The kernel layer at least comprises a display driver, a camera area, an audio driver, a network driver (such as Wi-Fi driver), a CPU driver, a USB driver, a storage driver, a print driver, and the like. The hardware at least comprises a processor, a display screen, a Wi-Fi module and the like.

It will be appreciated that the layers and components contained in the layers in the software structure shown in fig. 2b do not constitute a specific limitation of the electronic device 100. In other embodiments of the present application, electronic device 100 may include more or fewer layers than shown, and more or fewer components may be included in each layer, as the present application is not limited.

Fig. 3a is a schematic diagram of a hardware structure of the internet of things device 200. It should be noted that the schematic structural diagram of the internet of things device 200 may be applicable to the desk lamp in fig. 1a to 1 b. It should be understood that the internet of things device 200 shown in fig. 3a is only one example of an electronic device, and that the internet of things device 200 may have more or fewer components than shown in the figures, may combine two or more components, or may have different configurations of components. The various components shown in fig. 3a may be implemented in hardware, software, or a combination of hardware and software, including one or more signal processing and/or application specific integrated circuits.

The internet of things device 200 may include: processor 210, camera 201, wireless communication module 202, memory 203, audio module 204, usb interface 205, charge management module 206, power management module 207, battery 208, lighting device 209, keys 211, etc.

Processor 210 may include one or more processing units such as, for example: the processor 210 may include a GPU, ISP, controller, memory, video codec, etc. Wherein the different processing units may be separate devices or may be integrated in one or more processors.

The controller may be a neural hub and a command center of the internet of things device 200. The controller can generate operation control signals according to the instruction operation codes and the time sequence signals to finish the control of instruction fetching and instruction execution.

The camera 201 is used to capture still images or video. The object generates an optical image through the lens and projects the optical image onto the photosensitive element. The photosensitive element may be a charge coupled device (charge coupled device, CCD) or a Complementary Metal Oxide Semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, which is then transferred to the ISP to be converted into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. Taking a desk lamp as an example, the camera 201 may be disposed on a desk lamp stand for capturing images downwards.

The internet of things device 200 may implement a photographing function through an ISP, a camera 201, a video codec, a GPU, etc.

The ISP is used to process the data fed back by the camera 201. For example, when photographing, the shutter is opened, light is transmitted to the camera photosensitive element through the lens, the optical signal is converted into an electric signal, and the camera photosensitive element transmits the electric signal to the ISP for processing and is converted into an image visible to naked eyes. ISP can also optimize the noise, brightness and skin color of the image. The ISP can also optimize parameters such as exposure, color temperature and the like of a shooting scene. In some embodiments, an ISP may be provided in the camera 201.

The wireless communication module 202 may provide solutions for wireless communication including WLAN (e.g., wi-Fi network), bluetooth (BT), etc. for use on the internet of things device 200. In some embodiments, the antenna of the internet of things device 200 and the wireless communication module 202 are coupled such that the internet of things device 200 can communicate with a network and other devices through wireless communication techniques.

Memory 203 may be used to store computer executable program code that includes instructions. The processor 210 executes the instructions stored in the memory 203, thereby performing various functional applications and data processing of the internet of things device 200, for example, to enable the internet of things device 200 to implement the cooperative working method in the embodiments of the present application.

The internet of things device 200 may implement audio functions, such as music playing, etc., through the audio module 204, the speaker 212, etc.

The USB interface 205 is an interface conforming to the USB standard specification, and may specifically be a Mini USB interface, a Micro USB interface, a USB Type C interface, or the like. The USB interface 205 may be used to connect a charger to charge the internet of things device 200, or may be used to transfer data between the internet of things device 200 and a peripheral device.

The charge management module 206 is configured to receive a charge input from a charger. The charging management module 206 may also supply power to the internet of things device 200 through the power management module 207 while charging the battery 208.

The power management module 207 is used to connect the battery 208, the charge management module 206 and the processor 210. The power management module 207 receives input from the battery 208 and/or the charge management module 206 and provides power to the processor 210, the memory 203, the camera 201, the wireless communication module 202, the lighting device 209, and the like.

The keys 211 include a power-on key (or power key), and the like.

The software system of the internet of things device 200 may employ a layered architecture or other architecture, etc. The embodiment of the application takes a layered architecture as an example, and illustrates a software structure of the internet of things device 200.

Fig. 3b is a software architecture block diagram of the internet of things device 200 according to an embodiment of the present application.

The layered architecture of the internet of things device 200 divides the software into several layers, each of which has a distinct role and division of labor. The layers communicate with each other through a software interface. In some embodiments, the system of the internet of things device 200 is divided into three layers, from top to bottom, an application layer, an application framework layer, a system layer, and a kernel layer, respectively.

As shown in fig. 3b, the application layer may include a device application service, which may be understood as a system level application, and the device application service is started after the system of the internet of things device 200 is started.

As shown in fig. 3b, the application framework layer may include device interconnection services, hardware abstraction services, resource managers, and the like.

The device interconnection service can be used for establishing a physical transmission channel, providing data transmission capability and managing a starting switch of the hardware abstraction service.

The hardware abstraction service may be used to establish a logical channel between the electronic device side (i.e., the hub device side) and the IOT device, provide the ability to virtualize the camera, and provide a camera open interface for the IOT device.

The resource manager may provide various resources to the application.

As shown in fig. 3b, the system layer may include a multimedia platform, a graphic image processing library, a codec, a device adaptation module, and the like.

The multimedia platform can be used for managing multimedia and supporting various common audio, video, still image files and the like. The multimedia platform may support a variety of audio and video coding formats, such as: MPEG4, h.264, MP3, AAC, AMR, JPG, PNG, etc.

The codec may be used to implement codec operations on audio data, video data.

The device adaptation module can realize an interface of hardware abstraction service, can provide device information and capability query, and can also provide functions of executing related operations on the IOT device side, such as functions of opening a camera, photographing, previewing and the like.

It may be appreciated that, in order to implement the cooperative method in the embodiments of the present application, the electronic device 100 and the internet of things device 200 include corresponding hardware and/or software modules that perform the respective functions. The steps of an algorithm for each example described in connection with the embodiments disclosed herein may be embodied in hardware or a combination of hardware and computer software. Whether a function is implemented as hardware or computer software driven hardware depends upon the particular application and design constraints imposed on the solution. Those skilled in the art may implement the described functionality using different approaches for each particular application in conjunction with the embodiments, but such implementation is not to be considered as outside the scope of this application.

FIG. 4a is a schematic diagram showing the interaction of the modules. Referring to fig. 4a, the embodiment of the present application provides a method flow for cooperative work of a tablet and a desk lamp, specifically including:

0. device service initialization phase

S0.1, responding to user operation, starting the device application service of the desk lamp, and loading the device interconnection service.

For example, the user operation may be an operation in which the user turns on the desk lamp power supply. And responding to the user operation, starting the desk lamp system, starting the equipment application service, and loading the equipment interconnection service. The device interconnection service may be used to establish a physical transmission channel between the tablet and the desk lamp, and to provide data transmission capability.

S0.2, loading hardware abstraction service by the equipment interconnection service of the desk lamp.

The device interconnection service may also control the opening of the hardware abstraction service. For example, after the device interconnection service is started, the device interconnection service may load the hardware abstraction service in the form of a plug-in. The hardware abstraction service can be used for establishing a logic channel between the tablet and the desk lamp, providing the capability of virtualizing the camera, and simultaneously providing an open interface of the desk lamp camera.

Referring to the block interaction diagram shown in FIG. 4b, the hardware abstraction service may include at least a base component and a camera component. In the device service initialization stage, the device interconnection service loads the base component first, and initializes the base component. After the basic component is initialized, information interaction can be carried out with the equipment adaptation module of the desk lamp, and equipment information and virtualization capability information are obtained. Exemplary device information includes, but is not limited to, device name, device identification, device type, and the like. Exemplary virtualization capability information includes, but is not limited to, whether a virtualized camera is supported, whether a virtualized microphone is supported, and the like.

The desk lamp has the capability of supporting the virtualized camera, and can be understood that the camera of the desk lamp allows other electronic devices (such as a tablet) to call, namely, the camera of the desk lamp can be understood as a virtual camera of other electronic devices.

After the base component obtains the device information and the capability information of the desk lamp, if the desk lamp has the capability of supporting the virtualized camera, the base component loads the camera component to provide the capability of the virtualized camera. At this time, the base component may prepare for negotiation channel establishment to negotiate network connection related information (including but not limited to IP address and port, etc.) with the tablet to establish the negotiation channel. When the base component performs negotiation channel establishment preparation, a Session service (Session Server) is created, and a Session Name (Session Name) of the Session service is sent to the device interconnection service, so that a negotiation channel is established between the transmission management service on the tablet side and the device interconnection service on the desk lamp side.

1. Device discovery phase

S1.1, responding to user operation, and transmitting a device discovery instruction to a device management service by the education APP of the tablet.

The user operation may be an operation that the user clicks a function option in the education APP that needs to call the virtual camera. For example, the user may click on the operations of the click-to-read function, the word search function, the job function, and the photographing function in the education APP.

The education APP of the tablet receives the user operation, and in response to the operation, transmits a device discovery instruction to the device management service of the tablet. The device discovery instruction is used for indicating to search the IOT device capable of establishing connection with the tablet. By way of example, the device discovery instructions may include, but are not limited to, an instruction type and a device type to be discovered. In this embodiment, the device discovery instruction is specifically configured to find a desk lamp that can establish a connection with the tablet.

S1.2, the device management service in the tablet invokes an authentication service to authenticate the education APP, and an authentication result of the education APP is obtained.

After receiving the device discovery instruction, the device management service can acquire the name (or identification) of the education APP based on the existing mechanism of the android system, and perform APP authentication on the education APP according to the name of the education APP. The device management service can invoke the authentication service to authenticate the education APP so as to obtain an authentication result of the education APP.

In this embodiment, the tablet side application framework layer is further provided with a device management API corresponding to the device management service, and a hardware virtualization API corresponding to the hardware virtualization service. In order to implement the technical solution provided in this embodiment, the education APP needs to register on a related platform (for example, a platform provided by a tablet manufacturer), adapt a framework of a device management service, a hardware virtualization service, and a transmission management service, and apply for rights of a device management API and a hardware virtualization API.

Illustratively, the authentication service accesses an authentication server to authenticate the educational APP through the authentication server, including, but not limited to, whether the authentication is registered on the relevant platform, whether the relevant framework is adapted, and whether the relevant API rights are applied.

Also exemplary, the authentication service may authenticate the educational APP according to a local whitelist.

And obtaining an authentication result (authentication success or authentication failure) of the education APP at the authentication service, and sending the authentication result to the equipment management service.

S1.3, the device management service in the tablet transmits a device search instruction to the transmission management service when the education APP authentication is successful.

If the education APP authentication is successful, the device management service sends a device search instruction to the transmission management service. The device search instruction may include, but is not limited to, an instruction type, a device type to be searched, and a search mode. Exemplary search means include, but are not limited to, near field device scanning and obtaining device information from a cloud server. In this embodiment, the device to be searched is a desk lamp.

S1.4, the transmission management service in the tablet acquires a far-near field device list according to the device search instruction, and sends the far-near field device list to the device management service.

The near-far field device list includes a far-field device list and a near-field device list. The far-field devices included in the far-field device list refer to registered devices acquired from the cloud server, and the near-field devices included in the near-field device list refer to devices scanned through near-field communication. In the far field device list and the near field device list, the device information includes, but is not limited to, a device name, a device identification, a device type, and the like.

When the transmission management service receives the device searching command, performing related device searching operations, such as performing near-field device scanning operations and acquiring related device information in the cloud server, according to the device type to be searched and the searching mode carried in the device searching command, obtaining a far-field device list and a near-field device list, and sending the far-field device list and the near-field device list to the device management service.

And S1.5, performing device filtering by the device management service in the tablet according to the far-near field device list, and reporting the device information obtained after filtering to the education APP.

The device management service performs device filtering according to the far-field device list and the near-field device list, determines desk lamp information which can be linked with the tablet, and sends the desk lamp information to the education APP. The device management service may perform an intersection operation on the far-field device list and the near-field device list, filter out the table lamps only existing in the far-field device list or only in the near-field device list, and use the table lamps existing in the far-field device list and the near-field device list as table lamp information capable of being linked with the tablet. Thus, the device management service can filter out the desk lamp which is not registered in the cloud server, and also can filter out the desk lamp which cannot perform near field communication with the tablet.

In another alternative embodiment, if the tablet and the desk lamp are under the same lan, the transmission management service of the tablet may obtain the communication device list and the registered device list according to the device search instruction. The devices included in the communication device list refer to devices scanned through near field communication or far field communication, and the devices included in the registration device list refer to registered devices acquired from a cloud server. The near field devices included in the near field device list refer to devices scanned by near field communication. In the communication device list and the registered device list, the device information includes, but is not limited to, a device name, a device identification, a device type, and the like.

The transmission management service in the tablet sends the communication equipment list and the registration equipment list to the equipment management service, and the equipment management service performs equipment filtering according to the communication equipment list and the registration equipment list and reports the equipment information obtained after filtering to the education APP. The device management service may perform an intersection operation on the communication device list and the registration device list, filter out the table lamps only existing in the communication device list or only existing in the registration device list, and use the table lamps existing in both the communication device list and the registration device list as table lamp information capable of being linked with the tablet. Thus, the device management service can filter out the desk lamp which is not registered in the cloud server, and also can filter out the desk lamp which cannot perform near field communication with the tablet.

2. Virtual camera enabled phase

S2.1, determining a desk lamp to be linked by the education APP of the flat plate.

The number of the table lamps which can be linked with the tablet, which are filtered by the device management service, can be one or more. When the number of the desk lamps is one, the education APP defaults to be used as the desk lamp to be linked; when the number of the desk lamps is multiple, the education APP can display a desk lamp list to be linked for the user, so that the user can select the desk lamps in the list, and the education APP can respond to the selection operation of the user to select the desk lamps to be linked.

It should be noted that, the step of determining the desk lamp to be linked by the education APP may also be divided into a device discovery stage, which is not limited in this embodiment.

S2.2, performing equipment verification and equipment connection on the desk lamp by the aid of the education APP of the tablet computer, and obtaining that the desk lamp has the capability of supporting a virtualized camera.

S2.3, the education APP of the tablet sends a virtual camera enabling request to the hardware virtualization service.

The education APP of the tablet sends a virtual camera enabling request to the hardware virtualization service after obtaining that the desk lamp has the capability of supporting the virtualized camera. Wherein the virtual camera enable request is for indicating that the virtual camera is registered in the virtual camera HAL. The virtual camera enable request may include, but is not limited to, a request type, a device name, a device identification, a device type, and an identification of the virtual camera.

S2.4, the hardware virtualization service of the tablet registers the virtual camera with the virtual camera HAL.

After receiving the virtual camera enabling request, the hardware virtualization service registers the corresponding virtual camera with the virtual camera HAL according to the virtual camera enabling request.

S2.5, the tablet' S virtual camera HAL sends a virtual camera enable success indication to the educational APP after the virtual camera registration is completed.

The flow of the virtual camera enabled phase is explained in detail below with reference to the schematic interaction diagram of the modules shown in fig. 4 b. Referring to fig. 4b, the flow of the virtual camera enable phase mainly includes a device check sub-phase (S301-S309), a device connection sub-phase (S310-S321), a device service capability request sub-phase (S322-S325), and a virtual camera enable sub-phase (S326-S331).

Referring to fig. 4b, the process of the virtual camera enabling phase specifically includes the following steps:

s301, the education APP in the tablet sends a virtual camera enabling instruction to the hardware virtualization API.

The virtual camera enabling instruction is used for indicating to enable the virtual camera, and the virtual camera enabling instruction can include but is not limited to an instruction type, a device name, a device identifier and a device type.

S302, after receiving the virtual camera enabling instruction, the hardware virtualization API in the tablet sends a device checking instruction to an interface scheduling module of the hardware virtualization service.

And the device verification instruction is used for indicating to verify the device information carried in the virtual camera enabling instruction. The device verification instruction may include, but is not limited to, an instruction type, a device name, a device identifier, and a device type.

S303, an interface scheduling module of the hardware virtualization service in the tablet sends an APP authentication instruction to a right management module of the hardware virtualization service.

After receiving the device checking instruction, the interface scheduling module of the hardware virtualization service firstly sends an APP authentication instruction to the authority management module of the hardware virtualization service so as to authenticate the APP initiating the virtual camera enabling instruction. The APP authentication instruction may include, but is not limited to, a name of the APP.

S304, performing APP authentication on the education APP by the permission management module of the hardware virtualization service in the tablet.

Illustratively, the rights management module may access an authentication server to authenticate the educational APP through the authentication server, including, but not limited to, whether the authentication is registered on the relevant platform, whether the relevant framework is adapted, and whether the relevant API rights are applied. The rights management module may access the authentication server through the authentication service, which is not limited in this embodiment.

S305, when the authority management module of the hardware virtualization service in the tablet is successful in authenticating the education APP, an authentication success indication is sent to the interface scheduling module.

After the authority management module obtains the authentication result of the education APP, if the education APP is successfully authenticated, an authentication success indication is sent to the interface scheduling module, and if the education APP is failed in authentication, an authentication failure indication is sent to the hardware virtualization API, so that the hardware virtualization API returns the indication information of the lack of authority of the APP to the education APP according to the authentication failure indication.

S306, when the interface scheduling module of the hardware virtualization service in the tablet determines that the education APP authentication is successful, a device verification instruction is sent to the device management module.

And the interface scheduling module of the hardware virtualization service receives the authentication success indication, and sends a device verification instruction to the device management module after determining that the education APP is successfully authenticated. The device verification instruction is used for performing state verification on the device to be linked, and is specifically used for performing state verification on the desk lamp to be linked in the embodiment. Exemplary device verification instructions may include, but are not limited to, instruction type, device name, device identification, device type.

S307, the device management module of the hardware virtualization service in the tablet sends a device information inquiry instruction to the device profile module of the device management service.

Wherein, the equipment profile module of the equipment management service stores the equipment information of the current online.

After receiving the device verification instruction, the device management module of the hardware virtualization service sends a device information inquiry instruction to the device profile module of the device management service, wherein the device information inquiry instruction can include, but is not limited to, a device name, a device identifier and a device type.

S308, the device profile module of the device management service in the tablet returns device information to the device management module of the hardware virtualization service.

And the device profile module of the device management service returns the device information to the device management module of the hardware virtualization service if the corresponding device is queried according to the device information query instruction. Among the returned device information may include, but is not limited to, device name, device identification, device type, and presence status.

And the device profile module of the device management service returns a null value to the device management module of the hardware virtualization service according to the device information inquiry instruction if the corresponding device is not inquired so as to indicate that the corresponding device is not inquired. At this time, the device management module of the hardware virtualization service may send a device verification failure indication to the hardware virtualization API, so that the hardware virtualization API returns, to the education APP, indication information of the device verification failure according to the device verification failure indication.

S309, after receiving the device information, the device management module of the hardware virtualization service in the tablet sends a device verification success indication to the hardware virtualization API.

And how the equipment management module of the hardware virtualization service receives the equipment information returned by the equipment profile module of the equipment management service, sending an equipment verification success indication to the hardware virtualization API so as to indicate that the verification of the desk lamp to be linked is successful.

S310, the hardware virtualization API in the tablet sends a device connection request to a device management module of the hardware virtualization service.

After confirming that the verification of the desk lamp to be linked is successful, the hardware virtualization API sends a device connection request to a device management module of the hardware virtualization service. The device connection request is used for indicating that network connection is established with the device to be linked, and in this embodiment, the device connection request is specifically used for indicating that network connection is established with the desk lamp to be linked. The device connection request may include, but is not limited to, a request type, a device name, a device identification, and a device type.

S311, after receiving the device connection request, the device management module of the hardware virtualization service executes a negotiation channel setup preparation operation and sends a negotiation channel opening request to the transmission management service.

After receiving the device connection request, the device management module of the hardware virtualization service prepares to negotiate the channel. When the device management module prepares to negotiate a channel, a Session Server is created, and a Session Name of the Session service is sent to the transmission management service. After the negotiation path is ready, a negotiation path opening request is sent to the transmission management service. The negotiation channel opening request is used for indicating to establish a negotiation channel, and the negotiation channel opening request may include, but is not limited to, a peer device identifier (i.e. a table lamp identifier) and a Session Name.

In this embodiment, the negotiation channel opening request is actively initiated by the tablet side, that is, the tablet needs to establish a connection with the desk lamp. At this time, the desk lamp may be understood as a server, and the tablet may be understood as a client that needs to access the server.

S312, the transmission management service in the tablet establishes a negotiation channel with the device interconnection service in the desk lamp.

After receiving the negotiation channel opening request, the transmission management service interacts with the device interconnection service in the desk lamp according to the Session Name to establish a negotiation channel. Wherein, establishing the negotiation channel can be concretely establishing the session, and determining the session identification.

S313, the device interconnection service in the desk lamp sends a negotiation channel successful establishment instruction to the camera component in the hardware abstraction service.

After the establishment of the negotiation channel is completed, the device interconnection service in the desk lamp sends an indication of successful establishment of the negotiation channel to the camera component in the hardware abstraction service so as to indicate that the establishment of the negotiation channel is completed and that the devices needing to establish connection currently exist. The negotiation channel successful establishment indication may include, but is not limited to, device information (i.e., tablet device information) that needs to establish a connection, and a session identifier.

S314, the transmission management service in the tablet sends a negotiation channel successful establishment instruction to the device management module of the hardware virtualization service.

After the negotiation channel is established, the transmission management service in the tablet sends a negotiation channel successful establishment instruction to the device management module of the hardware virtualization service so as to indicate that the establishment of the negotiation channel is completed and that the device needing to establish the connection exists currently. The negotiation channel successful establishment indication may include, but is not limited to, device information (i.e., desk lamp device information) that needs to establish a connection, and a session identifier.

The present embodiment does not limit the execution order of S313 and S314.

S315, the device management module of the hardware virtualization service in the tablet sends a device information negotiation request to the camera component of the hardware abstraction service in the desk lamp based on the negotiation channel.

The device negotiation request may include, but is not limited to, device information (such as a device name, a device identifier, a device type, etc.) and a control channel connection request.

S316, after the camera component of the hardware abstraction service in the desk lamp receives the device negotiation request, a control channel is prepared, and device negotiation information is returned to the device management module of the hardware virtualization service in the tablet.

After receiving the equipment negotiation request, the camera component of the hardware abstraction service in the desk lamp analyzes the equipment negotiation request to acquire the equipment information of the opposite end, determines an IP address and a port to be monitored according to the control channel connection request, adds the IP address and the port into the equipment negotiation information, and returns the equipment negotiation information to the equipment management module of the hardware virtualization service in the tablet.

It is noted that the device negotiation request and the device negotiation information are transmitted based on the established negotiation channel.

S317, after the device management module of the hardware virtualization service in the tablet receives the returned device negotiation information, the negotiation channel is closed.

Wherein closing the negotiation channel may specifically be closing the session. After the device management module of the hardware virtualization service in the tablet receives the returned device negotiation information, the session ends, and the device management module of the hardware virtualization service can close the corresponding session according to the session identifier.

S318, the device management module of the hardware virtualization service in the tablet sends a control channel opening request to the transmission management service.

And the control channel opening request is used for indicating to establish network communication connection with the desk lamp. The control channel opening request may include, but is not limited to, a communication protocol, a source IP, a source port, a destination IP, and a destination port, where the destination IP and the destination port are an IP and a port monitored by a camera component of a hardware abstraction service in the desk lamp.

S319, the transmission management service in the tablet is connected with a control channel of a camera component of the hardware abstraction service in the desk lamp, and a successful connection instruction of the control channel is sent to a device management module of the hardware virtualization service in the tablet.

The transmission management service in the tablet receives the control channel opening request, and establishes control channel connection with the table lamp according to the information carried by the control channel opening request, namely, establishes network communication connection between the tablet and the table lamp. Furthermore, the device management module of the hardware virtualization service in the tablet and the camera component of the hardware abstraction service in the desk lamp can perform network communication based on the control channel.

After the control channel is successfully established, the transmission management service in the tablet sends a control channel successful connection indication to the device management module of the hardware virtualization service in the tablet. The indication of successful connection of the control channel may include, but is not limited to, a connection success identifier and information related to the control channel.

S321, the device management module of the hardware virtualization service in the tablet sends a device connection success indication to the hardware virtualization API.

The device connection success indication may include, but is not limited to, a connection success identifier and connected device information.

S322, the hardware virtualization API in the tablet sends a device capability request to a device management module of the hardware virtualization service.

After receiving the device connection success indication, the hardware virtualization API sends a device capability request to a device management module of the hardware virtualization service. The device capability request may be used to request to obtain virtualization capability information of the peer device (i.e., the desk lamp). Exemplary virtualized device capability information includes, but is not limited to, whether a virtualized camera is supported, whether a virtualized microphone is supported, and the like.

S323, the device management module of the hardware virtualization service in the tablet sends a device capability request to the camera component of the hardware abstraction service in the desk lamp in the control channel.

S324, the camera component of the hardware abstraction service in the desk lamp returns the device capability information to the device management module of the hardware virtualization service in the tablet in the control channel.

In this embodiment, the returned device capability information of the desk lamp may include at least the capability of supporting the virtualized camera and the camera identifier of the desk lamp.

S325, the device management module of the hardware virtualization service in the tablet sends device capability information to the hardware virtualization API.

The device management module of the hardware virtualization service in the tablet sends the received device capability information to the hardware virtualization API so that the hardware virtualization API can know whether the desk lamp has the capability of supporting the virtualized camera.

S326, the hardware virtualization API in the tablet sends a virtual camera enabling request to the device management module of the hardware virtualization service.

The hardware virtualization API in the tablet knows that the desk lamp has the capability of supporting the virtualized camera and sends a virtual camera enabling request to the device management module of the hardware virtualization service. The virtual camera enabling request may include, but is not limited to, a request type and a camera identifier of a desk lamp.

S327, the device management module of the hardware virtualization service in the tablet registers the virtual camera in the virtual camera HAL.

After receiving the virtual camera enabling request, the device management module of the hardware virtualization service sends a virtual camera registration request to the virtual camera HAL. The virtual camera registration request may include, but is not limited to, a request type and a camera identification of the desk lamp. After receiving the virtual camera registration request, the virtual camera HAL registers a virtual camera driver for the camera of the desk lamp in the virtual camera HAL, assigns a camera ID (i.e. virtual camera ID) for the camera of the desk lamp, and registers the camera ID in the system. Thus, the mapping relation between the desk lamp camera and the virtual camera is established in the virtual camera HAL.

S328, the device management module of the hardware virtualization service in the tablet sends a service state update indication to the camera component of the hardware abstraction service in the desk lamp.

The service state update instruction is used for instructing a camera component of the hardware abstraction service in the desk lamp to update the virtualized service state of the camera component. The virtualized service state may include an occupied state, an unoccupied state, or may include a registered state, an unregistered state, among others. Exemplary service status update indications may include, but are not limited to, device information of a peer device (i.e., a desk lamp), a hardware identifier (e.g., a desk lamp camera identifier), and a virtualized service status corresponding to the hardware identifier.

S329, the camera component of the hardware abstraction service in the desk lamp updates the service state according to the service state update instruction.

When the virtualized service state corresponding to the desk lamp camera is indicated to be the occupied state (or called registration state) in the service state update indication, the camera component updates the virtualized service state corresponding to the desk lamp camera to be the occupied state (or called registration state).

S330, the device management module of the hardware virtualization service in the tablet sends a virtual camera enabling success indication to the hardware virtualization API.

The virtual camera enabling success indication may include, but is not limited to, an enabling success identifier (or called a virtualization success identifier), a camera identifier of the desk lamp, and a camera ID corresponding to the virtual camera (or called a camera ID corresponding to the desk lamp camera).

The present embodiment does not limit the execution order of S328 and S330.

S331, the hardware virtualization API in the tablet sends a virtual camera enabling success indication to the education APP.

3. Virtual camera preview access phase

S3.1, the hardware virtualization API in the tablet sends a virtual camera access instruction to the camera service.

Virtual camera access instructions refer to instructions for invoking a virtual camera. The virtual camera access instruction may include, but is not limited to, an instruction type, a virtual camera ID, and a camera configuration parameter, where the configuration parameter includes, but is not limited to, a camera resolution and an acquisition frame rate.

S3.2, the camera service in the tablet sends an image preview request to the virtual camera HAL according to the virtual camera access instruction.

After receiving the virtual camera access instruction, the camera service generates a corresponding image preview request according to the virtual camera ID and sends the corresponding image preview request to the virtual camera HAL. Wherein the image preview request is for requesting a preview image data stream. Illustratively, the image preview request may include, but is not limited to, a request identification, a virtual camera ID, camera configuration parameters, and the like.

S3.3, the virtual camera HAL in the tablet sends an image preview request to the hardware virtualization service.

After receiving the image preview request, the virtual camera HAL determines the matched virtualized hardware identification according to the virtual camera ID carried in the image request. In this embodiment, the virtual camera HAL determines the linked desk lamp camera according to the virtual camera ID and the mapping relationship between the virtual camera ID and the desk lamp camera, and generates a corresponding image request according to the determined virtualized hardware identifier, and sends the image request to the hardware virtualized service. By way of example, the image preview request may include, but is not limited to, a request identifier, device information (i.e., table lamp information), virtualized hardware identifier (i.e., table lamp camera identifier), camera configuration parameters, and the like.

And S3.4, the hardware virtualization service in the tablet sends an image preview request to the transmission management service.

The hardware virtualization service sends an image preview request to the transport management service. The image preview request may include, but is not limited to, a request identifier, device information (i.e., table lamp information), a virtualized hardware identifier (i.e., table lamp camera identifier), and a camera configuration parameter.

When the hardware virtualization service in the tablet sends an image preview request to the transmission management service, if the hardware virtualization service finds that the data channel is not established with the desk lamp, the hardware virtualization service generates a data channel establishment request and sends the data channel establishment request to the transmission management service. And the data channel establishment request is used for indicating the transmission of data with the desk lamp. The data channel establishment request may include, but is not limited to, session identifier, connection information, data codec mode, etc.

The transmission management service in the flat plate receives the data channel establishment request, and establishes data channel connection with the desk lamp according to the information carried by the data channel establishment request, namely, establishes a data channel between the flat plate and the desk lamp. Further, the in-tablet transmission management service and the in-desk device interconnection service may transmit various data including, but not limited to, image data based on the data channel.

After the data channel is successfully established, the in-plane transmission management service sends a data channel successful connection indication to the in-plane hardware virtualization service, and the in-desk lamp device interconnection service sends a data channel successful connection indication to the camera component in the hardware abstraction service. The indication of successful connection of the data channel may include, but is not limited to, a connection success identifier and information related to the data channel.

And S3.5, the transmission management service in the tablet transmits an image preview request to the device interconnection service of the desk lamp.

And the transmission management service in the tablet determines a corresponding control channel according to the equipment information carried in the image preview request, and transmits the image preview request to the equipment interconnection service of the desk lamp in the control channel.

S3.6, the device interconnection service in the desk lamp sends an image preview request to the camera driver.

After receiving the image preview request, the device interconnection service in the desk lamp determines a hardware driver (in this embodiment, determines a camera driver) according to the virtualized hardware identifier, and sends the corresponding image preview request to the camera driver.

S3.7, the camera in the desk lamp drives the camera to collect images, and preview image data are transmitted to the hardware virtualization service of the tablet through the data channel.

The camera drives the camera to start, and drives the camera to collect images according to camera configuration parameters carried in the image preview request, so as to obtain a preview image data stream, and the preview image data stream is sent to the device interconnection service by the hardware abstraction service, so that the device interconnection service continuously transmits the preview image data stream to the hardware virtualization service of the tablet in the data channel. The packetization and encoding and decoding processes of the preview image data stream are not described herein.

S3.8, the in-panel hardware virtualization service sends the preview image data to the virtual camera HAL.

The hardware virtualization service continues to receive the preview image data stream and sends the preview image data stream to the virtual camera HAL.

S3.9, the virtual camera HAL in the tablet sends the preview image data to the camera service.

At this time, the virtual camera HAL continuously acquires preview image data acquired by the desk lamp camera, and continuously transmits the preview image data to the camera service.

And S3.10, the camera service in the tablet sends the preview image data to the education APP.

S3.11, displaying preview images by the education APP in the tablet.

After the education APP receives the preview image data stream through the camera service, the preview image can be displayed in the corresponding interface.

4. Virtual camera photographing stage

S4.1, responding to the received user operation, and sending a photographing request to the hardware virtualization service by the education APP in the tablet.

The user operation may be, for example, an operation of clicking a photographing option. In response to the received user operation, the education APP in the tablet sends a photographing request to the hardware virtualization service. The photographing request may include, but is not limited to, a photographing image sequence number, device information (i.e., table lamp information), a virtualized hardware identifier (i.e., table lamp camera identifier), a camera configuration parameter, and the like. Camera configuration parameters include, but are not limited to, image resolution.

The photographing request can also carry a task identifier so as to ensure orderly management of multiple photographing tasks.

S4.2, the hardware virtualization service in the tablet sends a photographing request to the transmission management service.

S4.3, the transmission management service in the tablet transmits a photographing request to the device interconnection service of the desk lamp.

S4.4, the equipment interconnection service in the desk lamp sends a photographing request to the camera driver.

After receiving the image preview request, the device interconnection service in the desk lamp determines a hardware driver for response (in this embodiment, determines a camera driver) according to the virtualized hardware identifier, and sends a corresponding photographing request to the camera driver.

S4.5, the camera in the desk lamp drives the camera to shoot an image, and shot image data is transmitted to the hardware virtualization service of the tablet through the data channel.

The camera drives the camera to acquire images according to camera configuration parameters carried in the photographing request, photographing image data are obtained, and the photographing image data are sent to the equipment interconnection service through the hardware abstraction service, so that the equipment interconnection service continuously transmits the photographing image data to the hardware virtualization service of the tablet in the data channel. The packetization and encoding and decoding processes of the captured image data are not described herein.

And S4.6, the hardware virtualization service in the tablet sends the shot image data to the education APP.

S4.7, displaying the shot image by the education APP in the plate.

After the education APP receives the shot image through the hardware virtualization service, the shot image can be displayed in the corresponding interface.

In this embodiment, the virtual camera preview access stage is implemented based on the android native camera frame, and the virtual camera photographing stage is implemented based on the private virtualized camera frame, so that the processing path involved in the virtual camera photographing stage is shorter, and the photographing delay is smaller. Meanwhile, because the image preview is realized based on the camera frame of the android native, the education APP is less in change of the technical scheme provided by the embodiment for adapting.

It should be noted that the above-mentioned phase division in the flow is merely an exemplary expression, and the embodiment of the present application is not limited thereto. In addition, after the preview image is displayed on the process panel in the virtual camera preview access stage, the process of displaying the preview image in real time and the process in the virtual camera photographing stage can be simultaneously performed. Where the above processes are not explained in detail, reference may be made to the prior art, and they are not described in detail herein.

Fig. 4a shows a communication architecture of the collaborative system, which is used to complete the management of the tablet to the virtual camera (i.e. the desk lamp camera), the control command interaction between the tablet and the desk lamp, and the return and processing of the image data.

It should be noted that, instructions, requests, etc. transmitted across devices (i.e. transmitted between the tablet and the desk lamp) need to be encapsulated based on a communication protocol and a parameter sequence, etc., which will not be described in detail in this embodiment. The hardware virtualization service in the tablet may also manage the life cycle of previewing image streams and capturing images by dynamically allocating memory and dynamically destroying memory.

In addition, it should be noted that, before executing the cooperative working method provided in this embodiment, the education APP needs to bind with the desk lamp and register the desk lamp in the cloud server.

The embodiment of the application provides a frame scheme that android system equipment uses the camera of external equipment to take a picture, and this scheme not only can be applied in educational scene, is applicable to other equipment that is provided with the camera simultaneously, and these equipment can share its camera ability and give android system equipment such as cell-phone, tablet to realize android system equipment and establish the interconnection intercommunication of these equipment.

Fig. 5 a-5 b illustrate an application scenario. As shown in fig. 5a (1), the interface 401 is displayed on a flat panel display, and a plurality of application icons are displayed on the interface 401, and the user clicks the educational application icon 4011. In response to the received user operation, the tablet opens the educational application, and the tablet displays an educational application interface, which may be referred to as (2) in fig. 5 a. As shown in fig. 5a (2), the educational application interface 402 is displayed on the panel, and various function options of the educational application are displayed on the interface 402, including but not limited to a word search function, a click-to-read function, a job function, a photo function, etc. With the user using the photographing function capability, the user clicks the photographing function option 4021, and in response to the user operation, the tablet executes the flows of the device discovery phase and the virtual camera service enabling phase and the virtual camera preview access phase.

In the device discovery stage, if the number of the table lamps which can be linked with the tablet is one and obtained by filtering by the device management service of the tablet, the tablet automatically executes the processes of the virtual camera service enabling stage and the virtual camera preview access stage, and displays an interface shown in (1) in fig. 5b, for example. In the device discovery stage, if the number of the table lamps which can be linked with the tablet is a plurality of the table lamps obtained through filtering by the device management service of the tablet, the tablet displays a table lamp selection interface. For example, a list of the desklamps to be linked is displayed on the desk lamp selection interface, the user may perform a selection operation, and in response to the selection operation by the user, the educational application determines one desk lamp to be linked and continues to perform the processes of the virtual camera service enabling phase and the virtual camera preview access phase to display the interface shown in (1) of fig. 5b, for example.

As shown in fig. 5b (1), an image preview window 4031 and a photographing option 4032 are displayed in the interface 403, and a preview image acquired by the desk lamp camera in real time is displayed in the image preview window 4031. At this time, if the user clicks the photographing option 4032, the tablet performs a process of the photographing stage of the virtual camera in response to the user operation, and displays an interface shown in (2) of fig. 5b, for example. As shown in an interface 404 of fig. 5b (2), an image captured by the desk lamp camera is displayed in an image preview window 4041. At this time, when the user clicks the confirm option 4041, the tablet saves the captured image in response to the user operation, and continues to display the preview interface shown in fig. 5b (1), for example. If the user clicks the cancel option 4042, the tablet may display a preview interface, for example, as shown in (1) of fig. 5b, in response to a user operation.

It should be noted that, the interface shown in fig. 5b (2) is merely an exemplary example, and the image captured by the desk lamp camera may be displayed in other areas of the interface instead of the image preview window 4041, and the preview image captured by the desk lamp camera in real time is continuously displayed in the image preview window 4041, which is not limited in this application.

The cooperative working method provided by the embodiment mainly explains a low-cost technical scheme for realizing the online education function based on the combination of the tablet equipment and the desk lamp equipment. The technical solution provided in this embodiment is described below in connection with several different functions related to online education, respectively.

Scene one

Referring to the application scenario schematic diagram shown in fig. 1a, the technical scheme is illustrated by taking a word searching function as an example. When a student encounters an unrecognized word, the finger can point to the lower part of the word, a picture is shot by using the desk lamp camera, the image is identified by the flat plate to determine the content of the word, and the meaning of the word is fed back to the student through the display screen after the online word searching is completed, for example, the word explanation is displayed and broadcasted on an interface.

FIG. 6a is a schematic diagram showing the interaction of the modules. Referring to fig. 6a, the embodiment of the present application provides a method flow for cooperative work of a tablet and a desk lamp, specifically including:

S501, responding to the operation of clicking the word searching function by a user, executing the processes of a device discovery stage, a virtual camera enabling stage and a virtual camera preview access stage by the tablet and the table lamp, and displaying a preview interface by the tablet.

For the flow of the device discovery phase, the virtual camera enabling phase, and the virtual camera preview access phase, reference may be made to the foregoing, and details thereof will not be repeated herein.

It should be noted that in the virtual camera preview access phase, the hardware virtualization API in the tablet will carry the camera configuration parameters in sending the virtual camera access instruction to the camera service. The camera configuration parameters may include, but are not limited to, image resolution and image acquisition frame rate, among others. The desk lamp camera is set according to the received configuration parameters, and the preview image data is acquired according to the corresponding image resolution and image acquisition frame rate.

In this scenario, the tablet needs to accurately identify the preview image to determine the text content pointed by the user, so that the image quality requirement of the present scenario on the preview image is high, for example, the image resolution may be set to 1080P. Thus, a higher success rate of word searching can be ensured.

S502, the education APP in the tablet performs finger recognition on the preview image.

The education APP may perform finger recognition for each received preview image, or may perform finger recognition for the latest received preview image at regular time, which is not limited in this embodiment.

For example, the education APP may integrate an image recognition algorithm to implement an image recognition operation, and the education APP may also call an image recognition service to perform the image recognition operation, which is not limited in this embodiment.

The image recognition algorithm may refer to the prior art, and this embodiment is not described herein.

S503, in response to the operation of using finger words by the user, the education APP in the tablet identifies the fingers of the user in the preview image.

When a user points to a certain word in the book by using a finger, the desk lamp camera can acquire a preview image of the finger word, and then the education APP in the tablet can identify the finger of the user in the preview image.

When the education APP carries out finger recognition on the preview image, if a finger is recognized, position information, such as coordinate information, of the finger in the preview image can be obtained.

S504, the education APP in the tablet determines an ROI (region of interest ) image according to the position of the finger in the preview image.

After the educational APP identifies the user's finger in the preview image, the ROI image may be determined from the finger's position information in the preview image. The education APP can determine the ROI information according to the coordinate information of the finger in the preview image, wherein the ROI information comprises, but is not limited to, the coordinates of a central point and the region range (such as width and height information). Furthermore, the educational APP can crop out the ROI image in the preview image based on the ROI information.

S505, education APP in the plate accurately identifies the ROI image and determines the word to be interpreted.

The education APP may integrate an image recognition algorithm to accurately recognize the ROI image, and may call an image recognition service to accurately recognize the ROI image to determine the word to be interpreted, which is not limited in this embodiment.

S506, the education APP in the tablet performs word searching operation on the word to be interpreted, and displays the paraphrasing of the word to be interpreted.

After determining the word to be interpreted, the education APP can perform online word searching operation or perform word searching operation in a database to obtain the paraphrasing of the word to be interpreted. Furthermore, the educational APP can display the paraphrasing of the new word to be interpreted for the user to view. In addition, the education APP may read the paraphrasing of the displayed new word, which is not limited in this embodiment.

Similarly, the user may perform the pointing operation using a pointing tool (or pointing tool) such as a stylus, which is not limited in this embodiment. Correspondingly, the education APP identifies the word pointing tool on the preview image to determine whether the user has word searching intention, and determines the ROI image according to the position information of the word pointing tool on the preview image.

Similarly, the user may also point to a picture in the book using a pointing device such as a finger or stylus. Correspondingly, the education APP determines the ROI image according to the position information of the finger or the finger-word tool such as the touch pen, performs picture content identification on the ROI image, displays the paraphrasing corresponding to the picture, and can read the displayed paraphrasing of the picture. In this case, the present embodiment will not be described in detail.

Fig. 1a, 7 a-7 b show an exemplary application scenario. As shown in fig. 7a (1), the tablet display education APP interface 701 displays various function options of the education application on the interface 701, including but not limited to a word search function, a click-to-read function, a job function, a photo function, etc. The user clicks the word search function option 7011, and in response to the user operation, the tablet executes the flows of the device discovery phase and the virtual camera service enabling phase, and the virtual camera preview access phase.

In the device discovery stage, if the number of the table lamps which can be linked with the tablet is one and obtained by filtering by the device management service of the tablet, the tablet automatically executes the processes of the virtual camera service enabling stage and the virtual camera preview access stage, and displays an interface shown in (2) in fig. 7a, for example. In the device discovery stage, if the number of the table lamps which can be linked with the tablet is a plurality of the table lamps obtained through filtering by the device management service of the tablet, the tablet displays a table lamp selection interface. For example, a list of the desklamps to be linked is displayed on the desk lamp selection interface, the user may perform a selection operation, and in response to the selection operation by the user, the educational application determines one desk lamp to be linked and continues to perform the processes of the virtual camera service enabling phase and the virtual camera preview access phase to display the interface shown in (2) of fig. 7a, for example.

As shown in fig. 7a (2), an image preview window 7021 and a word searching function operation diagram 7022 are displayed in the interface 702. The preview window 7021 displays a preview image acquired by the desk lamp camera in real time. The user may perform a word or figure pointing operation with reference to the word searching function operation diagram 7022 to trigger the word searching function. The education APP carries out word or figure recognition according to the preview image. With continued reference to fig. 7a (2), when the user uses a finger in a book, the desk lamp camera captures a finger preview image, which is displayed in the image preview window 7021. Further, the educational APP can identify the user's finger in the preview image and determine the position information of the finger in the preview image, such as coordinate information, etc. The education APP determines an ROI image according to the position of the finger in the preview image, and accurately identifies the ROI image to determine the word to be interpreted. After the education APP queries the paraphrasing of the new words, the corresponding paraphrasing of the new words is displayed on the interface, and the education APP can refer to FIG. 7 b.

However, in the above procedure, in order to ensure the success rate of word searching, the table lamp camera needs to continuously return to the preview image stream with high resolution (for example 1080 p), and the requirement on bandwidth is relatively high, and a bandwidth of 4-8Mbps is required. Therefore, the requirements of the desk lamp on the hardware chip are higher, so that the cost of the desk lamp can be increased.

In order to realize the scheme of realizing the word searching function by the cooperation of the flat plate and the desk lamp and reduce the hardware cost of the desk lamp, the embodiment also provides a technical scheme. In this case, since finger recognition or word recognition is performed in the image and the requirement of position recognition on the image resolution is not high, a low-resolution (for example, 480 p) preview image stream is continuously returned by using a desk-lamp camera in the virtual camera preview access stage. If the education APP identifies the user's word or figure operation based on the preview image, triggering the desk lamp to shoot a high-resolution (1080 p for example) image, so that the education APP can accurately identify and determine the word or picture to be interpreted according to the high-resolution image. Therefore, the desk lamp camera is used for continuously returning the low-resolution preview image stream, the bandwidth requirement is not high, only 0.5-1Mbps bandwidth is needed, and only more bandwidth is needed when the high-resolution image is transmitted. Therefore, the technical scheme not only reduces the requirement of the desk lamp on the hardware chip and reduces the cost of the desk lamp, but also can ensure the success rate of word searching.

Fig. 6b shows an interaction diagram of the modules. Referring to fig. 6b, the embodiment of the present application provides a method flow for cooperative work of a tablet and a desk lamp, specifically including:

s601, responding to the operation of clicking the word searching function by a user, executing the processes of a device discovery stage, a virtual camera enabling stage and a virtual camera preview access stage by the tablet and the table lamp, and displaying a preview interface by the tablet.

In this scenario, in order to reduce the bandwidth occupied by the preview image stream, in the quasi-camera preview access phase, the desk-lamp camera may collect the preview image according to the low resolution. For example, the hardware virtualization API in the tablet may send a virtual camera access instruction to the camera service with a first configuration parameter, where the first configuration parameter includes a first image resolution (e.g., 480P). And setting the table lamp camera according to the received configuration parameters, and acquiring preview image data according to the first image resolution and the corresponding image acquisition frame rate.

S602, the education APP in the tablet performs finger recognition on the preview image.

S603, in response to the operation of using finger words by the user, the education APP in the tablet identifies the fingers of the user in the preview image.

S604, the education APP in the tablet determines ROI information according to the position of the finger in the preview image, and generates a photographing request according to the ROI information.

The ROI information refers to information for determining the ROI, and may include, but is not limited to, center point coordinates and region ranges (e.g., width-height information).

S605, the tablet transmits a photographing request to the desk lamp side.

The photographing request may include, but is not limited to, a virtual camera ID corresponding to the desk lamp camera, a second configuration parameter of the desk lamp camera, and ROI information. The second configuration parameters include, but are not limited to, a second image resolution that is higher than the first image resolution, such as the second image resolution set to 1080P. Therefore, the education APP can accurately identify and determine the word or picture to be interpreted according to the high-resolution image.

S606, the camera of the desk lamp is set according to a second configuration parameter carried in the photographing request, images are photographed according to a second image resolution, and the photographed images are sent to the hardware abstraction service.

S607, the hardware abstraction service determines the ROI image according to the ROI information.

The hardware abstraction service may crop the ROI image in the captured image based on the ROI information.

In an alternative embodiment, the photographing request includes, but is not limited to, the second configuration parameter, but does not include ROI information. Like this, the dull and stereotyped request of shooing is transmitted to the desk lamp side, and the camera of desk lamp sets up according to the second configuration parameter that carries in the request of shooing to shoot the image according to second image resolution, and returns the image of shooing to education APP in the dull and stereotyped. Furthermore, the educational APP may determine the ROI image from the ROI information, for example, crop the ROI image from the captured image based on the ROI information.

S608, the desk lamp transmits the ROI image to the education APP in the tablet.

Compared with the table lamp, the method has the advantages that the high-resolution shooting image is directly returned to the education APP in the panel, the cut ROI image is returned to the education APP in the panel, the data transmission quantity can be reduced, and the bandwidth occupation is reduced.

S609, education APP in the plate accurately identifies the ROI image and determines the word to be interpreted.

S610, the education APP in the tablet performs word searching operation on the word to be interpreted, and displays the paraphrasing of the word to be interpreted.

The present process is not explained in detail, and reference is made to the foregoing, and will not be repeated here.

Similarly, the user may use a pointing tool such as a stylus to perform a pointing operation, which is not limited in this embodiment. Correspondingly, the education APP identifies the word pointing tool on the preview image to determine whether the user has word searching intention, and determines the ROI information according to the position information of the word pointing tool on the preview image.

Similarly, the user may also point to a picture in the book using a pointing device such as a finger or stylus. Correspondingly, the education APP determines the ROI information according to the position information of the finger or the finger-word tool such as a touch pen, determines the ROI image in the shot image for the ROI information, performs picture content identification for the ROI, displays the paraphrasing corresponding to the picture, and can read the displayed picture paraphrasing. In this case, the present embodiment will not be described in detail.

For the application scenario of this flow, reference may be made to the application scenario shown in fig. 1a, fig. 7 a-fig. 7 b. Referring to fig. 7a (2), when a user uses a finger in a book, a desk lamp camera captures a finger preview image, which is displayed on an image preview window 7021. Furthermore, the education APP can identify the finger of the user in the preview image, determine the ROI information, and generate a photographing request according to the ROI information and the image high resolution to trigger the desk lamp camera to photograph the image. The desk lamp camera shoots images according to the high resolution of the images, the desk lamp side cuts the high resolution shooting images according to the ROI information to obtain the ROI images, and the ROI images are returned to the education APP in the flat plate. The education APP accurately identifies the ROI image and determines the word to be interpreted. After the education APP queries the paraphrasing of the new words, the corresponding paraphrasing of the new words is displayed on the interface, and the education APP can refer to FIG. 7 b.

Scene two

Referring to the application scenario schematic diagram shown in fig. 1a, the present scenario is illustrated by taking a job function as an example. When students need to submit homework online, the students can click to shoot in the education APP, the desk lamp camera is used for shooting homework images, and the homework images are uploaded to the database through the education APP.

FIG. 8 is a schematic diagram showing the interaction of the modules. Referring to fig. 8, the embodiment of the application provides a method flow for cooperative work of a tablet and a desk lamp, specifically including:

s801, in response to operation of clicking a job function by a user, the education APP in the tablet displays a job submission list.

Note that the job submission list refers to a list including a plurality of job submission options, one job submission option corresponding to each job, and can be referred to as interface 704 in fig. 9 a. The job options may be divided according to disciplines or time, which is not limited in this embodiment.

If only one job image is submitted for one job in the job function of the education APP, the education APP does not display the job submission list. At this time, in response to the operation of clicking the job function by the user, the tablet and the desk lamp execute the flows of the device discovery stage, the virtual camera enabling stage, and the virtual camera preview access stage, and the tablet displays the preview interface.

S802, responding to the click of a user to submit a job option, the flat panel and the desk lamp execute the processes of a device discovery stage, a virtual camera enabling stage and a virtual camera preview access stage, and the flat panel displays a preview interface.

S803, responding to the operation of clicking the photographing option by the user, and generating a photographing request by the education APP in the tablet.

When a user places a job or a book and the like in the acquisition area of the desk lamp camera, the user can click a photographing option to trigger the desk lamp camera to photograph a job image.

The photographing request may include, but is not limited to, a virtual camera ID corresponding to the desk lamp camera, and a second configuration parameter of the desk lamp camera. The second configuration parameters include, but are not limited to, a second image resolution that is higher than the first image resolution, such as the second image resolution set to 1080P. Thus, the education APP can upload the high-resolution operation images.

S804, the tablet transmits a photographing request to the desk lamp side.

S805, the camera of the desk lamp is set according to a second configuration parameter carried in the photographing request, and images are photographed according to a second image resolution.

S806, transmitting the shot image to the education APP in the flat plate by the table lamp.

S807, education APP in the tablet displays the photographed image.

The education APP in the tablet receives the operation image shot by the desk lamp camera, and displays the operation image. If the user is satisfied with the captured job image, the user may click on a submit option to upload the job image to the database. If the user is not satisfied with the photographed job image, the user may re-click the photographing option to trigger the desk lamp camera to re-photograph the job image.

S808, in response to the user clicking the submit option, the education APP in the tablet uploads the shot image to the database.

It should be noted that the above mentioned job images are only exemplary examples, and the user may click on the photographing option to trigger the desk lamp camera to take other images. After the table lamp sends the shot image to the education APP in the tablet, the education APP can upload the received shot image to a corresponding database.

Fig. 1a, 9 a-9 c show an exemplary application scenario. As shown in fig. 9a (1), the tablet display education APP interface 701 displays various function options of the education application on the interface 701, including but not limited to a word search function, a click-to-read function, a job function, a photo function, etc. The user clicks the word search function option 7012, and in response to a user operation, the education APP in the tablet displays a job submission list, as shown in (2) of fig. 9 a. In the job submission list interface 704 shown in fig. 9a (2), a plurality of submitted job options (e.g., submitted job 1, submitted job 2, submitted job 3, submitted job 4, etc.) are displayed, with different job submission options corresponding to different jobs. Taking the example of a user needing to upload a job image for commit job 4, the user clicks commit job 4 option 7042. In response to a user operation, the tablet may display a job submission interface 705 as shown in fig. 9b (1).

With continued reference to (1) in fig. 9b, in the job submission interface 705, an image preview window 7041, a photograph option 7051, and a submission option 7052 are displayed. The preview window 7041 displays a preview image acquired by the desk lamp camera in real time. When a user places a job or a book or the like in the desk lamp camera acquisition area, the user can click the photographing option 7051 to trigger the desk lamp camera to photograph a job image. In response to user operation, the education APP generates a photographing request and sends the photographing request to the desk lamp side so as to call the desk lamp camera to photograph the operation image. The desk lamp camera shoots the operation image according to the high resolution of the image carried in the shooting request, and returns the shot operation image to the education APP in the tablet for display through the education APP, and reference can be made to an interface 706 as shown in (2) in fig. 9 b.

With continued reference to (2) in fig. 9b, an image preview window 7041, a photographing option 7051, a submitting option 7052, and a job image 7061 photographed by a desk lamp camera are displayed in the interface 706. A close option 7062 is also displayed in the job image 7061. If the user is not satisfied with the job image 7061, the close option 7062 may be clicked and the job image 7061 will not be displayed on the interface. At this time, the user may click on the photographing option 7051 to trigger the desk lamp camera to re-photograph the job image. If the user is satisfied with the job image 7061, the submit option 7052 may be clicked. In response to a user operation, the tablet may display an interface to be confirmed 701 as shown in fig. 9 c. Therein, a job image 7061 to be submitted and whether to confirm the submit window 7071 are displayed in the interface to be confirmed 701. If the user clicks on the confirmation option 7072 in the confirmation submission window 7071, the educational APP uploads the job image 7061 to the database in response to the user operation. If the user clicks the cancel option 7071 in the confirm submission window 7071, the education APP may display an interface as shown in (1) of fig. 9b in response to the user operation to trigger the desk lamp camera to re-photograph the job image with the standby click photographing option 7051.

Note that, with continued reference to (2) in fig. 9b, if the user is satisfied with the job image 7061, the submit option 7052 may be clicked. In response to a user operation, the educational APP may not display the interface as shown in fig. 9c any more, and directly upload the job image 7061 to the database. This embodiment is not limited thereto.

Scene three

Referring to the application scenario schematic diagram shown in fig. 1a, the present scenario is illustrated by taking a point-to-read function (or referred to as a read function) as an example. When students need to read the contents in books by the education APP, the desk lamp camera can be used for collecting book images in real time, so that the education APP can load corresponding book contents according to the book images, and the book contents to be read are determined to be read according to the finger positions or page turning operation of the students.

An interaction diagram of the modules is shown in fig. 10. Referring to fig. 10, the embodiment of the application provides a method flow for cooperative work of a tablet and a desk lamp, specifically including:

s901, responding to the operation of clicking the click-to-read function by a user, executing the processes of a device discovery stage, a virtual camera enabling stage and a virtual camera preview access stage by the tablet and the desk lamp, and displaying a preview interface by the tablet.

S902, identifying the preview image by the education APP in the tablet, and determining the name of the book.

The education APP may perform book information identification for each received preview image, or may perform book information identification for the latest received preview image at regular time, which is not limited in this embodiment.

S903, the education APP in the tablet computer searches the database according to the book names and loads book contents corresponding to the book names.

If the education APP in the tablet retrieves the database according to the book names and determines books with different versions, a corresponding book list can be displayed for the user to select. Further, in response to a user's selection operation of a certain version of a book, the educational APP loads content corresponding to the version of the book

S904, responding to page turning or finger click operation of a user, identifying the preview image by the education APP in the tablet, determining a paragraph to be read, and reading the corresponding paragraph.

After the book content is loaded, the education APP can recognize the instruction tool of the finger or the touch pen instruction for each received preview image, or can recognize the instruction tool of the finger or the touch pen instruction for the latest received preview image at regular time, which is not limited in the embodiment. In response to a click operation of a user, the education APP can identify the page number of the book and click position information of the user, such as coordinate information and the like, according to the preview image, and further can determine paragraphs to be read in the loaded book content according to the page number of the book and the click position information of the user, and read corresponding paragraphs.

After the book content is loaded, the education APP can also perform page turning identification according to the preview image stream. In response to a page turning operation of a user, the education APP can determine paragraphs to be read in the loaded book content according to the identified book page numbers, and read corresponding paragraphs.

Fig. 1a, 11 a-11 b show an exemplary application scenario. As shown in (1) of fig. 11a, the tablet display education APP interface 701 displays various function options of the education application on the interface 701, including but not limited to a word search function, a click-to-read function, a job function, a photo function, etc. The user clicks the click function option 7013, and in response to the user operation, the tablet executes the flows of the device discovery phase and the virtual camera service enable phase, and the virtual camera preview access phase.

In the device discovery stage, if the number of table lamps that can be linked with the tablet is one, the tablet automatically executes the processes of the virtual camera service enabling stage and the virtual camera preview access stage, and displays an interface shown in (2) in fig. 11a, for example. In the device discovery stage, if the number of the table lamps which can be linked with the tablet is a plurality of the table lamps obtained through filtering by the device management service of the tablet, the tablet displays a table lamp selection interface. For example, a list of the desklamps to be linked is displayed on the desk lamp selection interface, the user may perform a selection operation, and in response to the selection operation by the user, the educational application determines one desk lamp to be linked and continues to perform the flow of the virtual camera service enabling phase and the virtual camera preview access phase to display the interface shown in (2) of fig. 11a, for example.

As shown in fig. 11a (2), an image preview window 7081 is displayed in the interface 708. The preview window 7081 displays a preview image acquired by the desk lamp camera in real time. The educational APP identifies the preview image to determine the book name. After identifying the book name, the educational APP retrieves the database based on the book name. If a corresponding book is retrieved, a loading operation of the book contents is performed, as shown in (1) of fig. 11 b.

With continued reference to fig. 11b (1), an image preview window 7081, an identified book name 7091, and a book content loading progress identifier 7092 are displayed in the interface 709. After the book content is loaded, the education APP can identify the click-to-read operation or page turning operation of the user according to the preview image. Taking the example of a user performing a page turning operation, referring to the interface 710 as shown in (2) in fig. 11b, a page turning action of the user may be displayed in the image preview window 7081. In response to a user operation, the education APP recognizes the page number of the book according to the preview image. Furthermore, the education APP can determine the paragraphs to be read in the loaded book content according to the identified book page numbers, and read the corresponding paragraphs.

In the collaborative work method provided by the embodiment of the application, the household flat plate and the table lamp with the camera are used for combining the professional online education experience. The camera of the desk lamp equipment is matched with the tablet education APP to complete the scenes of students such as finger word searching, homework submitting, book reading and the like needing photographing.

In order to enable the tablet and the desk lamp to perform data interaction with each other efficiently and quickly when the electronic device 100 (for example, a tablet) and the IOT device 200 (for example, a desk lamp) are used in conjunction with each other in the above-mentioned various learning scenarios, for example, the desk lamp may quickly transmit the photographed photo data to the tablet, in some implementations, a physical transmission channel, such as a P2P channel, capable of implementing point-to-point transmission may be first established between the tablet and the desk lamp, and then, when data transmission, such as a preview stream and photo data, is required, the tablet and the desk lamp determine a logical channel specially used for transmitting the preview stream and the photo data through negotiating over the P2P channel, so that the desk lamp may efficiently and quickly acquire the preview stream and the photo data for the tablet through the camera, for example, the camera 201 above, to perform processing.

In particular, in practical applications, the current process of establishing the P2P channel needs to involve scanning of the medium access control address (Media Access Control Address, MAC address), which takes a lot of time. In addition, after the MAC address is scanned, a broadcast needs to be sent to a device corresponding to the scanned MAC address, then the device sending the broadcast and the device receiving the broadcast need to perform role negotiation, such as negotiation to determine which device is a hotspot creator (i.e., GO), which device is a hotspot access party (i.e., GC), after the roles of the two parties are determined by negotiation, the P2P network port can be invoked to initiate a P2P connection request, which takes a long time and requires user intervention.

In addition, a P2P network port is called to initiate a P2P connection request, a connection stage is entered, a user is required to manually input hot spot information or authentication information in the current standard flow, and finally, after authentication is successful, a P2P channel between the desk lamp and the flat plate can be established.

Therefore, the current process of establishing the P2P channel involves more flows, complex operations, and redundant flows, and IOT devices are usually thin devices (system resources and limited processing capacity), so the current manner of establishing the P2P channel is limited by the capabilities of the IOT devices, and the establishment process is slow, which causes huge consumption of resources and time for IOT devices with fewer resources.

In view of this, the present application provides a method for quickly and efficiently establishing a P2P channel, where the method omits the conventional P2P channel establishment standard, when establishing a P2P channel, an electronic device initiating a request directly designates its own role information, and identifiers identifying identities of both parties are exchanged in an authentication phase, so that the electronic device receiving a P2P channel establishment request packet authenticates the identifier carried in the P2P channel establishment request packet directly according to the identifier obtained in the authentication phase, determines whether to establish a hotspot or directly access the hotspot according to the role information after the authentication passes, and finally realizes connection between the two according to the hotspot information, such as a password and an account number, thereby greatly simplifying the process of establishing the P2P channel, and establishing the P2P channel between the two in a wireless access point manner based on the hotspot information, so that the success rate of establishing the P2P channel is greatly improved.

Referring to fig. 12, functional modules involved in establishing a P2P channel between a tablet and a desk lamp, and an interaction schematic between the functional modules are illustrated.

Referring to fig. 12, illustratively, when setting up a P2P channel between a tablet and a desk lamp, a user may trigger a request to set up the P2P channel through a function button provided by an educational application at an application layer.

Accordingly, the tablet responds to the operation behavior, the educational application accesses the transmission management service also located in the application framework layer through the multi-device virtualization service platform framework in the application framework layer, so that the transmission management service generates a P2P channel establishment request message according to the redefined message requirement, and sends the P2P channel establishment request message to the device interconnection service located in the application layer in the desk lamp.

The framework of the multi-device virtualization service platform is specifically a virtual device for converting peripheral devices or device devices into flat boards, and can provide and use the peripheral device capabilities as the universal capabilities of the flat board system, and simultaneously realize synchronous and collaborative use of the multi-virtual device capabilities. In this way, through the basic multi-device virtualization service platform framework in the application framework layer of the tablet, external devices including but not limited to cameras, speakers, displays, microphones, wearable external devices and related device capabilities can be conveniently used, flexible control and switching can be supported, the resource advantages of the external devices can be fully used, and better collaborative experience is created for consumers. Therefore, in the embodiment, the education application accesses the transmission management service through the interface provided by the multi-device virtualization service platform framework, so that communication with the desk lamp can be realized.

In addition, in this embodiment, the redefined message requires that the data format carried in the message be JSON format (JavaScript Object Notation, JS object profile), that is, a lightweight data interaction format.

In addition, in this embodiment, the P2P channel setup request message generated based on the redefined message requirement at least needs to include role information, an identifier for identifying the device that initiates the message, and the type of information, and so on carried.

Further, in order to distinguish whether the message is a request message, a response message or a disconnection message, the request type needs to be carried.

In addition, in order to enable the message requirement to be applicable to the bridge mode, field information of whether the bridge mode is supported or not and a corresponding type can be further included.

In addition, in order to enable the two devices that have established the P2P channel to learn the previously allocated internet protocol address (Internet Protocol Address, IP address) when they subsequently initiate a request for establishing the P2P channel again, the message requires that field information related to the IP address can also be set.

Based on the above requirements, this embodiment gives a specific message requirement, as shown in table 1.

Table 1 message requirement table

Command type	KEY_COMMAND_TYPE	int
			Version number	KEY_VERSION	int
P2P MAC	KEY_MAC	string
			IP address	KEY_IP	string
Current role	KEY_ROLE	int
			Desired roles	KEY_EXPECTED_ROLE	int
Whether or not to support bridge mode	KEY_BRIDGE_SUPPORTED	boolean
			Type of information carried	KEY_CONTENT_TYPE	int

The content of the command type field in table 1 is specifically a request message for identifying whether the message is a request message for establishing a P2P channel, a response message for a request message for establishing a P2P channel, or a request message for disconnecting a P2P channel. In this embodiment, the contract "8" identifies that the message is a request message for establishing a P2P channel, the "9" identifies that the message is a response message made for the request message for establishing a P2P channel, and the "5" identifies that the message is a request message for disconnecting a P2P channel, i.e., using a digital representation of the contracted integer (int).

The content of the field of the version number in table 1 is also of the int type, in particular determined according to the device system version number of the transmitted message, for example.

Illustratively, the content of the P2P MAC field in table 1 is specifically a string type, an identifier for identifying the device that sent the message.

Regarding the selection of the identifier, for a usage scenario in which the MAC address is fixed, the identifier may be the MAC address of the device; for usage scenarios where the MAC address is not fixed, the identifier may be a basic service group identification (Basic Service Set ID, BSSID).

Illustratively, the content of the IP address field in table 1 is specifically a string (string) type, which is used to identify the IP of the transmitted message. In this embodiment, for a party accessing to a hotspot, the IP address is specifically allocated by the party creating the hotspot after accessing to the hotspot according to the hotspot information of the hotspot created by the party creating the hotspot and creating the P2P channel. Therefore, when a request message for establishing a P2P channel is initiated, if the role of the initiator is GC and the GC has not been accessed to a hot spot created by GO receiving the request message before, the content corresponding to the field may be null or may not carry the field in this scenario.

Illustratively, the current role field and the desired role field in table 1, i.e., the role information described above, are both of the int type. The content corresponding to the current role field is specifically filled according to the real-time state of the device, for example, when the current role field is determined to be GO, GO is filled, and other NONs are filled; when the content corresponding to the expected ROLE field is used for identifying whether the device expects to be used as GO or GC when the P2P channel is established, if the device determines that GO is determined according to the product characteristics, the GO is filled, if the device does not, the GC is filled, and otherwise, the device can be represented by default content ROLE_AUTO.

For example, the content of the bridge mode field in table 1 is specifically boolean (boolean), and for example, it may be agreed that "1" indicates that the bridge mode is supported and "0" indicates that the tap mode is not supported. When the field is carried in the message and the corresponding content is "1", it means that one GO can be connected with the GC through another GO already connected with the GC.

It will be appreciated that in some implementations, the contents of this field may be empty, or not carry the field directly, if a bridge mode is not considered.

The information type field carried in table 1 is specifically an int type, and the content of the field is the authentication information of the role described above. When the role is GO, the field needs to carry a 'GO' mark and hot spot information of a hot spot created by the equipment; when the role is GC, the field needs to carry a GC flag, and at the same time, carry the PIN code of the device.

It should be understood that, table 1 is only an example listed for better understanding the technical solution of the present embodiment, in practical application, each name corresponding to the automation, such as "key_command_type" corresponding to the COMMAND TYPE and "key_version" corresponding to the VERSION number, may be named according to the actual requirement, and in addition, the TYPE (int, string, boolean) corresponding to each field may also be adjusted according to the actual requirement, which is not limited in this embodiment.

With continued reference to fig. 12, after receiving a P2P channel setup request message sent by the transport management service located at the application framework layer in the tablet, the device interconnection service located at the application layer in the table lamp analyzes the P2P channel setup request message according to the redefined message generation requirement, thereby determining whether to create a hotspot or access the hotspot.

Correspondingly, if the creation of the hot spot is determined, the device interconnection service creates the hot spot, a P2P channel establishment response message is generated according to the redefined message generation requirement and the hot spot information of the generated hot spot, and the P2P channel establishment response message is sent to the transmission management service positioned in the application program framework layer in the flat plate, so that the transmission management service establishes a P2P channel with the desk lamp according to the hot spot information in the P2P channel establishment response message.

Otherwise, if the access hot spot is determined, the device interconnection service establishes a P2P channel with the flat plate according to the hot spot information carried in the P2P channel establishment request message.

That is, according to the method for establishing a P2P channel provided in this embodiment, when the P2P channel between the desk lamp and the tablet is established, no matter the desk lamp or the tablet, only one side is required to initiate a P2P channel establishment request message, and the other side is required to make a P2P channel establishment response message, so that the whole P2P channel establishment process does not need to scan MAC addresses, send broadcasting, negotiate roles, and the like, and does not need to input authentication information, hotspot information, and the like, thereby greatly simplifying the P2P channel establishment flow, and based on this method, effectively reducing consumption of thin equipment resources and time such as the desk lamp, and achieving rapid and efficient establishment of the P2P channel between the desk lamp and the tablet.

With continued reference to fig. 12, exemplary, if the user wants to disconnect the P2P channel between the desk lamp and the tablet, in some implementations, the user can trigger a request to disconnect the P2P channel through a function button provided by the educational application at the application layer in the tablet.

Accordingly, the tablet responds to the operation behavior, the educational application accesses the transmission management service also located in the application framework layer through the multi-device virtualization service platform framework in the application framework layer, so that the transmission management service generates a P2P channel disconnection notification message according to the redefined message requirement, and sends the P2P channel disconnection notification message to the device interconnection service located in the application layer in the desk lamp.

With continued reference to fig. 12, after receiving a P2P channel disconnection notification message sent by the transmission management service located at the application framework layer in the tablet, the device interconnection service located at the application layer in the table lamp analyzes the P2P channel disconnection notification message according to the redefined message generation requirement, thereby disconnecting the P2P channel with the tablet identified by the identifier.

In addition, it should be noted that, in other implementations, if the user wants to disconnect the P2P channel between the desk lamp and the tablet, the process of the educational application may be directly killed, so that the above-mentioned process of disconnecting the P2P channel is automatically triggered.

Further, it is understood that in some implementations, the function buttons for triggering the request to establish the P2P channel and the request to disconnect the P2P channel may be the same or different.

In addition, in other implementations, whether the request for establishing the P2P channel or the request for disconnecting the P2P channel is triggered, the request may also be triggered by a specific number of taps, a tap position, a gesture, a voice command, a mechanical key, or the like, which is not limited in this embodiment.

In order to better explain the P2P channel establishment method provided by the present embodiment, details of implementation of establishing a P2P channel between a desk lamp and a tablet are described below with reference to the accompanying drawings.

It should be noted that, according to the redefined message requirements given above, in a specific implementation, the party initiating the P2P channel establishment request message may be a party creating a hotspot (GO) or a party accessing the hotspot (GC) according to its product characteristics. When the user is used as a party for creating the hot spot, the P2P channel establishes authentication information corresponding to the role carried in the request message, namely the carried information type and specific content are hot spot information, such as an account number and a password of the created hot spot; when the user is a party accessing to the hot spot, the authentication information corresponding to the role carried in the P2P channel establishment request message is, for example, a PIN code.

Referring to fig. 13, taking a tablet as a GC, sending a P2P channel setup request message (a request message for setting up a P2P channel) to a table lamp, and taking a GO as an example, sending a P2P channel setup response message (a response message for the request message for setting up a P2P channel) to the tablet lamp, the method for setting up a P2P channel provided in this embodiment specifically includes:

s1001, the tablet generates a P2P channel setup request packet including an identifier a, role information a, and authentication information a (PIN code) corresponding to the role.

It will be appreciated that in practical applications, the characteristics of a product may be determined based on the ID of the product, i.e., the usage scenario in which the product is typically applied, and the roles in that scenario, may be known based on the ID. Therefore, according to the ID of the product and the message requirements, the P2P channel establishment request message meeting the requirements can be assembled and generated, and then the generated P2P channel establishment request message is sent to the desk lamp which has exchanged the identification numbers in the authentication stage.

S1002, the table lamp checks an identifier A included in the P2P channel establishment request message according to the identifier of the flat panel obtained in the authentication stage.

Specifically, after receiving the P2P channel establishment request message sent by the tablet, the table lamp analyzes the P2P channel establishment request message according to the above message requirements, and further verifies the identifier a carried in the P2P channel establishment request message according to the identifier of the tablet obtained in the authentication stage.

It can be understood that, in order to ensure validity of the verification result, the source of the identifier exchanged in the authentication phase and the identifier a carried in the P2P channel setup request packet should be all the time, that is, if the exchanged identifier in the authentication phase is the BSSID, the identifier carried in the P2P channel setup request packet should also be the BSSID, and if it is the MAC address, the identifier carried in the P2P channel setup request packet should also be the MAC address. Otherwise, the verification failure process is performed, for example, the desk lamp does not respond at all, and the flat plate side waits for the request to exceed the set timeout time and then automatically prompts on the user interface or resends the P2P channel establishment request message once.

S1003, when the verification is successful and the role information A indicates that the flat panel is one of the access hotspots, the table lamp creates the hotspots.

In addition, it should be noted that, in practical application, the table lamp may also verify the tablet according to the authentication information, such as a PIN code, carried in the P2P channel establishment request message, and create a hotspot after the verification is passed, so as to further ensure that the tablet with which the P2P channel is established is a legal authorized device. For the PIN code verification procedure, reference may be made to the existing standard, and will not be described here again.

S1004, the desk lamp generates a P2P channel establishment response message comprising the identifier B and the hotspot information of the created hotspots.

Similarly, the source of the identifier B of the desk lamp should be consistent with the identifier sent to the tablet in the authentication phase, that is, if the identifier exchanged in the authentication phase is the BSSID, the identifier carried in the P2P channel setup request message should also be the BSSID, and if it is the MAC address, the identifier carried in the P2P channel setup request message should also be the MAC address.

In addition, it can be understood that, in order to ensure the security of the P2P channel established later, the hotspot created by the desk lamp may set a corresponding connection password, and the hotspot information in the fixed P2P channel establishment response message may include, for example, the account number of the created hotspot and the password for connecting the hotspot.

S1005, the flat panel verifies the identifier B included in the P2P channel establishment response message according to the identifier of the table lamp obtained in the authentication stage.

Specifically, after receiving the P2P channel setup response message sent by the desk lamp, the tablet analyzes the P2P channel setup response message according to the above message requirements, and further verifies the identifier B carried in the P2P channel setup response message according to the identifier of the desk lamp obtained in the authentication stage.

Accordingly, when the verification is successful, step S1006 is executed; otherwise, the verification failure flow is carried out. The verification and identification process is similar to that of the desk lamp side, and will not be described in detail herein.

S1006, when verification is successful, the flat plate establishes a P2P channel with the desk lamp according to the hot spot information included in the P2P channel establishment response message.

Namely, the method is automatically connected to the hot spot created by the table lamp in an AP mode according to the account number and the password of the hot spot, and further the establishment of a P2P channel between the two is realized.

Accordingly, after the P2P channel is established, when the tablet computer performs data interaction with the table lamp, the logic channels for transmitting different data, such as preview stream and photo data, can be negotiated and determined in the P2P channel, so that the preview stream and photo data provided by the table lamp side can be rapidly and efficiently transmitted to the tablet computer side on the basis of the P2P channel.

Referring to fig. 14, taking a tablet as GO, sending a P2P channel setup request message (a request message for setting up a P2P channel) to a table lamp, and taking a table lamp as GC, sending a P2P channel setup response message (a response message for the request message for setting up a P2P channel) to the tablet lamp as an example, the method for setting up a P2P channel provided in this embodiment specifically includes:

S1101, the tablet creates a hotspot.

It can be understood that when the role of the tablet is GO, the authentication information carried in the P2P channel establishment request message sent to the table lamp is specifically hotspot information, such as an account number and a password, of a hotspot created by the tablet. Therefore, the tablet needs to create a hotspot first to obtain hotspot information of the hotspot, so that step S1102 can be performed.

S1102, the tablet generates a P2P channel setup request message including an identifier a, role information a, and authentication information a (hotspot information) corresponding to the role.

In this embodiment, the manner in which the flat panel generates the P2P channel setup request message is the same as the manner in which the flat panel generates the P2P channel setup request message when the flat panel is used as a GC, and the authentication information carried in this embodiment is specifically hotspot information, but not PIN code. Specific implementation logic may be referred to above, and will not be described herein.

S1103, the table lamp checks the identifier A included in the P2P channel establishment request message according to the identifier of the flat panel obtained in the authentication stage.

It is not difficult to find that step S1103 in the present embodiment is substantially the same as step S1002 in the above embodiment, and specific implementation details are not described here again.

S1104, when the verification is successful and the role information A indicates that the flat panel is a hot spot provider, the table lamp establishes a P2P channel with the flat panel according to the hot spot information included in the P2P channel establishment request message.

It can be understood that, because the hotspot information is already carried in the P2P channel establishment request message, the smart platform does not need to create a hotspot again, and directly accesses the hotspot created by the tablet according to the hotspot information, such as the account number and the password, carried in the P2P channel establishment request message, so that the P2P channel between the smart platform and the tablet can be established.

In addition, it should be noted that, when the flat panel sending the P2P channel setup request message is one party that creates a hotspot, because the flat panel has not yet verified the table lamp, even if the table lamp accesses the hotspot created by the flat panel according to the hotspot information in the P2P channel setup request message, the P2P channel between the two is not actually being opened, that is, the table lamp cannot transmit data. Therefore, the desk lamp also needs to send a P2P channel setup response message carrying its own related information to the tablet, i.e. execute step S1105.

S1105, the desk lamp generates a P2P channel establishment response message comprising an identifier B, role information B and authentication information B (PIN code) corresponding to the role.

It can be understood that the manner of generating the P2P channel setup response message by the desk lamp in step S1105 is similar to the manner of generating the P2P channel setup request message by the tablet in step S1001, and will not be described herein. The difference is that the source of the data in each field is different, one is from the tablet, one is from the desk lamp, and the command type carried in addition is "8" specified in the message request in step S1001, and "9" specified in the message request in step S1105.

S1106, the flat panel checks the identifier B included in the P2P channel establishment response message according to the identifier of the table lamp obtained in the authentication stage, and determines that the verification is successful.

It is not difficult to find that step S1104 in this embodiment is substantially the same as step S1005 in the above embodiment, and specific implementation details are not described here again.

Accordingly, after verification is passed, the P2P channel between the tablet and the desk lamp is only being available.

In addition, it should be noted that, considering that in practical application, there may be a hot spot created by accessing one flat plate (as GO) into a plurality of table lamps, the flat plate establishes a P2P channel with each table lamp respectively; there may also be multiple plates accessing the same table lamp (as GO), which establishes P2P channels with each plate separately. Therefore, the party (GO) creating the hotspot can also establish a correspondence between the IP address currently corresponding to each GC for which the P2P channel is established and the identifier for identifying the GC, and add the correspondence to the IP address/identifier management table for management.

It can be appreciated that with respect to the IP corresponding to each GC, it is specifically assigned by GO. Because the identifier of each GC is unique and fixed, the corresponding relationship is established between the IP address currently corresponding to each GC and the identifier identifying the GC, and the IP address currently corresponding to the GC and the identifier identifying the GC are added to the IP address/identifier management table according to the corresponding relationship, so that it can be known clearly that the GO currently establishes a P2P channel with those GCs. Thus, not only is the management convenient, but also different data can be sent to different GCs according to the content recorded in the IP address/identifier management table when the preview stream and the photo data are transmitted later.

Further, after obtaining the IP address/identifier management table, the GO may further obtain the state information of the established P2P channel in real time, and add the obtained state information to the IP address/identifier management table, so as to establish a correspondence with the identifier of the GC of the P2P channel and the IP address.

It can be understood that, in this embodiment, the state information of the P2P channel is specifically used to indicate the state of opening the P2P channel, that is, the state of performing data transmission, or the state of disconnection, that is, the state of failing to perform data transmission.

According to the description of the two scenes, whether the flat plate is GO or GC, based on the method for establishing the P2P channel, two message interactions are carried out between the flat plate and the table lamp, the P2P channel establishes a request message once, and the P2P channel establishes a response message once, so that the establishment of the P2P channel can be realized, and compared with the existing P2P channel establishment flow, the P2P channel establishment flow is greatly simplified, so that the consumption of IOT equipment resources and time can be reduced, and the P2P channel between IOT equipment and electronic equipment can be established more quickly and efficiently.

In addition, it should be noted that after the P2P channel is established between the desk lamp and the tablet, the P2P channel may be disconnected based on the above-mentioned message requirements.

Referring to fig. 15, the method for disconnecting a P2P channel provided in this embodiment specifically includes:

s1201, the tablet generates a P2P channel disconnection notification message including the identifier a.

Specifically, the 2P channel disconnection notification message is also based on the above message requirements.

In addition, in this embodiment, the condition for triggering the tablet to generate the 2P channel disconnection notification message may be that the user clicks a control in the educational application in the application layer, or may also use a specific gesture, or voice, or tap an arbitrary position or a specific position of the tablet, and then the tablet responds to the operation behavior, and accesses the transmission management service through the multi-device virtualization service platform framework in the application framework layer to generate the 2P channel disconnection notification message according to the above message requirement.

In addition, the condition for triggering the flat panel to generate the 2P channel disconnection notification message can also be that the process is triggered when the progress of the education application is killed.

S1202, the table lamp checks an identifier A included in the P2P channel disconnection notification message according to the identifier of the flat panel obtained in the authentication stage.

The verification operation in step S1202 is the same as step S1002 and step S1103, and will not be repeated here.

S1203, when the verification is successful, the desk lamp disconnects the P2P channel with the flat plate.

In addition, it can be understood that, in practical application, the P2P channel disconnection notification message may also be sent by the desk lamp, and specifically, the above process may be triggered when the user turns off the desk lamp, for example, after pressing the switch key.

In addition, it should be noted that, in order to ensure validity of the content recorded in the IP address/identifier management table, the corresponding state information of the P2P channel in the IP address/identifier management table may be modified each time the state of the P2P channel is monitored to change. For example, when the P2P channel between the desk lamp and the flat panel is disconnected according to the method for disconnecting the P2P channel, the state information corresponding to the identifier of the flat panel corresponding to the P2P channel may be modified to be disconnected. In this way, when the P2P channel is disconnected, the corresponding state information is modified instead of being deleted directly, so that the IP address is not required to be reassigned when the disconnected P2P channel is reconnected immediately, and the corresponding relationship is established, and the method can be better suitable for the scene that the P2P channel is disconnected immediately after Ma Chong is disconnected immediately due to network reasons or other anomalies.

Accordingly, in other implementation scenarios, when the P2P channel is disconnected, the identifier of the flat corresponding to the P2P channel in the IP address/identifier management table may be directly deleted, where the IP address and the status information corresponding to the identifier are directly deleted. Thus, redundant information in the IP address/identifier management table can be reduced, and occupation of equipment resources and space can be reduced.

Furthermore, as is apparent from the above description, the tablet and the desk lamp have completed the authentication phase before the P2P channel is established, and the identifier is interacted during the authentication phase. To better understand the process of exchanging the respective identifiers during the authentication phase, a specific description will be given with reference to fig. 16, taking the table lamp and tablet interactive identifier as a fixed and unchanged BSSID.

Referring to fig. 16, the authentication phase specifically includes:

s1301, the transmission management service on the flat panel side acquires the BSSID (bssid_1) of the home terminal.

In S1302, the transmission management service of the tablet transmits bssid_1 to the desk lamp based on the COAP protocol.

S1303, the device interconnect service on the desk lamp side stores bssid_1 of the tablet.

S1304, the device interconnect service on the desk lamp side acquires the BSSID (bssid_2) of the home terminal.

S1305, the device interconnection service on the desk lamp side transmits bssid_2 to the tablet based on the COAP protocol.

S1306, the transmission management service on the tablet side stores bssid_2 of the desk lamp.

It can be appreciated that in practical application, the desk lamp may first send its own bssid_2 to the tablet. That is, the process of exchanging identifiers between the tablet and the desk lamp does not distinguish between the order.

That is, in the authentication phase, the transmission management service on the tablet side acquires bssid_1 capable of identifying its own unique, and the device interconnection service on the desk lamp side also acquires bssid_2 capable of identifying its own unique. Then, the two parties split the respective identifiers into messages based on the COAP protocol, send the messages to the other party, store the identifiers sent by the other party, and perform identity verification based on the identifiers when the P2P channel is established and disconnected later.

In addition, on the premise that the tablet and the desk lamp can transmit the message which encapsulates the identifiers based on the COAP protocol in the authentication stage, two devices which perform authentication need to start a softtap without a password through software, and then interaction of the messages is realized by accessing the same softtap.

In this embodiment, the table lamp may be used as a party for starting a softtap, for example, after the table lamp is started for the first time, a softtap is automatically started, so that a nearby tablet can search for the softtap and access the softtap, so that the tablet and the table lamp are located in the same local area network. After the above operation is completed, the tablet and the desk lamp may package the message carrying the respective identifiers, such as BSSID, based on the COAP protocol, and then send the message to the other party, that is, the operations from step S1301 to step S1306 are implemented.

In addition, it can be understood that in practical application, in addition to the exchange of identifiers in the authentication stage, identity authentication can also be performed according to the PIN code. For authentication of the PIN code, reference may be made to existing standards, which are not described in detail herein.

Thus, when the desk lamp and the flat board are bound for the first time, the authentication is completed to realize the exchange of identifiers. When the method is applied to learning scenes, the P2P channel is directly built according to the method for building the P2P channel, MAC address scanning, broadcast sending and PIN code authentication are not needed again, and therefore redundant operation is reduced.

Referring to fig. 17, the P2P channel establishment stage specifically includes:

s1401, the multi-device virtualization service platform framework on the tablet side receives the user' S operation of the control for turning on the camera on the desk lamp.

It can be understood that, in practical application, the user's operation on the control for opening the camera on the desk lamp may, for example, operate a control in the interface provided by the education application, or the user triggers the operation through a voice command, or the user triggers the operation through a specific gesture, the number of times of tapping the screen, the position, and so on. However, in either trigger mode, the framework of the multi-device virtualization service platform that provides interface invocation functions for educational applications is responsive to sending a P2P connection request to the transport management service on the tablet side.

S1402, the transmission management service on the flat panel side assembles the message according to the identifier of the flat panel and the flat panel ID.

Specifically, after receiving a P2P connection request sent by the framework of the multi-device virtualization service platform, the transmission management service on the tablet side determines product characteristics according to its own ID, further determines role information (current role and expected role) of the tablet, and then can assemble a P2P channel establishment request message according to its own identifier, determined role information, and other parameters and the above message requirements.

Taking a learning scenario as an example, one desk lamp can be controlled by a plurality of tablet computers and mobile phones, namely, the desk lamp is one-to-many, for the scenario, the desk lamp usually takes a role of GO, and the tablet computers take a role of GC. Therefore, the product characteristics are determined based on the respective IPs, and the respective roles can be determined according to the current scene, so that the role negotiation step in the existing P2P channel establishment flow can be omitted, and the overall P2P channel establishment flow is simplified.

For easy understanding, this embodiment exemplarily shows a P2P channel setup request packet, which is specifically as follows: { KEY_COMMAND_TYPE:8, KEYVERSION:1.0.0, KEY_MAC:0a:94:ef:be:6c:a9, KEY_ROLE:NONE, KEY_EXPECT_ROLE:GC, KEY_CONTENT_TYPE:GC, "GC authentication information" }. According to the P2P channel establishment request message and the above description of the parameters of table 1, the command type of the message sent by the tablet is a request message for establishing a P2P channel, the current version number is "1.0.0", the identifier of the tablet is "0a:94:ef:be:6c:a9", the current role of the tablet is "NONE", that is, not GO, the expected role of the tablet is "GC", that is, the party accessing the hot spot, the type of authentication information carried in the request message is "GC", and the specific authentication information is GC authentication information, such as PIN code.

For example, in some implementations, after the transmission management service on the tablet side generates the P2P channel setup request packet, the transmission management service may send the P2P channel setup request packet to the device interconnection service on the desk lamp side based on the COAP protocol.

For example, in other implementations, after the transmission management service on the tablet side generates the P2P channel setup request packet, the transmission management service may further send the request packet to the device interconnection service on the desk lamp side based on a message queue telemetry transport protocol (Message Queuing Telemetry Transport, MQTT).

For example, in other implementations, after the transmission management service on the tablet side generates the P2P channel setup request packet, the transmission management service may also send the request packet to the device interconnection service on the desk lamp side based on a bluetooth low energy protocol (Bluetooth Low Energy, BLE) that is a protocol followed by bluetooth low energy.

For details of the implementation of data interaction based on the above several protocols, reference may be made to corresponding protocol identifiers, which are not described herein.

It should be understood that the above description is only an example for better understanding of the technical solution of the present embodiment, and is not to be taken as the only limitation of the present embodiment.

In addition, it should be noted that, in this embodiment, a plate is taken as an example for GC, if the plate is GO, a hotspot needs to be created, and the role information needs to be determined as GO in the generated P2P channel creation request message, and meanwhile, the type of authentication information is set as GO, and the hotspot information of the hotspot is carried.

S1403, when the tablet is a party accessing the hotspot, the device interconnection service on the desk lamp side creates the hotspot.

In this embodiment, the device interconnection service on the desk lamp side receives the P2P channel setup request message { key_command_type }, which is given above, by taking the plate side as an example for performing the GC role: 8, KEYVERSION:1.0.0, KEY_MAC:0a:94:ef:be:6c:a9, KEY_ROLE:NONE, KEY_EXPECT_ROLE:GC, KEY_CONTENT_TYPE:GC, and after the GC authentication information is analyzed according to the message requirement, the content '0a:94:ef:be:6c:a9' corresponding to the KEY_MAC field in the message is extracted, namely the identifier for identifying the flat plate. And further verifying the extracted '0 a:94:ef: be:6c: a 9' according to the identifier of the flat plate stored in the authentication stage.

Correspondingly, after verification is successful, the device interconnection service on the desk lamp side creates a hot spot, and relevant information of the hot spot, namely hot spot information such as an account number, a password and the like, is obtained.

S1404, the device interconnection service on the desk lamp side assembles the message according to the identifier and the hot spot information of the desk lamp.

For example, in some implementations, the P2P channel setup response message assembled by the device interconnection service on the desk lamp side may further include role information of the desk lamp.

Understandably, because IOT devices such as large-screen devices, desk lamps, cameras and the like are currently used as GO roles intelligently, role information carried in a P2P channel setup response message is specifically GO.

In addition, in order to make the flat panel that receives the message or the message be a response message made for the P2P channel setup request message that it sends, the P2P channel setup response message also needs to carry a command type, specifically "9".

For easy understanding, this embodiment exemplarily shows a P2P channel setup response message, which is specifically as follows: { KEY_COMMAND_TYPE:9, KEYVERSION:1.0.0, KEY_MAC:0a:94:ef:be:6c:7e, KEY_ROLE:GO, KEY_EXPECT_ROLE:GO, KEY_CONTENT_TYPE:GO, "GO hotspot information" }. According to the P2P channel setup response message and the above description of the parameters of table 1, the command type of the message sent by the desk lamp is a response message made for the P2P channel setup request message, the current version number is "1.0.0", the identifier of the desk lamp is "0a:94:ef:be:6c:7e", the current role of the desk lamp is "GO", the expected role of the desk lamp is "GO", that is, the party creating the hot spot, the type of authentication information carried in the response message is "GO", and the specific authentication information is hot spot information, such as account number and password of the hot spot.

For example, in some implementations, after generating the P2P channel setup response packet, the device interconnection service on the desk lamp side may send the P2P channel setup response packet to the transmission management service on the tablet side based on the COAP protocol.

For example, in other implementations, after the device interconnection service on the desk lamp side generates the P2P channel setup response packet, the device interconnection service may further send the P2P channel setup response packet to the transmission management service on the tablet side based on a message queue telemetry transport protocol (Message Queuing Telemetry Transport, MQTT).

For example, in other implementations, after generating the P2P channel setup response message, the device interconnection service on the desk lamp side may further send the P2P channel setup response message to the transmission management service on the tablet side based on a bluetooth low energy protocol (Bluetooth Low Energy, BLE) that is a protocol followed by bluetooth low energy.

Correspondingly, after the P2P channel establishment response message is obtained, the device interconnection service of the desk lamp side can also call back the IP address allocated to the flat plate and the identifier of the flat plate to the DVSDK through a pre-registered call back interface special for monitoring the P2P channel. Thus, the DVSDK can add the IP address and the identifier to the IP address/identifier management table according to the corresponding relation between the IP address and the identifier. The state information of the P2P channels acquired in real time can be added to the IP address/identifier management table later to better manage each P2P channel.

S1405, the transmission management service on the flat side establishes a P2P channel with the desk lamp according to the hot spot information.

It can be understood that, according to the description of the above embodiment, after receiving the P2P channel setup response message sent by the device interconnection service of the desk lamp side, the transmission management service of the tablet lamp side needs to parse the P2P channel setup response message, and then verify according to the identifier in the P2P channel setup response message.

Accordingly, after verification is successful, the hot spot created by the desk lamp is accessed in an AP direct connection mode according to the hot spot information, so that a P2P channel is established with the desk lamp. Therefore, after the hot spot information of the opposite terminal equipment is obtained, the method for establishing the P2P channel establishes the P2P channel between the opposite terminal equipment and the opposite terminal equipment in an AP direct connection mode instead of the traditional P2P network port scanning mode, and has the advantages of shorter connection time and higher success rate.

Regarding the procedure of establishing the P2P channel based on the method for establishing the P2P channel provided in the present application, it should be understood that the content carried in the P2P channel establishment request message and the content carried in the P2P channel establishment response message illustrated in the foregoing description are only examples listed for better understanding of the technical solution of the present embodiment, and are not the only limitation of the present embodiment.

Referring to fig. 18, the P2P channel disconnection phase specifically includes:

s1501, the multi-device virtualization service platform framework on the tablet side receives a user operation for turning off a control of a camera on a desk lamp.

It can be understood that in practical application, the user's operation on the control for closing the camera on the desk lamp may, for example, operate a control in the interface provided by the education application, or the user triggers the operation through a voice command, or the user triggers the operation through a specific gesture, the number of times of tapping the screen, the position, and the like. However, in either trigger mode, the framework of the multi-device virtualization service platform that provides interface call functions for point applications responds by sending a P2P disconnect request to the transport management service on the tablet side.

S1502, the transmission management service on the flat panel side assembles the message according to the identifier of the flat panel.

It can be understood that when the P2P channel is disconnected, the same disconnection process is executed regardless of the GO role or the GC role, so that the role information and the authentication information corresponding to the role need not be included in the P2P channel disconnection notification message generated by the transmission management service on the tablet side.

For easy understanding, the embodiment exemplarily shows a P2P channel disconnection notification message, which is specifically as follows: { KEY_COMMAND_TYPE:5, KEYVERSION:1.0.0, KEY_MAC:0a:94:ef:be:6c:a9}. According to the P2P channel disconnection notification message and the above description of the parameters of table 1, the command type of the message sent by the tablet is a request message for disconnecting the P2P channel, the current version number is "1.0.0", and the identifier of the tablet is "0a:94:ef:be:6c:a9".

For example, in some implementations, after the transmission management service on the tablet side generates the P2P channel disconnection notification packet, the transmission management service may also send the P2P channel disconnection notification packet to the device interconnection service on the desk lamp side based on any one of COAP protocol, MQTT protocol, and BLE protocol.

S1503, the device interconnection service on the desk lamp side clears the P2P related information and turns off the P2P channel.

Specifically, after receiving the P2P channel disconnection notification message sent by the transmission management service on the tablet side, the device interconnection service on the table lamp side analyzes the carried identifier, and determines whether the tablet sending the P2P channel disconnection notification message is a legal device or not by checking the identifier.

Accordingly, after the verification is successful, the device interconnection service on the desk lamp side can clear the related information of the P2P channel established with the flat plate, and turn off the P2P channel. Thus, the P2P channel is disconnected.

Further, the device interconnection service on the desk lamp side may also notify the DVSDK through the callback interface to delete all relevant information of the P2P channel in the IP address/identifier management table, such as the IP address allocated to the tablet, the identifier of the tablet, and status information.

Regarding the procedure of disconnecting the P2P channel, it should be understood that the content carried in the P2P channel disconnection notification message illustrated in the foregoing description is merely an example for better understanding the technical solution of the present embodiment, and is not the only limitation of the present embodiment.

Furthermore, it is understood that the electronic device (the first electronic device, the second electronic device described above) comprises corresponding hardware and/or software modules for performing the respective functions in order to achieve the above-described functions. The steps of an algorithm for each example described in connection with the embodiments disclosed herein may be embodied in hardware or a combination of hardware and computer software. Whether a function is implemented as hardware or computer software driven hardware depends upon the particular application and design constraints imposed on the solution. Those skilled in the art may implement the described functionality using different approaches for each particular application in conjunction with the embodiments, but such implementation is not to be considered as outside the scope of this application.

In addition, it should be noted that, in an actual application scenario, the methods provided in the foregoing embodiments implemented by the first electronic device and the second electronic device may also be executed by a chip system included in the first electronic device and the second electronic device, where the chip system may include a processor. The chip system may be coupled to a memory such that the chip system, when running, invokes a computer program stored in the memory, implementing the steps performed by the electronic device described above. The processor in the chip system can be an application processor or a non-application processor.

In addition, the embodiment of the application further provides a computer readable storage medium, where computer instructions are stored, which when executed on an electronic device, cause the first electronic device and the second electronic device to execute the related method steps to implement the method in the embodiment.

In addition, the embodiment of the application further provides a computer program product, which when run on an electronic device, causes the electronic device to execute the above-mentioned related steps to implement the method in the above-mentioned embodiment.

In addition, embodiments of the present application also provide a chip (which may also be a component or module) that may include one or more processing circuits and one or more transceiver pins; wherein the transceiver pin and the processing circuit communicate with each other via an internal connection path, and the processing circuit performs the related method steps to implement the method in the above embodiment, so as to control the receiving pin to receive the signal, and control the transmitting pin to transmit the signal.

In addition, as can be seen from the foregoing description, the electronic device, the computer-readable storage medium, the computer program product, or the chip provided in the embodiments of the present application are used to perform the corresponding methods provided above, and therefore, the advantages achieved by the method can refer to the advantages in the corresponding methods provided above, which are not repeated herein.

The above embodiments are merely for illustrating the technical solution of the present application, and not for limiting the same; although the present application has been described in detail with reference to the foregoing embodiments, it should be understood by those of ordinary skill in the art that: the technical scheme described in the foregoing embodiments can be modified or some technical features thereof can be replaced by equivalents; such modifications and substitutions do not depart from the spirit of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A method of establishing a point-to-point channel, applied to a first electronic device that exchanges an identifier identifying itself with a second electronic device during an authentication phase, the method comprising:

receiving a point-to-point P2P channel establishment request message sent by a second electronic device, wherein the P2P channel establishment request message comprises an identifier of the second electronic device, role information and authentication information corresponding to a role, and the role information is used for indicating that the second electronic device is one party for creating a hot spot or one party for accessing the hot spot;

verifying the identifier included in the P2P channel establishment request message according to the identifier of the second electronic device obtained in the authentication stage;

when the verification is successful and the role information indicates that the second electronic equipment is one party accessing the hot spot, creating the hot spot;

and sending a P2P channel establishment response message to the second electronic equipment, wherein the P2P channel establishment response message comprises an identifier of the first electronic equipment and first hotspot information of the created hotspot.

2. The method according to claim 1, wherein when the role information indicates that the second electronic device is a party that creates a hotspot, the authentication information is second hotspot information of the hotspot created by the second electronic device; the method further comprises the steps of:

When the verification is successful and the role information indicates that the second electronic equipment is a party for creating a hot spot, a P2P channel between the second electronic equipment and the second electronic equipment is established according to the second hot spot information;

and sending a P2P channel establishment response message to the second electronic equipment, wherein the P2P channel establishment response message comprises an identifier of the first electronic equipment, role information and authentication information corresponding to the role.

3. The method of claim 1, wherein after the creating a hotspot, the method further comprises:

distributing an Internet Protocol (IP) address for the second electronic equipment, and establishing a corresponding relation between the IP address and an identifier of the second electronic equipment;

and adding the IP address and the identifier of the second electronic equipment to an IP address/identifier management table according to the corresponding relation.

4. A method according to claim 3, characterized in that the method further comprises:

after a P2P channel between the second electronic device and the second electronic device is established, acquiring state information of the P2P channel, wherein the state information indicates that the P2P channel is opened or disconnected;

and adding the state information into the IP address/identifier management table, and establishing a corresponding relation with the identifier of the second electronic equipment and the IP address.

5. The method of claim 4, wherein after establishing the P2P channel with the second electronic device, the method further comprises:

receiving a P2P channel disconnection notification message sent by the second electronic device, wherein the P2P channel disconnection notification message comprises an identifier of the second electronic device;

and disconnecting the P2P channel between the second electronic equipment identified by the identifier in the P2P channel disconnection notification message.

6. The method of claim 5, wherein the method further comprises:

and modifying the state information corresponding to the identifier of the second electronic equipment in the IP address/identifier management table into disconnection.

7. The method of claim 5, wherein the method further comprises:

and deleting the identifier of the second electronic equipment recorded in the IP address/identifier management table, and state information and IP address corresponding to the identifier of the second electronic equipment.

8. The method according to any of claims 1 to 7, wherein the identifier of the first electronic device is a media access control, MAC, address.

9. The method according to any of claims 1 to 7, wherein the identifier of the first electronic device identifies a BSSID for a basic service group.

10. The method according to any one of claims 1 to 7, wherein the character information of the second electronic device is determined according to the product characteristics of the second electronic device.

11. A method of establishing a point-to-point channel, characterized by being applied to a second electronic device that exchanges an identifier identifying itself with a first electronic device during an authentication phase, the method comprising:

sending a point-to-point P2P channel establishment request message to a first electronic device, wherein the P2P channel establishment request message comprises an identifier of a second electronic device, role information and authentication information corresponding to a role, and the role information is used for indicating that the second electronic device is one party for creating a hot spot or one party for accessing the hot spot;

when the role information indicates that the second electronic device is one of access hotspots, receiving a P2P channel establishment response message sent by the first electronic device, wherein the P2P channel establishment response message comprises an identifier of the first electronic device and first hotspot information of the created hotspots;

verifying the identifier included in the P2P channel establishment response message according to the identifier of the first electronic device obtained in the authentication stage;

And when the verification is successful, a P2P channel between the second electronic equipment and the first hot spot information is established.

12. The method of claim 11, wherein the method further comprises:

and sending a P2P channel disconnection notification message to the first electronic device, wherein the P2P channel disconnection notification message comprises an identifier of the second electronic device.

13. The method according to claim 11 or 12, wherein the identifier of the second electronic device is a media access control, MAC, address.

14. The method according to claim 11 or 12, wherein the second electronic device identifies a BSSID for a basic service group.

15. An electronic device, characterized by being configured to perform a method of establishing a point-to-point channel according to any of claims 1 to 10 or to perform a method of establishing a point-to-point channel according to any of claims 11 to 14.

16. A co-operating system comprising a first electronic device for performing the method of establishing a point-to-point channel according to any of claims 1 to 10 and a second electronic device for performing the method of establishing a point-to-point channel according to any of claims 11 to 14.