CN115297356B

CN115297356B - Equipment interaction method, system and device

Info

Publication number: CN115297356B
Application number: CN202210753687.0A
Authority: CN
Inventors: 乔頔
Original assignee: Beijing Youku Technology Co Ltd
Current assignee: Beijing Youku Technology Co Ltd
Priority date: 2022-06-29
Filing date: 2022-06-29
Publication date: 2024-04-09
Anticipated expiration: 2042-06-29
Also published as: CN115297356A

Abstract

The embodiment of the application discloses a device interaction method, a device interaction system and a device interaction device. The main technical scheme comprises the following steps: the control side equipment acquires and records the computing capability information of the play side equipment; responding to an operation event of a user, and determining a calculation requirement required by control processing corresponding to the operation event; and if the computing capability of the playing side device does not meet the computing requirement, after the control side device or other devices except the control side device and the playing side device execute the control processing on the data to be processed aimed at by the operation event, the control side device sends the media data generated by the control processing to the playing side device so that the playing side device plays the media data. The method and the device can ensure that the playing side equipment normally plays the media data.

Description

Equipment interaction method, system and device

Technical Field

The present disclosure relates to the field of computer applications, and in particular, to a device interaction method, system, and apparatus.

Background

With the continuous development of computer application technology, multi-device interaction technology such as screen projection has been widely applied to the production and life of people. For example, a user may display media data by projecting it on a mobile terminal to a device having a larger screen, such as a smart television, computer, or the like. The device responsible for the screen-on control function (for example, the mobile terminal) is referred to as a control-side device, and the device that receives the screen-on control (for example, the device such as the smart television or the computer) is referred to as a play-side device.

However, in some application scenarios, it often occurs that the play-side device cannot normally play the media data of the control-side device due to the limited performance.

Disclosure of Invention

In view of this, the present application provides a device interaction method, system and apparatus, which are used to solve the problem that a playback device cannot normally play media data of a control device due to performance limitation.

The application provides the following scheme:

in a first aspect, the present application provides a device interaction method, the method comprising:

the control side equipment acquires and records the computing capability information of the play side equipment;

responding to an operation event of a user, and determining a calculation requirement required by control processing corresponding to the operation event;

and if the computing capability of the playing side device does not meet the computing requirement, after the control side device or other devices except the control side device and the playing side device execute the control processing on the data to be processed aimed at by the operation event, the control side device sends the media data generated by the control processing to the playing side device so that the playing side device plays the media data.

According to an implementation manner in the embodiments of the present application, the obtaining and recording, by the control side device, the computing capability of the play side device includes:

After the control side equipment is accessed to a local area network, inquiring the computing capability information of the play side equipment in the local area network in a broadcasting mode; or,

after the control side equipment is accessed to a local area network, acquiring computing capability information registered by the play side equipment in the management equipment from the management equipment in the local area network; or,

acquiring computing capability information of the playing side equipment in the process of establishing connection between the control side equipment and the playing side equipment; or,

after the control side device establishes connection with the play side device, the computing capability information of the play side device is obtained through a heartbeat mechanism.

According to an achievable manner in the embodiment of the present application, the performing, by the other devices except the control-side device and the play-side device, the control processing includes:

the control side equipment establishes connection with the other equipment;

transmitting the data to be processed and the information of the control processing to the other equipment;

and acquiring media data generated after the other equipment executes the control processing on the data to be processed.

According to an implementation manner of the embodiments of the present application, the establishing connection between the control side device and the other device includes:

The control side device acquires the computing capability information and the address information registered by the other devices in the management device from the management device in the local area network, and if the registered computing capability information meets the computing requirement, the control side device sends a connection request to the other devices by using the address information and establishes connection with the other devices; or,

and the control side equipment inquires the computing capability information and the address information of other equipment in the local area network in a broadcasting mode, and if the computing capability of the other equipment meets the computing requirement, the address information is used for sending a connection request to the other equipment and establishing connection with the other equipment.

According to an implementation manner in the embodiment of the present application, if the computing capability of the playing side device meets the computing requirement, the data to be processed and the information of the control process are sent to the playing side device, and the playing side device performs the control process on the data to be processed and plays the data.

According to an implementation manner in the embodiment of the present application, the computing capability information includes: the type of the provided computing service and the remaining state of the computing resources.

According to an implementation manner in the embodiments of the present application, the control processing includes decoding, synthesizing, editing, clipping, format conversion or view switching of media data in a projection screen playing scene.

In a second aspect, the present application provides a device interaction method, the method comprising:

the method comprises the steps that a play side device provides calculation capability information of the play side device for a control side device, so that when calculation capability of the play side device does not meet calculation requirements required by control processing corresponding to an operation event of a user, the control side device or other devices except the control side device and the play side device execute the control processing on data to be processed, which are targeted by the operation event;

and receiving and playing the media data generated by the control processing and sent by the control side equipment.

According to an implementation manner of the embodiments of the present application, the providing, by the playback-side device, the computing capability information of the playback-side device to the control-side device includes:

responding to the broadcast inquiry of the control side equipment, and sending the computing capability information of the play side equipment to the control side equipment; or,

the playing side device registers the computing capability information of the playing side device with a management device in a local area network so that the control side device can inquire the computing capability information of the playing side device from the management device; or,

In the process of establishing connection between the playing side device and the control side device, computing capability information of the playing side device is provided for the control side device; or,

after the playing side device establishes connection with the control side device, the computing capability information of the playing side device is sent to the control side device through a heartbeat mechanism.

In a third aspect, the present application provides a device interaction system, including a control side device and a play side device;

the control side device is configured to acquire and record computing capability information of the play side device; responding to an operation event of a user, and determining a calculation requirement required by control processing corresponding to the operation event; if the computing capability of the playing side device does not meet the computing requirement, after the control processing is executed on the data to be processed aimed at by the operation event, media data generated by the control processing is sent to the playing side device;

the play-side device is configured to play media data from the control-side device.

In a fourth aspect, the present application provides a device interaction system, where the system includes a control side device, a play side device, and other devices;

The control side device is configured to acquire and record computing capability information of the play side device; responding to an operation event of a user, and determining a calculation requirement required by control processing corresponding to the operation event; if the calculation capability of the playing side equipment does not meet the calculation requirement, sending the data to be processed and the information of the control processing, which are aimed at by the operation event, to the other equipment;

the other equipment is configured to control the data to be processed according to the information of the control processing to obtain media data, and send the media data to the control side equipment;

the control side device is further configured to send the media data to the play side device;

In a fifth aspect, the present application provides an apparatus interaction device, provided in a control side apparatus, including:

a capability acquisition unit configured to acquire and record calculation capability information of the playback-side apparatus;

a requirement determining unit configured to determine, in response to an operation event of a user, a calculation requirement required for control processing corresponding to the operation event;

A control processing unit configured to execute the control processing on the data to be processed for which the operation event is directed, or to provide the data to be processed to other devices except the control side device and the play side device, if the calculation capability of the play side device does not satisfy the calculation requirement;

and a first communication unit configured to transmit the media data generated by the control process to the playback-side apparatus so that the playback-side apparatus plays back the media data.

In a sixth aspect, the present application provides an apparatus interaction device, provided in a playback-side apparatus, including:

a capability providing unit configured to provide calculation capability information of the playback side device to a control side device so that the control side device performs, when calculation capability of the playback side device does not satisfy calculation requirements required for control processing corresponding to an operation event of a user, the control processing on data to be processed for the operation event by the control side device or other devices other than the control side device and the playback side device;

a reception processing unit configured to receive media data generated by the control process transmitted by the control-side device;

And the play processing unit is configured to play the media data.

According to a seventh aspect, there is provided a computer readable storage medium having stored thereon a computer program which when executed by a processor performs the steps of the method of any of the first aspects described above.

According to an eighth aspect, there is provided an electronic apparatus, comprising:

one or more processors; and

a memory associated with the one or more processors, the memory for storing program instructions that, when read for execution by the one or more processors, perform the steps of the method of any of the first aspects above.

According to the specific embodiment provided by the application, the application can have the following technical effects:

1) The control side equipment acquires and records the calculation capability information of the play side equipment, and if the calculation capability of the play side equipment does not meet the calculation requirement required by the control processing corresponding to the user operation event, the control side equipment or other equipment executes the control processing on the data to be processed and then sends the generated fluid stream to the play side equipment, so that the play side equipment can play the media data normally.

2) The method and the device can improve the playing capability of the playing side device to the media data by means of the computing capability of the control side device or other devices except the control side device and the playing side device.

3) The method and the device provide various ways for acquiring the computing capability information of the playing side device and other devices, so that the realization is richer and more flexible.

Of course, not all of the above-described advantages need be achieved at the same time in practicing any one of the products of the present application.

Drawings

In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings that are needed in the embodiments will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present application, and that other drawings can be obtained according to these drawings without inventive effort for a person skilled in the art.

FIG. 1 illustrates an exemplary system architecture diagram to which embodiments of the present application may be applied;

fig. 2 is a flowchart of a method executed by a control side device according to an embodiment of the present application;

fig. 3 is a flowchart of a method executed by a playback side device according to an embodiment of the present application;

fig. 4 is a schematic diagram of a basic architecture in a screen-projection application scenario provided in an embodiment of the present application;

FIG. 5 is a flowchart of a method for video delivery according to an embodiment of the present application;

FIG. 6 is a flowchart of another method for video delivery according to an embodiment of the present application;

fig. 7 is a flowchart of a method for playing high definition VR video by a VR device according to an embodiment of the present application;

FIG. 8 shows a schematic block diagram of a device interaction apparatus according to one embodiment;

FIG. 9 shows a schematic block diagram of a device interaction apparatus according to another embodiment;

fig. 10 illustrates an exemplary architecture of an electronic device.

Detailed Description

The following description of the embodiments of the present application will be made clearly and fully with reference to the accompanying drawings, in which it is evident that the embodiments described are only some, but not all, of the embodiments of the present application. All other embodiments obtained by a person of ordinary skill in the art based on the embodiments in the present application are within the scope of the protection of the present application.

The terminology used in the embodiments of the invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. As used in this application and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise.

It should be understood that the term "and/or" as used herein is merely one relationship describing the association of the associated objects, meaning that there may be three relationships, e.g., a and/or B, may represent: a exists alone, A and B exist together, and B exists alone. In addition, the character "/" herein generally indicates that the front and rear associated objects are an "or" relationship.

Depending on the context, the word "if" as used herein may be interpreted as "at … …" or "at … …" or "in response to a determination" or "in response to detection". Similarly, the phrase "if determined" or "if detected (stated condition or event)" may be interpreted as "when determined" or "in response to determination" or "when detected (stated condition or event)" or "in response to detection (stated condition or event), depending on the context.

FIG. 1 illustrates an exemplary system architecture to which embodiments of the present application may be applied. As shown in fig. 1, there may be a plurality of terminal devices, such as smartphones, smart televisions, tablet computers, PCs, laptops, etc., in the same local area network. The user can take one of the devices convenient to operate as a control end device, for example, a smart phone as the control end device, or a tablet computer as the control end device, and the like; one of the devices convenient to watch is used as a playing end device, for example, a smart television with a larger screen is used as the playing end device, or a PC is used as the playing end device, and the like. In the system shown in fig. 1, only a smart phone is used as a control end device, a smart television is used as a playing end device, and other devices such as a PC and a portable computer exist in the local area network.

The terminal devices are interconnected through a switching device such as a router to form a local area network. The terminal devices can be accessed to the local area network in a wired or wireless mode, and the current common mode is to access the local area network in a WiFi mode. For example, each terminal device is networked by WiFi within the effective range of the wireless network coverage generated by the wireless router. In addition, the terminal devices may be interconnected by other wireless means such as bluetooth. Applications supporting multi-device interaction, such as a screen throwing application, a communication application, and the like, and other applications such as a voice interaction application, a web browser application, and the like, are installed and run on each terminal device, which is not limited in this application.

The device interactions between the control side device and the playback side device may be implemented based on a variety of protocols including, but not limited to: air play), DNLA (Digital Living Network Alliance ), and Miracast.

Wherein, the AirPlay is mainly used for supporting the iOS device, which basically supports the function, and iPhone, iPad, macBook transmits pictures, audio, video and the like to the display device supporting the AirPlay through WiFi. Miracast was established by Wi-Fi alliance in 2012, a Wi-Fi Direct (Wi-Fi Direct) based wireless display standard, and 3C devices supporting this standard can share video or images wirelessly. DLNA uses universal plug and play (UPnP) protocol to accomplish management, discovery and control of media, and aims to solve the interconnection and interworking of wireless networks and wired networks including PCs (Personal Computer, personal computers), intelligent appliances, mobile devices, etc.

It should be understood that the number of various devices in fig. 1 is merely illustrative. There may be any number of control side devices, playback side devices, and other devices as desired for implementation.

Fig. 2 is a main flowchart of a device interaction method provided in an embodiment of the present application, where the method flowchart may be executed by a control end device in the system shown in fig. 1. As shown in fig. 2, the method may include the steps of:

step 202: the control side device acquires and records the computing capability information of the play side device.

Step 204: in response to an operation event of a user, a calculation requirement required for control processing corresponding to the operation event is determined.

Step 206: judging whether the computing capability of the playing side equipment meets the computing requirement, if not, executing step 208; if so, step 210 is performed.

Step 208: after the control-side device or other devices except the control-side device and the playing-side device execute the control processing on the data to be processed aimed at by the operation event, the control-side device sends the media data generated by the control processing to the playing-side device so that the playing-side device plays the media data.

Step 210: the data to be processed and the control processing information are sent to the playing side equipment, and the playing side equipment plays the data to be processed after the control processing.

As can be seen from the above flow, the control side device obtains and records the calculation capability information of the play side device, and if the calculation capability of the play side device does not meet the calculation requirement required by the control processing corresponding to the user operation event, the control side device or other devices execute the control processing on the data to be processed, and then send the generated fluid stream to the play side device, so as to ensure that the play side device can play the media data normally.

Each step in the above-described flow is described in detail below. The above step 202, i.e. "the control side device acquires and records the computing capability information of the playback side device" will be described in detail with reference to the embodiment.

The control side device may acquire the computing power information of the play side device in the following ways:

the first way is: after the control side equipment is accessed to the local area network, the computing capability information of the play side equipment in the local area network is queried in a broadcasting mode.

After the control side device accesses the local area network, the control side device can inquire the device which provides the specific service in the local area network through SSDP (Simple Service Discovery Protocol ). For example, the control-side device may broadcast a query request within the lan, where the query request includes the queried device type information or service type information. If the equipment receiving the query request accords with the equipment type information or can provide the service with the consistent queried service type information, responding to the query request, returning a response message and carrying the computing capability information of the equipment in the response message.

The control side device can record the calculation capability information returned by each device, and the calculation capability information of the play side device can be directly obtained after the play side device is selected subsequently.

Further, the calculation capability information may be referred to when the control side device selects the play side device. For example, when the control side device obtains the screen throwing operation of the user, all the device information which can be used as the playing side device in the local area network is displayed to the user, and the computing capacity information of each device is displayed for the user to refer to.

Wherein the computing capability information of the device may be determined by the type of computing service the device is provided with and the remaining status of the computing resources. For example, depending on the remaining state of the current CPU, GPU, memory, etc.

As one of the realizations, different computing resource conditions may be pre-assigned to different levels of computing power, e.g., divided into high computing power, low computing power, etc. And then determining the corresponding computing capacity grade according to the residual condition of the current computing resource of the equipment, and recording the computing capacity grade and the type of the computing service of each equipment in the local area network by the control side equipment.

As another implementation manner, the remaining status of the computing resources and the type of computing service of the device, such as how much CPU, GPU, memory, etc., remain may also be recorded directly.

The type of computing service that the device is provided with may be determined by the type of application installed on the device, such as whether a particular video composition application is installed, whether a particular video decoding application is installed, and so forth.

Other forms of computing capability information may also be employed, and are not intended to be exhaustive. The form of the computing capability information is applicable in other subsequent implementations, and will not be described in detail.

The second way is: after the control side device accesses the local area network, the computing capability information registered by the play side device in the management device is acquired from the management device in the local area network.

One device may be selected as a management device in a local area network, and all devices may register the computing capability information of the device with the management device, and may acquire the computing capability information of the other devices from the management device.

For example, the management device may be a switching device such as a router, or may be a control-side device designated by the user.

As one of the possible ways, when all devices access the local area network, the computing capability information of the device is registered with the management device in the local area network.

As another implementation, the type of computing service and the remaining status of resources that a device has vary with the application running on the device. Thus, each device in the local area network may also register the computing power information of the device with the management device in the local area network where it is located periodically. The management device can update the recorded computing capability information of other devices in real time, and at the same time, the management device can record the own computing capability information of the management device.

In addition, the control side device can acquire the computing power information registered by the playing side device in the management device from the management device in the local area network before selecting the playing side device, and can directly acquire the computing power information of the playing side device after selecting the playing side device. Further, the calculation capability information may be referred to when the control side device selects the play side device. For example, when the control side device obtains the screen throwing operation of the user, all the device information which can be used as the playing side device in the local area network is displayed to the user, and the computing capacity information of each device is displayed for the user to refer to.

Alternatively, the control-side device may acquire, from the management device in the local area network, the calculation capability information registered by the playback-side device at the management device after the playback-side device is selected.

Third mode: and in the process of establishing connection between the control side equipment and the playing side equipment, acquiring the computing capability information of the playing side equipment.

The control-side device establishes a connection with the playback-side device, typically after the playback-side device is selected. For example, the user performs a screen-casting search on the control-side device, and the control-side device searches for device information providing a screen-casting service in the local area network and presents the device information to the user in the form of a list, from which the user selects one device as the playback-side device. For another example, the user operates on the control side device to throw a screen, and the control side device uses the default device set by the user as the play side device.

After the play side device is selected, a connection request can be sent to the play side device under the triggering of a user or automatically, and connection is established with the play side device through a handshake mechanism. In the connection establishment process, the playing side device can carry the computing capability information of the playing side device through the message sent to the control side device in the handshake process.

Fourth mode: after the control side device establishes connection with the playing side device, the computing capability information of the playing side device is obtained through a heartbeat mechanism.

After the control side device and the playing side device are connected, the computing capability information can be inquired from the playing side device through the heartbeat message which is sent periodically. After receiving the heartbeat message from the control side device, the play side device acquires own computing capacity information in real time and returns the computing capacity information to the control side device through heartbeat response. The control side device updates the recorded computing capability information of the playing side device by using the received computing capability information, so that the recorded computing capability information of the playing side device is ensured to have certain real-time performance.

The above step 204, i.e. "in response to an operation event of a user, determining a calculation requirement required for a control process corresponding to the operation event" will be described in detail with reference to the embodiments.

The user can control the media data played by the playing side device by operating the control side device. The user may operate the control-side device by inputting a control command, triggering a control component, or the like, or by inputting a voice control command, or the like, in the user UI.

The operation event of the user generally corresponds to a certain control process, and the control process in the embodiment of the present application refers to: the processing mode of the media data corresponding to the media control requirement generated by the operation event of the user. Some of which are relatively simple control processes such as start of play, pause of play, double-speed play, forward, reverse, etc. These control processes typically require lower computational requirements. Yet others are relatively complex control processes such as decoding, compositing, editing, clipping, format conversion, view switching, etc. of media data. These control processes typically require high computational requirements.

The control terminal device can estimate the calculation requirement required by the control processing by using the type of the control processing corresponding to the operation event and the data quantity of the data to be processed. The computational requirements may be embodied in the type of computational service required and the status of the computational resources, such as how much CPU, GPU, memory, network bandwidth, etc. are required to be occupied. The usage of the computing resources may also be associated with different levels of computing power, e.g., divided into high computing requirements, low computing requirements, etc.

This enables a determination to be made in step 206 as to whether the computing power of the playback-side device meets the computing requirements. For example, if the playback-side apparatus has a computing service type required for the control process and the remaining computing resources reach the computing resources required for the control process, the computing requirements are considered to be satisfied. For another example, if the playback-side device has a type of computing service required for the control process and the computing capability level is higher than or equal to the computing requirement level corresponding to the control process, the computing requirement is considered to be satisfied, and so on.

The following describes in detail the above step 208, that is, "after the control-side device or the other devices except the control-side device and the playback-side device perform the control processing on the data to be processed, the control-side device sends the media data generated by the control processing to the playback-side device so that the playback-side device plays the media data" with reference to the embodiment.

As one of the realizable modes, when the control side device determines that the computing capability of the play side device does not meet the computing requirement of the control processing, the control side device may send the media data generated by the control processing to the play side device after performing the control processing on the data to be processed by the control side device.

Here, the user operation event is directed to media data, and accordingly, the control process corresponding to the user operation event is also directed to the media data. In the embodiment of the present application, media data targeted by a user operation event is referred to as pending data. And performing control processing on the data to be processed to influence the media data played by the playing side equipment. For example, the data to be processed may be media data to be played by the playback-side apparatus, such as decoding processing, clipping processing, view switching, and the like, for the media data to be played. The data to be processed may also be data input by the user and media data to be played, for example, the user inputs a barrage text, the control process is to synthesize the barrage text into the media data to be played, and the video played by the playing side device in future may include the barrage text input by the user. The source of the data to be processed may be various, for example, the data to be processed may be media data acquired from a media server, media data acquired from other user terminal devices, media data locally stored in the control side device, or even media data input by a user at the control side device as described in the above example. Etc.

As another implementation manner, the control processing may be performed on the data to be processed by other devices except the control side device and the play side device, and then the control side device acquires the media stream generated after the control processing and sends the media stream to the play side device.

Specifically, the following cases may be included, but are not limited thereto:

first case: the control side device may first determine whether the computing capability of the control side device satisfies the computing requirement of the control process, and if so, the control side device performs the control process on the data to be processed; otherwise, the control processing may be performed on the data to be processed by other devices than the control-side device and the playback-side device.

Second case: the control-side device may not execute any control processing on the data to be processed, and may execute the control processing on the data to be processed by a device other than the control-side device and the playback-side device when it is determined that the calculation capability of the playback-side device does not satisfy the calculation requirement of the control processing.

Third case: or may be determined according to a user setting, if the user setting prioritizes execution of the control processing by the control-side device, the control-side device may first judge whether the calculation capability of the control-side device satisfies the calculation requirement of the control processing, and if so, the control-side device executes the control processing on the data to be processed; otherwise, the control processing may be performed on the data to be processed by other devices than the control-side device and the playback-side device. If the user sets to prohibit the control-side device from executing the control process, in accordance with the second case, the control-side device gives to other devices than the control-side device and the play-side device to execute the control process on the data to be processed when it is determined that the calculation capability of the play-side device does not satisfy the calculation requirement of the control process.

In this implementation manner, the control side device may establish a connection with the other device described above, and send the data to be processed and the information of the control process to the other device; and then acquiring media data generated after the control processing of the data to be processed by other equipment.

As one of the possible modes, before the control side device establishes connection with other devices, the computing capability information and address information of other devices in the local area network can be queried in a broadcast mode, and one other device with computing capability meeting the above computing requirement can be selected. And then sending a connection request to the other device by using the address information of the other device and establishing connection with the other device.

As another possible way, the control-side device may acquire, from the management device in the local area network, the computing power information and the address information registered in the management device by the other device. For example, if the device supports the device interaction manner provided in the embodiments of the present application, the management device may register the type of computing service and the status of computing resources that can be provided by itself when joining the lan, for example, services such as decoding, synthesizing, editing, clipping, format conversion, and view switching of media data may be provided. If the computing service registered by some other device contains the control processing required by the control side device and the computing resource condition meets the computing requirement, the address information of the other device is utilized to send a connection request to the other device and establish connection with the other device.

Accordingly, for the playback-side device, the method flow executed by the playback-side device may be as shown in fig. 3, and includes the following steps:

step 302: the playing side device provides the computing capability information of the playing side device for the control side device, so that the control side device can execute control processing on the data to be processed by the control side device or other devices except the control side device and the playing side device when the computing capability of the playing side device does not meet the computing requirement of the control processing corresponding to the operation event of the user.

The manner in which the playback-side apparatus provides the control-side apparatus with the computing capability information may be, but is not limited to, the following:

the first way is: and transmitting the computing capability information of the playing side equipment to the control side equipment in response to the broadcast query of the control side equipment.

For example, the control-side device may broadcast a query request within the lan, where the query request includes the queried device type information or service type information. If the playing side equipment receiving the query request accords with the equipment type information or can provide the service with the same queried service type information, responding to the query request, returning a response message and carrying the computing capability information of the equipment in the response message.

The second way is: the playback-side device registers the calculation capability information of the playback-side device with the management device in the local area network for the control-side device to inquire the calculation capability information of the playback-side device from the management device.

One device may be selected as a management device in a local area network, and all devices may register the computing capability information of the device with the management device, and may acquire the computing capability information of the other devices from the management device. For example, the management device may be a switching device such as a router, or may be a control-side device designated by the user.

Third mode: in the process of establishing connection between the playing side device and the control side device, the computing capability information of the playing side device is provided for the control side device.

In the connection establishment process, the playing side device can carry the computing capability information of the playing side device through the message sent to the control side device in the handshake process.

Fourth mode: after the playing side device and the control side device are connected, the computing capability information of the playing side device is sent to the control side device through a heartbeat mechanism.

After the control side device and the playing side device are connected, the computing capability information can be inquired from the playing side device through the heartbeat message which is sent periodically. After receiving the heartbeat message from the control side device, the play side device acquires own computing capacity information in real time and returns the computing capacity information to the control side device through heartbeat response.

Step 304: and receiving and playing the media data generated by the control processing sent by the control side equipment.

If the computing capability of the playing side device meets the computing requirement, the playing side device receives the data to be processed and the information of the control processing sent by the control side device, and at the moment, the playing side device plays the data to be processed after the control processing.

The computing power information includes: the type of the provided computing service and the remaining state of the computing resources.

The following describes the above method flow by way of example, taking several specific application scenarios as examples. For facilitating understanding of application scenarios, first, a basic architecture of each application scenario is briefly introduced. As shown in fig. 4, the data source is responsible for providing various media data. Taking a screen projection as an example, the data source may be video data provided by a server, and the server may be a cloud server. The cloud server is also called a cloud computing server or a cloud host, and is a host product in a cloud computing service system, so as to solve the defects of large management difficulty and weak service expansibility in the traditional physical host and virtual special server (VPs, virtual Private Server) service.

The control-side device has a function of downloading media data, a function of enabling control processing of video data, a function of storing processed data, a UI interaction function, a function of interacting with a device in a local area network as a control point (DLNA control point is exemplified in the figure), a network communication function (e.g., a service socket, etc.).

The playback-side apparatus includes a function (DLNA function is exemplified in the figure) of communicating with the control-side apparatus, a decoding function, a rendering function, and the like.

Negotiation and interaction of SSDP, GENA, SOAP protocol and data interaction can be performed between the control side device and the play side device.

Application scenario 1: and the user inputs the multi-channel video data to the intelligent television through the mobile phone.

Fig. 5 is a flowchart of a method for video projection provided in the embodiment of the present application, where in the method, a mobile phone is taken as a control side device, and an intelligent television is taken as a play side device as an example. As shown in fig. 5, the method may include the steps of:

step 502: and the mobile phone starts the screen-throwing operation in response to the trigger of the user, and displays a screen-throwing control interface.

Step 504: the mobile phone searches the equipment supporting the screen throwing playing in the local area network, and displays the searched equipment information to the user in a list form.

Step 506: the mobile phone responds to the user to select the intelligent television from the list as playing equipment, establishes connection with the intelligent television, and acquires the computing capability information of the intelligent television in the connection establishment process.

Step 508: the mobile phone determines the video selected to be played by the user as a plurality of paths of video data, estimates the calculation requirement for decoding the plurality of paths of video, and decodes the video stream by the mobile phone after determining that the calculation capability of the intelligent television does not meet the calculation requirement.

In general, a certain computing power is required for decoding multiple paths of video, for example, an H264 decoding function is required, and computing resources meet computing requirements, if the smart television does not have the H264 decoding function or the remaining conditions of the computing resources do not meet the computing requirements, and the mobile phone can meet the computing requirements, the mobile phone decodes the video stream.

Step 510: and the mobile phone transmits the video stream obtained after the decoding processing to the intelligent television.

Step 512: and the intelligent television plays the received video stream.

Application scenario 2: and the user throws a free visual angle video to the intelligent television through the mobile phone and performs visual angle switching.

Fig. 6 is a flowchart of another method for video projection provided in the embodiment of the present application, where in the method, a mobile phone is taken as a control side device, and an intelligent television is taken as a play side device as an example. As shown in fig. 6, the method may include the steps of:

step 602: and the mobile phone starts the screen-throwing operation in response to the trigger of the user, and displays a screen-throwing control interface.

Step 604: the mobile phone searches the equipment supporting the screen throwing playing in the local area network, and displays the searched equipment information to the user in a list form.

Step 606: the mobile phone responds to the user to select the intelligent television from the list as playing equipment, establishes connection with the intelligent television, and acquires the computing capability information of the intelligent television in the connection establishment process.

Step 608: and the mobile phone transmits the video stream selected by the user to be played by the screen to the intelligent television for playing.

Step 610: and the mobile phone responds to the operation of triggering the visual angle switching by a user, and estimates the calculation requirement required by the visual angle switching.

Step 612: after the mobile phone determines that the computing capacity of the intelligent television does not meet the computing requirement and the computing capacity of the mobile phone also does not meet the computing requirement, other devices with the computing capacity meeting the computing requirement in the local area network are queried in a broadcasting mode.

With the continuous improvement of video processing technology and network transmission technology, researchers have developed a free view technology. A circle of synchronous acquisition cameras are deployed on a shooting site, and the pictures on the site are acquired and shot in an omnibearing manner and then uploaded to a cloud end. And carrying out three-dimensional reconstruction on the acquired video at the cloud to synthesize a free view video. The freeview video is different from the conventional video, each frame contains texture information and depth information of a photographing view, and user interaction input is required as a basis for transforming the view. Therefore, the terminal device needs to have a strong computing power to process the freeview video according to the view information selected by the user for switching, including parsing the texture depth map, preprocessing and mapping the depth map, fusing the multi-camera mapping texture map, and so on. In this embodiment, if the mobile phone does not meet the calculation requirement, the PC installed with a specific application (for example, a video processing application having the above-mentioned processing function) and having the remaining state of the calculation resource meeting the calculation requirement may perform the above-mentioned processing to complete the view angle switching, so as to obtain the video stream after the switching.

Step 614: and the mobile phone establishes connection with the PC.

Step 616: and the mobile phone sends the received video stream and the received visual angle switching information to the PC.

Step 618: and after the PC performs view angle switching processing on the video stream, the generated video stream is sent to the mobile phone.

Step 620: the mobile phone transmits the video stream to the smart television.

Step 622: and the intelligent television plays the received video stream.

Application scenario 3: VR devices play high definition VR videos.

Fig. 7 is a flowchart of a method for playing high-definition VR video by using VR device provided in an embodiment of the present application, where a VR device processor is used as a control side device and VR glasses are used as a playing side device. As shown in fig. 7, the method may include the steps of:

step 702: after the VR equipment joins the local area network, the computing power information and the address information registered by each equipment are acquired from a router in the local area network.

Step 704: responding to the operation of triggering and playing the high-definition VR video by a user, determining that the processor and the VR eyes of the VR equipment do not have the decoding capability of the high-definition VR video, and inquiring that the PC in the local area network is registered with the decoding function of the high-definition VR video and the residual condition of the computing resource meets the computing requirement of playing the high-definition VR video.

Step 706: the processor of the VR device establishes a connection with the PC.

Step 708: the processor of the VR device provides high definition VR video and decoding information to the PC.

Step 710: and decoding the high-definition VR video by the PC, and transmitting the VR video stream obtained after decoding to a processor of the VR device.

Step 712: the processor of the VR device sends the VR video stream to the VR glasses.

Step 714: VR glasses play VR video streams.

By the embodiment, some VR devices with lower performance can have higher playing capability by virtue of other devices in the local area network.

According to the embodiment of the method, compared with the traditional push stream type screen projection, the scheme related to the embodiment of the application has higher media playing capability. The traditional plug-in screen projection is that a control side device sends a media link to be played to a play side device, and the play side device accesses the link and plays the media. If the required media playing capability exceeds the computing capability of the playing side device, normal playing cannot be performed, or the computing requirement required by the control requirement of the user on the media in the media playing process exceeds the computing capability of the playing side device, and the control requirement cannot be normally realized. However, after the control side device or other devices with enough computing power are adopted to control the data to be processed, the scheme provided by the embodiment of the application can send the media data generated by the control process to the playing side device, so that the media data can be normally played, and the control requirement of a user on the media is met.

In addition, compared with the traditional mirror image type screen projection, the scheme related to the embodiment of the application does not need to be truly played at the control side equipment. The traditional mirror image type screen projection is that the control side equipment records the screen while playing the media data, and the recorded screen data is transmitted to the play side equipment for playing, so that the control side equipment is required to play the media data truly, and the whole process can influence other use requirements of a user on the control side equipment. The method provided by the embodiment of the application only needs the control side equipment to send the media data generated by the control processing to the play side equipment, and does not need to play the media data truly, so that the method is more convenient and does not influence the normal use of the control side equipment by a user.

The foregoing describes specific embodiments of the present disclosure. Other embodiments are within the scope of the following claims. In some cases, the actions or steps recited in the claims can be performed in a different order than in the embodiments and still achieve desirable results. In addition, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

According to an embodiment of another aspect, a device interaction apparatus is provided. Fig. 8 shows a schematic block diagram of a device interaction apparatus according to one embodiment, which is provided to a control side device in the architecture shown in fig. 1. As shown in fig. 8, the apparatus 800 includes: the capability acquisition unit 801, the requirement determination unit 802, the control processing unit 803, and the first communication unit 804 may further include a second communication unit 805. Wherein the main functions of each constituent unit are as follows:

the capability acquisition unit 801 is configured to acquire and record calculation capability information of the playback-side apparatus.

A requirement determination unit 802 configured to determine, in response to an operation event of a user, a calculation requirement required for control processing corresponding to the operation event.

The control processing unit 803 is configured to execute control processing on the data to be processed if the calculation capability of the playback-side device does not satisfy the calculation requirement, or execute control processing on the data to be processed for which the operation event is directed by other devices than the control-side device and the playback-side device.

The first communication unit 804 is configured to transmit the media data generated by the control process to the playback-side apparatus so that the playback-side apparatus plays back the media data.

The capability acquiring unit 801 may be specifically configured to: after the control side equipment is accessed to the local area network, inquiring the computing capability information of the play side equipment in the local area network in a broadcasting mode; or,

after the control side equipment is accessed to the local area network, acquiring the computing capability information registered by the play side equipment in the management equipment from the management equipment in the local area network; or,

in the process of establishing connection between control side equipment and play side equipment, acquiring computing capability information of the play side equipment; or,

after the control side device establishes connection with the playing side device, the computing capability information of the playing side device is obtained through a heartbeat mechanism.

A second communication unit 805 configured to establish a connection with other devices; transmitting the data to be processed and the information for controlling the processing to other devices; and acquiring media data generated after the other equipment performs control processing on the data to be processed.

As one of the realizable modes, when the second communication unit 805 establishes a connection with other devices, the second communication unit 805 obtains, from the management device in the local area network, the computing capability information and the address information registered in the management device by the other devices, and if the registered computing capability information satisfies the computing requirement, the second communication unit 805 sends a connection request to the other devices by using the address information and establishes a connection with the other devices.

As another implementation manner, when the second communication unit 805 establishes a connection with another device, the computing capability information and the address information of the other device in the local area network are queried in a broadcast manner, and if the computing capability of the other device meets the computing requirement, the second communication unit 805 sends a connection request to the other device by using the address information and establishes a connection with the other device.

The first communication unit 804 is further configured to: and if the computing capability of the playing side equipment meets the computing requirement, sending the data to be processed and the information for controlling the processing to the playing side equipment, and playing the data after the playing side equipment controls the data to be processed.

The above-mentioned calculation capability information includes: the type of the provided computing service and the remaining state of the computing resources.

As a typical application scenario, the control process includes decoding, synthesizing, editing, format conversion, or view switching of media data in a cast-in-place play scenario.

Fig. 9 shows a schematic block diagram of a device interaction apparatus according to another embodiment, which is arranged at a play-side device in the architecture shown in fig. 1. As shown in fig. 9, the apparatus 900 includes: a capability providing unit 901, a reception processing unit 902, and a play processing unit 903. Wherein the main functions of each constituent unit are as follows:

A capability providing unit 901 configured to provide the control-side device with the calculation capability information of the playback-side device so that the control-side device performs, when the calculation capability of the playback-side device does not satisfy the calculation requirement required for the control processing corresponding to the operation event of the user, the control processing on the data to be processed for the operation event by the control-side device or by other devices than the control-side device and the playback-side device.

The reception processing unit 902 is configured to receive media data generated by control processing transmitted by the control-side device.

A play processing unit 903 configured to play the media data.

Wherein the capability providing unit 901 may be specifically configured to: responding to the broadcasting inquiry of the control side equipment, and sending the computing capability information of the play side equipment to the control side equipment; or,

the playing side device registers the computing capability information of the playing side device with the management device in the local area network, so that the control side device inquires the computing capability information of the playing side device from the management device; or,

in the process of establishing connection between the playing side equipment and the control side equipment, the computing capability information of the playing side equipment is provided for the control side equipment; or,

After the playing side device and the control side device are connected, the computing capability information of the playing side device is sent to the control side device through a heartbeat mechanism.

Further, if the computing capability of the playback device meets the computing requirement, the receiving processing unit 902 receives the data to be processed and the information of the control process sent by the control device, and at this time, the playback processing unit 903 performs the control process on the data to be processed and plays the data.

In addition, the embodiment of the application further provides a computer readable storage medium, on which a computer program is stored, which when executed by a processor, implements the steps of the method of any one of the foregoing method embodiments.

And an electronic device comprising:

one or more processors; and

a memory associated with the one or more processors for storing program instructions that, when read for execution by the one or more processors, perform the steps of the method of any of the preceding method embodiments.

Fig. 10 illustrates an architecture of an electronic device, which may include a processor 1010, a video display adapter 1011, a disk drive 1012, an input/output interface 1013, a network interface 1014, and a memory 1020, among others. The processor 1010, the video display adapter 1011, the disk drive 1012, the input/output interface 1013, the network interface 1014, and the memory 1020 may be communicatively connected by a communication bus 1030.

The processor 1010 may be implemented by a general-purpose CPU, a microprocessor, an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), or one or more integrated circuits, etc. for executing related programs to implement the technical solutions provided herein.

The Memory 1020 may be implemented in the form of ROM (Read Only Memory), RAM (RandomAccess Memory ), static storage device, dynamic storage device, or the like. The memory 1020 may store an operating system 1021 for controlling the operation of the electronic device 1000, and a Basic Input Output System (BIOS) 1022 for controlling the low-level operation of the electronic device 1000. In addition, web browser 1023, data storage management system 1024, device interaction means 1025, and the like may also be stored. The device interaction means 1025 may be an application program for implementing the operations of the foregoing steps in the embodiment of the present application. In general, when implemented in software or firmware, the relevant program code is stored in memory 1020 and executed by processor 1010.

The input/output interface 1013 is used to connect with an input/output module to realize information input and output. The input/output module may be configured as a component in a device (not shown) or may be external to the device to provide corresponding functionality. Wherein the input devices may include a keyboard, mouse, touch screen, microphone, various types of sensors, etc., and the output devices may include a display, speaker, vibrator, indicator lights, etc.

The network interface 1014 is used to connect communication modules (not shown) to enable communication interactions of the device with other devices. The communication module may implement communication through a wired manner (such as USB, network cable, etc.), or may implement communication through a wireless manner (such as mobile network, WIFI, bluetooth, etc.).

Bus 1030 includes a path to transfer information between components of the device (e.g., processor 1010, video display adapter 1011, disk drive 1012, input/output interface 1013, network interface 1014, and memory 1020).

It is noted that although the above-described devices illustrate only the processor 1010, video display adapter 1011, disk drive 1012, input/output interface 1013, network interface 1014, memory 1020, bus 1030, etc., the device may include other components necessary to achieve proper operation in an implementation. Furthermore, it will be understood by those skilled in the art that the above-described apparatus may include only the components necessary to implement the present application, and not all the components shown in the drawings.

From the above description of embodiments, it will be apparent to those skilled in the art that the present application may be implemented in software plus a necessary general purpose hardware platform. Based on such understanding, the technical solutions of the present application may be embodied essentially or in a part contributing to the prior art in the form of a software product, which may be stored in a storage medium, such as a ROM/RAM, a magnetic disk, an optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, a server, or a network device, etc.) to perform the methods described in the embodiments or some parts of the embodiments of the present application.

In this specification, each embodiment is described in a progressive manner, and identical and similar parts of each embodiment are all referred to each other, and each embodiment mainly describes differences from other embodiments. In particular, for a system or system embodiment, since it is substantially similar to a method embodiment, the description is relatively simple, with reference to the description of the method embodiment being made in part. The systems and system embodiments described above are merely illustrative, wherein the elements illustrated as separate elements may or may not be physically separate, and the elements shown as elements may or may not be physical elements, may be located in one place, or may be distributed over a plurality of network elements. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment. Those of ordinary skill in the art will understand and implement the present invention without undue burden.

The foregoing has outlined the detailed description of the preferred embodiment of the present application, and the detailed description of the principles and embodiments of the present application has been provided herein by way of example only to facilitate the understanding of the method and core concepts of the present application; also, as will occur to those of ordinary skill in the art, many modifications are possible in view of the teachings of the present application, both in the detailed description and the scope of its applications. In view of the foregoing, this description should not be construed as limiting the application.

Claims

1. The device interaction method is characterized by being applied to a scene that a control side device screens media data to a playing side device for playing, and the method comprises the following steps:

the control side equipment acquires computing capacity information registered by each play side equipment in a local area network in the management equipment from the management equipment in the local area network;

displaying the equipment information and the computing capability information of each play side equipment in the local area network so as to be used for a user to refer to and select the play side equipment;

2. The method according to claim 1, wherein the method further comprises:

3. The method according to claim 1, wherein the execution of the control process by the other devices than the control-side device and the playback-side device includes:

the control side equipment establishes connection with the other equipment;

4. A method according to claim 3, wherein the control-side device establishing a connection with the other device comprises:

5. The method according to any one of claims 1 to 4, wherein the computing capability information comprises: the type of the computing service and the residual state of the computing resources;

the control processing comprises decoding, synthesizing, editing, clipping, format conversion or visual angle switching of the media data in the screen-throwing playing scene.

6. The device interaction method is characterized by being applied to a scene that a control side device screens media data to a playing side device for playing, and the method comprises the following steps:

the method comprises the steps that a playing side device registers computing capability information of the playing side device with a management device in a local area network, so that the control side device can acquire and display the computing capability information of the playing side device from the management device for a user to select the playing side device to be used as a reference, and when the computing capability of the playing side device does not meet the computing requirement of control processing corresponding to an operation event of the user, the control side device or other devices except the control side device and the playing side device execute the control processing on data to be processed corresponding to the operation event;

7. The method of claim 6, wherein the method further comprises:

8. The device interaction system is characterized by being applied to a scene that a control side device screens media data to a playing side device for playing, and comprises the control side device and the playing side device;

the control side device is configured to acquire computing capability information registered by each play side device in the local area network at the management device from the management device in the local area network; displaying the equipment information and the computing capability information of each play side equipment in the local area network so as to be used for a user to refer to and select the play side equipment; responding to an operation event of a user, and determining a calculation requirement required by control processing corresponding to the operation event; if the computing capability of the playing side device does not meet the computing requirement, executing the control processing on the data to be processed aimed at by the operation event or executing the control processing on the data to be processed aimed at by other devices except the control side device and the playing side device, and then transmitting the media data generated by the control processing to the playing side device;

9. The device interaction device is arranged on a control side device, and is characterized in that the device interaction device is applied to a scene that the control side device throws media data to a playing side device for playing, and the device interaction device comprises:

a capability acquisition unit configured to acquire, from a management device in a local area network, calculation capability information registered by each playback-side device in the local area network at the management device; displaying the equipment information and the computing capability information of each play side equipment in the local area network so as to be used for a user to refer to and select the play side equipment;

10. The device interaction device is arranged on a playing side device, and is characterized in that the device interaction device is applied to a scene that the control side device throws media data to the playing side device for playing, and the device interaction device comprises:

a capability providing unit configured to register calculation capability information of the playback-side device with a management device in a local area network so that the control-side device acquires and presents the calculation capability information of the playback-side device from the management device for reference when a user selects the playback-side device, and when the calculation capability of the playback-side device does not satisfy calculation requirements required for control processing corresponding to an operation event of the user, the control processing is performed on data to be processed for the operation event by the control-side device or other devices than the control-side device and the playback-side device;

and the play processing unit is configured to play the media data.

11. A computer readable storage medium, on which a computer program is stored, characterized in that the program, when being executed by a processor, implements the steps of the method according to any one of claims 1 to 7.

12. An electronic device, comprising:

one or more processors; and

a memory associated with the one or more processors for storing program instructions that, when read for execution by the one or more processors, perform the steps of the method of any of claims 1 to 7.