CN113938633B

CN113938633B - Video call processing method and display device

Info

Publication number: CN113938633B
Application number: CN202010606735.4A
Authority: CN
Inventors: 高琨; 王光强; 孙浩然
Original assignee: Juhaokan Technology Co Ltd
Current assignee: Juhaokan Technology Co Ltd
Priority date: 2020-06-29
Filing date: 2020-06-29
Publication date: 2023-09-08
Anticipated expiration: 2040-06-29
Also published as: CN113938633A

Abstract

The application discloses a video call processing method and display equipment, comprising a first controller connected with a display and a sound player, and a second controller connected with the first controller and a server in a communication way; the second controller is configured to perform: controlling the second application to carry out video call in the form of a full-screen call interface according to the signaling session information received from the server; responding to the operation of switching the playing source, controlling the first controller to start the playing source, and enabling the first controller to control the display to display a video call interface in a thumbnail window mode on the upper layer of the interface of the playing source; and responding to the operation of switching the thumbnail window to the full-screen call interface, recording the ID of the playing source, wherein the ID of the playing source is used for enabling the display to redisplay the playing source when the user switches the full-screen call interface to the thumbnail window mode again or when the user hangs up the video call on the full-screen call interface. The application can realize accurate control of chat and play while watching, and improve user experience.

Description

Video call processing method and display device

Technical Field

The present application relates to the field of display devices, and in particular, to a video call processing method and a display device.

Background

Fig. 1 is a schematic structural diagram of a dual-system display device, which includes a first controller and a second controller, and the first controller and the second controller can communicate with each other through an interface module, such as HDMI (High Definition Multimedia Interface, high-definition multimedia interface), a network port, USB (Universal Serial Bus ), and the like. The first controller is respectively connected with the signal source, the display and the sound player and is used for playing the audio and video of the signal output by the signal source, wherein the signal source can be an external device such as a set top box, or the signal source can be a second controller.

For the display device with the dual system structure, the video call application is generally preset to the second controller, and the related application in the second controller is mainly used for processing the logic of the video call, so that the user can join the video call with the identity of the initiator or the invitee, and therefore, the second controller can also be used as a signal source of the first controller. The signal source externally connected with the first controller and the application program installed in the second controller can be used as the playing source of the display device. However, in practical applications, the logic of processing the video call of the dual-system display device cannot bring better experience to the user due to the limitation of its own structure.

Disclosure of Invention

In order to solve the technical problems, the application provides a video call processing method and display equipment.

In a first aspect, the present application provides a display apparatus comprising:

a display;

a sound player;

the communicator is used for being in communication connection with the server;

a user input interface for receiving an operation input by a user;

a first controller connected with the display and the sound player, and a second controller connected with the first controller and the server in a communication way respectively;

the second controller is configured to perform:

controlling the second application to carry out video call in the form of a full-screen call interface according to the signaling session information received from the server; the signaling session information comprises the current session ID of the virtual room, the call state of each call member, the camera state and the microphone state;

responding to the operation of switching the playing source, controlling a first controller to start the playing source, and enabling the first controller to control a display to display a video call interface in a thumbnail window mode on the upper layer of the interface of the playing source;

and responding to the operation of switching the thumbnail window to the full-screen call interface, and recording the ID of the playing source, wherein the ID of the playing source is used for enabling the display to redisplay the playing source when the user switches the full-screen call interface to the thumbnail window mode again or when the user hangs up the video call on the full-screen call interface.

In a second aspect, the present application provides another display apparatus comprising:

a display;

a sound player;

the communicator is used for being in communication connection with the server;

a user input interface for receiving an operation input by a user;

the first controller is configured to perform: in the process that the second controller controls the second application to carry out video call in the form of a full-screen call interface according to signaling session information received from the server, starting a play source according to a processing instruction sent by the second controller after responding to the operation of switching the play source, and controlling a display to display the video call interface in a thumbnail window mode on the upper layer of the interface of the play source; according to a processing instruction sent by the second controller after responding to the operation of switching the thumbnail window into the full-screen call interface, controlling the display to display the full-screen call interface;

receiving an ID of the play source sent by the second controller when the user switches the full-screen call interface into a thumbnail window mode again or the user hangs up the video call on the full-screen call interface, and controlling a display to redisplay the play source; the ID of the play source is recorded when the second controller responds to the operation of switching the thumbnail window to a full-screen call interface;

The signaling session information comprises the current session ID of the virtual room, and the call state, the camera state and the microphone state of each call member.

In a third aspect, the present application provides a video call processing method of a display device, the display device including a first controller, a second controller communicatively connected to the first controller and a server, respectively, a display and a sound player connected to the first controller, the method being used for the second controller, including:

In a fourth aspect, the present application provides another method for processing a video call of a display device, where the display device includes a first controller, a second controller communicatively connected to the first controller and a server, respectively, and a display and a sound player connected to the first controller, and the method is used for the first controller, and includes

In the process that the second controller controls the second application to carry out video call in the form of a full-screen call interface according to signaling session information received from the server, starting a play source according to a processing instruction sent by the second controller after responding to the operation of switching the play source, and controlling a display to display the video call interface in a thumbnail window mode on the upper layer of the interface of the play source; according to a processing instruction sent by the second controller after responding to the operation of switching the thumbnail window into the full-screen call interface, controlling the display to display the full-screen call interface;

The technical scheme provided by the application is suitable for processing video calls initiated or accepted by the dual-system display equipment. When the display device initiates or accepts the video call, the server establishes a virtual room for the video call, and the user can chat with the call object by joining the virtual room. The server records the corresponding signaling session information of the video call, including the current session ID of the virtual room, the call status of each call member (such as to answer, call, hang-up, etc.), the camera status (camera on or off), and the microphone status (microphone on or off), and because the second application in the second controller is the main application of the video call, the video call is processed by the second application, and the video call is performed in the form of a full-screen call interface.

When the user performs video chat, the user may switch the playing source, such as to a cable television or to some application program of the first/second controller, so that the user cannot see the video call interface while watching other playing sources, and the user cannot perform video chat. In response to the operation of switching the playing source, the first controller needs to be controlled to start the playing source to be switched to, namely normal playing of the playing source is ensured, meanwhile, the video call interface is displayed on the upper layer of the interface of the playing source in the form of a thumbnail window, and the thumbnail window is a small window interface of the video call.

When the video call is displayed in the form of the thumbnail window, the user can switch the thumbnail window into a full-screen call interface at any time, and when the window switching operation is responded, the ID of the playing source before the window is not switched needs to be recorded, so that when the user switches the full-screen call interface into the thumbnail window mode again, or the user hangs up the video call on the full-screen call interface, the display can redisplay the playing source corresponding to the ID, namely, the playing state before the thumbnail window is switched to the full screen is restored, the accurate control of playing is ensured, and the user experience is improved.

Drawings

In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings that are needed in the embodiments will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.

FIG. 1 is a diagram of a dual system display device;

fig. 2 schematically shows an operational scenario between a display device and a control apparatus;

fig. 3 exemplarily shows a hardware configuration block diagram of the control apparatus 100;

Fig. 4 exemplarily shows a hardware configuration block diagram of the display apparatus 200;

fig. 5 illustrates a hardware architecture block diagram of the display device 200;

fig. 6 exemplarily shows a functional configuration diagram of the display apparatus 200;

fig. 7 (a) is a schematic diagram exemplarily showing a software configuration in the display device 200;

fig. 7 (b) is a schematic diagram exemplarily showing a configuration of an application program in the display apparatus 200;

FIG. 8 illustrates a schematic diagram of a user interface in display device 200;

FIG. 9 illustrates an application scenario diagram;

fig. 10 is a flowchart of a method for performing video call processing by the display device according to the first embodiment;

FIG. 11 is an interactive flow chart of video call logic shown in the second embodiment;

FIG. 12 is an interactive flow chart of video call logic shown in embodiment three;

FIG. 13 is an interactive flow chart of video call logic shown in embodiment four;

FIG. 14 is an interactive flow chart of video call logic shown in embodiment five;

FIG. 15 is an interactive flow chart of video call logic shown in the sixth embodiment;

FIG. 16 is an interactive flow chart of video call logic shown in embodiment seven;

FIG. 17 is an interactive flow chart of video call logic shown in embodiment eight;

FIG. 18 is an interactive flow chart of video call logic shown in embodiments nine and ten;

Fig. 19 is an interactive flow chart of video call logic shown in embodiment eleven.

Detailed Description

The following description of the embodiments of the present application will be made clearly and completely with reference to the accompanying drawings, in which it is apparent that the embodiments described are only some embodiments of the present application, but not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the application without making any inventive effort, are intended to be within the scope of the application.

The present application is directed to a display device having a dual system structure, i.e., having a first controller and a second controller, and a video call processing method thereof. The first controller controls the display and the sound player to play the audio and video of the playing source, and the playing source of the first controller can be an external signal source device, such as a set top box connected to play the cable television; alternatively, the play source may also be an application preset in the first controller; alternatively, the play source may be an application installed in the first controller; or, the playing source may be an application program installed in the second controller, so as to play the network video, conduct video call and support the user to use other preset application programs (apps) in the second controller.

In practice, the applicant found that, due to the limitation of the structure of the dual-system display device, the logic of processing the video call cannot bring better experience to the user, for example, the display of the play source in the second controller and the display of the play source in the first controller are mutually exclusive, when the user views the play source in the first controller, for example, when the user views the cable television, the display cannot display the call interface corresponding to the video call application in the second controller, and the sound player cannot play the audio related to the video call, and only when the user operates to exit the cable television and switches to the video call application in the second controller, the user can answer and participate in the video call, so that the actual experience of the user is poor.

In this regard, the present application aims to implement, through interaction coordination between the first controller and the second controller in the software layer, video call in a thumbnail window mode by a user when the first controller is switched to other play sources except the second application, that is, implementing chat while watching, and providing specific video call processing logic based on the switching operation of the user on the play source and the call window state. The following first describes in detail the structure, function, implementation, and the like of a display device having a dual system hardware structure.

For convenience of use, various external device interfaces are usually provided on the display device, so as to connect different peripheral devices or cables to realize corresponding functions. When the high-definition camera is connected to the interface of the display device, if the hardware system of the display device does not have a hardware interface of the high-pixel camera for receiving the source code, the data received by the camera cannot be presented on the display of the display device.

Also, due to the hardware structure, the hardware system of the conventional display device only supports one path of hard decoding resource, and usually only supports video decoding with a resolution of 4K at maximum, so when video chat while watching the network television is to be implemented, in order not to reduce the definition of the network video picture, it is necessary to decode the network video using the hard decoding resource (typically, GPU in the hardware system), and in this case, only the video chat picture can be processed in such a way that the video is soft decoded by a general processor (e.g. CPU) in the hardware system.

The soft decoding is adopted to process the video chat pictures, so that the data processing load of the CPU is greatly increased, and when the data processing load of the CPU is too heavy, the problems of picture blocking or unsmooth can occur. Furthermore, due to the data processing capability of the CPU, when the video chat frame is processed by adopting the soft decoding of the CPU, the multi-channel video call cannot be realized, and when the user wants to perform video chat with a plurality of other users at the same time in the same chat scene, the situation of access blocking occurs.

Based on the above-mentioned aspects, to overcome the above-mentioned drawbacks, the present application discloses a dual hardware system architecture to implement multiple video chat data (at least one local video).

The concept of the present application will be described with reference to the accompanying drawings. It should be noted that the following descriptions of the concepts are only for making the content of the present application easier to understand, and do not represent a limitation on the protection scope of the present application.

The term "module" as used in various embodiments of the present application may refer to any known or later developed hardware, software, firmware, artificial intelligence, fuzzy logic, or combination of hardware and/or software code that is capable of performing the functionality associated with that element.

The term "remote control" as used in the various embodiments of the present application refers to a component of an electronic device (such as a display device as disclosed herein) that can typically wirelessly control the electronic device over a relatively short range of distances. The assembly may be connected to the electronic device generally using infrared and/or Radio Frequency (RF) signals and/or bluetooth, and may also include functional modules such as WiFi, wireless USB, bluetooth, motion sensors, etc. For example: the hand-held touch remote controller replaces most of the physical built-in hard keys in a general remote control device with a touch screen user interface.

The term "gesture" as used in embodiments of the present application refers to a user's behavior through a change in hand or hand movement, etc., to express an intended idea, action, purpose, or result.

The term "hardware system" as used in the various embodiments of the application may refer to a physical component comprising mechanical, optical, electrical, magnetic devices such as integrated circuits (Integrated Circuit, ICs), printed circuit boards (Printed circuit board, PCBs) with computing, control, storage, input and output functions. In various embodiments of the present application, the hardware system may also be generally referred to as a motherboard (or chip).

A schematic diagram of an operation scenario between a display device and a control apparatus according to an embodiment is exemplarily shown in fig. 2. As shown in fig. 2, a user may operate the display apparatus 200 by controlling the device 100.

The control device 100 may be a remote controller 100A, which may communicate with the display device 200 through infrared protocol communication, bluetooth protocol communication, zigBee protocol communication, or other short-range communication, and is used to control the display device 200 through wireless or other wired modes. The user may control the display device 200 by inputting user instructions through keys on a remote control, voice input, control panel input, etc. Such as: the user can input corresponding control instructions through volume up-down keys, channel control keys, up/down/left/right movement keys, voice input keys, menu keys, on-off keys, etc. on the remote controller to realize the functions of the control display device 200.

The control apparatus 100 may also be a smart device, such as a mobile terminal 100B, a tablet, a computer, a notebook, etc., which may communicate with the display device 200 through a local area network (LAN, local Area Network), a wide area network (WAN, wide Area Network), a wireless local area network ((WLAN, wireless Local Area Network) or other networks, and control the display device 200 through an application program corresponding to the display device 200.

For example, the mobile terminal 100B and the display device 200 may each be provided with a software application, so that a connection listening between the two may be achieved through a network communication protocol, thereby achieving the purpose of one-to-one control operation and data communication. Such as: the mobile terminal 100B and the display device 200 can be made to establish a control instruction protocol, the remote control keyboard is synchronized to the mobile terminal 100B, and the functions of controlling the display device 200 are realized by controlling the user interface on the mobile terminal 100B; the audio/video content displayed on the mobile terminal 100B may also be transmitted to the display device 200, so as to implement a synchronous display function.

As shown in fig. 2, the display device 200 may also be in data communication with the server 300 via a variety of communication means. In various embodiments of the present application, display device 200 may be allowed to communicate with server 300 via a local area network, wireless local area network, or other network. The server 300 may provide various contents and interactions to the display device 200.

By way of example, the display device 200 receives software program updates by sending and receiving information, and electronic program guide (EPG, electronic Program Guide) interactions, or accesses a remotely stored digital media library. The servers 300 may be one group, may be multiple groups, and may be one or more types of servers. Other web service content such as video on demand and advertising services are provided through the server 300.

The display device 200, in one aspect, may be a liquid crystal display, OLED (Organic Light Emitting Diode) display, projection display device; in another aspect, the display device may be a smart television or a display system of a display and a set-top box. The particular display device type, size, resolution, etc. are not limited, and those skilled in the art will appreciate that the display device 200 may be subject to some changes in performance and configuration as desired.

The display device 200 may additionally provide an intelligent network television function of a computer support function in addition to the broadcast receiving television function. Examples include web tv, smart tv, internet Protocol Tv (IPTV), etc. In some embodiments, the display device may not have broadcast receiving television functionality.

As shown in fig. 2, a camera may be connected or disposed on the display device, so as to present the image captured by the camera on a display interface of the display device or other display devices, so as to implement interactive chat between users. Specifically, the picture shot by the camera can be displayed in a full screen, a half screen or any optional area on the display device.

As an optional connection mode, the camera is connected with the display back shell through the connecting plate, is fixedly arranged in the middle of the upper side of the display back shell, and can be fixedly arranged at any position of the display back shell in a mountable mode, so that an image acquisition area of the camera can be prevented from being blocked by the back shell, for example, the display orientation of the image acquisition area is the same as that of display equipment.

As another alternative connection mode, the camera is connected with the display back shell in a liftable manner through a connection plate or other conceivable connectors, and the lifting motor is installed on the connectors, so that when a user needs to use the camera or has an application program to use the camera, the camera is lifted out of the display, and when the user does not need to use the camera, the camera can be embedded behind the back shell, so that the camera is protected from being damaged and privacy security of the user is protected.

As an embodiment, the camera adopted by the application can be 1600 ten thousand pixels so as to achieve the purpose of ultra-high definition display. In practical use, cameras higher or lower than 1600 ten thousand pixels may also be used.

After the camera is installed on the display device, the contents displayed in different application scenes of the display device can be fused in a plurality of different modes, so that the function which cannot be realized by the traditional display device is achieved.

For example, a user may conduct a video chat with at least one other user while watching a video program. The presentation of the video program may be a background picture over which a window of video chat is displayed. The function is visual and can be called "chat while watching".

Optionally, in the scene of "watch while chat", at least one video chat is performed across terminals while live video or network video is being watched.

In another example, a user may conduct a video chat with at least one other user while entering the educational application study. For example, students may be able to achieve remote interaction with teachers while learning content in educational applications. Vividly, the function can be called "chat while learning".

In another example, a user may conduct a video chat with a player entering a game while playing a card game. For example, a player may enable remote interaction with other players when entering a gaming application to participate in a game. The function may be said to be visual, as "play while watching".

Optionally, the game scene is fused with the video picture, the portrait in the video picture is scratched, and the portrait is displayed in the game picture, so that the user experience is improved.

Optionally, in somatosensory games (such as ball playing, boxing, running, dancing, etc.), the body gestures and actions are obtained through the camera, limb detection and tracking, detection of key point data of the bones of the body, and then the body gestures and actions are fused with animation in the games, so that the games of scenes such as sports, dance, etc. are realized.

In another example, a user may interact with at least one other user in a karaoke application, video and voice. The function can be said to be visual, i.e. "watch while sing". Optionally, when at least one user enters the application in the chat scene, a plurality of users can jointly complete recording of a song.

In another example, the user may turn on the camera locally to take pictures and video, and the function may be referred to as "looking at the mirror" for an image.

In other examples, more functions may be added or the above functions may be reduced. The function of the display device is not particularly limited by the present application.

A block diagram of the configuration of the control apparatus 100 according to an exemplary embodiment is exemplarily shown in fig. 3. As shown in fig. 3, the control device 100 includes a controller 110, a communicator 130, a user input/output interface 140, a memory 190, and a power supply 180.

The control apparatus 100 is configured to control the display device 200 and to receive an input operation instruction from a user, and to convert the operation instruction into an instruction recognizable and responsive to the display device 200, and to perform an interaction between the user and the display device 200. Such as: the user responds to the channel addition and subtraction operation by operating the channel addition and subtraction key on the control apparatus 100.

In some embodiments, the control apparatus 100 may be a smart device. Such as: the control apparatus 100 may install various applications for controlling the display device 200 according to user's needs.

In some embodiments, as shown in fig. 2, the mobile terminal 100B or other intelligent electronic device may function similarly to the control apparatus 100 after installing an application for manipulating the display device 200. Such as: the user may implement the functions of the physical keys of the control apparatus 100 by installing an application, various function keys or virtual buttons of a graphical user interface available on the mobile terminal 100B or other intelligent electronic device.

The controller 110 includes a processor 112, RAM113 and ROM114, a communication interface, and a communication bus. The controller 110 is used to control the operation and operation of the control device 100, as well as the communication collaboration among the internal components and the external and internal data processing functions.

The communicator 130 performs communication of control signals and data signals with the display device 200 under the control of the controller 110. Such as: the received user input signal is transmitted to the display device 200. The communicator 130 may include at least one of a WIFI module 131, a bluetooth module 132, an NFC module 133, and the like.

A user input/output interface 140, wherein the input interface includes at least one of a microphone 141, a touch pad 142, a sensor 143, keys 144, etc. Such as: the user can implement a user instruction input function through actions such as voice, touch, gesture, press, and the like, and the input interface converts a received analog signal into a digital signal and converts the digital signal into a corresponding instruction signal, and sends the corresponding instruction signal to the display device 200.

The output interface includes an interface that transmits the received user instruction to the display device 200. In some embodiments, an infrared interface may be used, as well as a radio frequency interface. Such as: when the infrared signal interface is used, the user input instruction needs to be converted into an infrared control signal according to an infrared control protocol, and the infrared control signal is sent to the display device 200 through the infrared sending module. And the following steps: when the radio frequency signal interface is used, the user input instruction is converted into a digital signal, and then the digital signal is modulated according to a radio frequency control signal modulation protocol and then transmitted to the display device 200 through the radio frequency transmission terminal.

In some embodiments, the control device 100 includes at least one of a communicator 130 and an output interface. The control device 100 is provided with a communicator 130 such as: the modules such as WIFI, bluetooth, NFC, etc. may send the user input instruction to the display device 200 through the WIFI protocol, or the bluetooth protocol, or the NFC protocol code.

A memory 190 for storing various operation programs, data and applications for driving and controlling the control device 100 under the control of the controller 110. The memory 190 may store various control signal instructions input by a user.

The power supply 180 is configured to provide operation power support for each electrical component of the control device 100 under the control of the controller 110. May be a battery and associated control circuitry.

A hardware configuration block diagram of a hardware system in the display device 200 according to an exemplary embodiment is exemplarily shown in fig. 4.

When the dual hardware system architecture is adopted, the organization relationship of the hardware system can be shown in fig. 4. The first controller comprises various processors and various interfaces of the first controller, and the second controller comprises various processors and various interfaces of the second controller. The first controller and the second controller may be respectively provided with a relatively independent operating system, and the operating system of the second controller and the operating system of the first controller may communicate with each other through a communication protocol, which is exemplary: the frame work layer of the operating system of the second controller and the frame work layer of the operating system of the first controller may communicate for command and data transmission such that there are two separate but interrelated subsystems in the display device 200.

As shown in fig. 4, the second controller and the first controller may be connected, communicated, and powered through a plurality of different types of interfaces. The interface type of the interface between the second controller and the first controller may include General-purpose input/output (GPIO), USB interface, HDMI interface, UART interface, and the like. One or more of these interfaces may be used between the second controller and the first controller for communication or power transfer. For example, as shown in fig. 4, in the dual hardware system architecture, the first controller may be powered by an external power source (power), while the second controller may not be powered by the external power source, but by the first controller. Alternatively, the external power supply may be connected to the first controller and the second controller, respectively, to supply power to the first controller and the second controller.

The second controller may include an interface for connecting other devices or components, such as an MIPI interface for connecting a Camera (Camera), a bluetooth interface, etc., shown in fig. 4, in addition to an interface for connecting with the first controller.

Similarly, the first controller may include, in addition to an interface for connecting with the first controller, a VBY interface for connecting with the display TCON (Timer Control Register), an I2S interface for connecting with a power Amplifier (AMP) and a Speaker (Speaker); and IR/Key interfaces, USB interfaces, wifi interfaces, bluetooth interfaces, HDMI interfaces, tuner interfaces, etc.

The dual hardware system architecture of the present application is further described below with reference to fig. 5. It should be noted that fig. 5 is only an exemplary illustration of the dual hardware system architecture of the present application, and is not meant to limit the present application. In practical applications, both hardware systems may include more or fewer hardware or interfaces as desired.

A hardware architecture block diagram of the display device 200 according to fig. 4 is exemplarily shown in fig. 5. As shown in fig. 5, the hardware system of the display apparatus 200 may include a first controller 210 and a second controller 310, and modules connected to the first controller 210 or the second controller 310 through various interfaces.

The first controller 210 mainly implements conventional television functions (such as a set-top box may be externally connected, etc.), and the second controller 310 may be configured to receive an instruction sent by the first controller 210.

The modules connected to the first controller 210 may include a modem 220, a communicator 230, an external device interface 250, a memory 290, a user input interface 260-3, a video processor 260-1, an audio processor 260-2, a display 280, an audio output interface 270, and a power module 240. The first controller 210 may also include more or fewer modules in other embodiments.

In other embodiments, the first controller 210 may also be connected to more or fewer modules.

In other embodiments, the first controller 210 may include any one or more of the modules described above. When any one or more of the above-described modules are included in the first controller 210, the first controller 210 may not be connected to modules having the same function.

The modem 220 is configured to perform modulation and demodulation processes such as amplification, mixing, and resonance on a broadcast television signal received by a wired or wireless manner, so as to demodulate an audio/video signal carried in a frequency of a television channel selected by a user and additional information (e.g., an EPG data signal) from a plurality of wireless or wired broadcast television signals. Depending on the broadcasting system of the television signal, the signal paths of the modem 220 may be various, such as: terrestrial broadcasting, cable broadcasting, satellite broadcasting, internet broadcasting, or the like; according to different modulation types, the signal adjustment mode can be a digital modulation mode or an analog modulation mode; and the modem 220 may demodulate analog signals and/or digital signals according to the kind of received television signals.

The tuning demodulator 220 is further configured to respond to the frequency of the television channel selected by the user and the television signal carried by the frequency according to the user selection and controlled by the first controller 210.

In other exemplary embodiments, the modem 220 may also be in an external device, such as an external set-top box, or the like. In this way, the set-top box outputs the television audio/video signal after modulation and demodulation, and inputs the television audio/video signal to the display apparatus 200 through the external device interface 250.

Communicator 230 is a component for communicating with external devices or external servers according to various communication protocol types. For example: communicator 230 may include a WIFI module 231, a bluetooth communication protocol module 232, a wired ethernet communication protocol module 233, and other network communication protocol modules or near field communication protocol modules (not shown) such as an infrared communication protocol module.

The display device 200 may establish a connection of control signals and data signals with an external control device or a content providing device through the communicator 230. For example, the communicator may receive a control signal of the remote controller 100A according to the control of the controller.

The external device interface 250 is a component that provides data transmission between the first controller 210 and the second controller and other external devices. The external device interface 250 may be connected to an external device such as a set-top box, a game device, a notebook computer, etc., in a wired/wireless manner, and may receive data such as a video signal (e.g., a moving image), an audio signal (e.g., music), additional information (e.g., an EPG), etc., of the external device.

Among other things, the external device interface 250 may include: the High Definition Multimedia Interface (HDMI) terminals are also referred to as HDMI251, composite Video Blanking Sync (CVBS) terminals are also referred to as AV252, analog or digital component terminals are also referred to as component 253, universal Serial Bus (USB) terminals are also referred to as USB254, red Green Blue (RGB) terminals (not shown), etc. The present application is not limited in the number and type of external device interfaces.

The first controller 210 controls the operation of the display device 200 and responds to the user's operations by running various software control programs (e.g., an operating system and/or various application programs) stored on the memory 290.

As shown in fig. 5, the first controller 210 includes a read only memory RAM214, a random access memory ROM213, a graphics processor 216, a CPU processor 212, a communication interface 218, and a communication bus. The RAM214 and the ROM213, and the graphics processor 216, the CPU processor 212, and the communication interface 218 are connected by a bus.

A ROM213 for storing instructions for various system starts. When the power of the display device 200 starts to be started when the power-on signal is received, the CPU processor 212 executes the system start-up instruction in the ROM213 and copies the operating system stored in the memory 290 to the RAM214 to start to run the start-up operating system. When the operating system is started, the CPU processor 212 copies various applications in the memory 290 to the RAM214, and then starts running the various applications.

A graphics processor 216 for generating various graphical objects, such as: icons, operation menus, user input instruction display graphics, and the like. The device comprises an arithmetic unit, wherein the arithmetic unit is used for receiving various interaction instructions input by a user to carry out operation and displaying various objects according to display attributes. And a renderer that generates various objects based on the results of the operator, and displays the results of rendering on the display 280.

CPU processor 212 is operative to execute operating system and application program instructions stored in memory 290. And executing various application programs, data and contents according to various interactive instructions received from the outside, so as to finally display and play various audio and video contents.

In some exemplary embodiments, the CPU processor 212 may include multiple processors. One of the plurality of processors may include one main processor, and a plurality of or one sub-processor. A main processor for performing some operations of the display apparatus 200 in the pre-power-up mode and/or displaying a picture in the normal mode. A plurality of or a sub-processor for performing an operation in a standby mode or the like.

Communication interface 218 may include first interface 218-1 through nth interface 218-n. These interfaces may be network interfaces that are connected to external devices via a network.

The first controller 210 may control the overall operation of the display apparatus 200. For example: in response to receiving a user command for selecting a UI object to be displayed on the display 280, the first controller 210 may perform an operation related to the object selected by the user command.

Wherein the object may be any one of selectable objects, such as a hyperlink or an icon. Operations related to the selected object, such as: operations to connect to a hyperlink page, document, image, etc., or operations to execute a program corresponding to an icon are displayed. The user command for selecting the UI object may be an input command through various input means (e.g., mouse, keyboard, touch pad, etc.) connected to the display device 200 or a voice command corresponding to a voice uttered by the user.

Memory 290 includes memory for storing various software modules for driving and controlling display device 200. Such as: various software modules stored in memory 290, including: a base module, a detection module, a communication module, a display control module, a browser module, various service modules, and the like (not shown in the figure).

The base module is a bottom software module for signal communication between the various hardware in the display device 200 and for sending processing and control signals to the upper modules. The detection module is a management module for collecting various information from various sensors or user input interfaces, and performing digital-to-analog conversion and analysis management. The voice recognition module comprises a voice analysis module and a voice instruction database module. The display control module is a module for controlling the display 280 to display image content, and can be used for playing multimedia image content, UI interface and other information. The communication module is used for controlling and data communication with external equipment. The browser module is a module for performing data communication between the browsing servers. The service module is used for providing various services and various application programs.

Meanwhile, the memory 290 is also used to store received external data and user data, images of various items in various user interfaces, visual effect maps of focus objects, and the like.

The user input interface 260-3 is used to transmit an input signal of a user to the first controller 210 or transmit a signal output from the first controller 210 to the user. Illustratively, the control device (e.g., mobile terminal 100B or remote control 100A) may send input signals such as a power switch signal, a channel selection signal, a volume adjustment signal, etc., input by the user to the user input interface and then transferred to the first controller 210 by the user input interface 260-3; alternatively, the control device may receive an output signal such as audio, video, or data, which is output from the user input interface 260-3 via the first controller 210, and display the received output signal or output the received output signal in an audio or vibration form.

In some embodiments, a user may input a user command through a Graphical User Interface (GUI) displayed on the display 280, and the user input interface 260-3 receives the user input command through the Graphical User Interface (GUI). Alternatively, the user may input the user command by inputting a specific sound or gesture, and the user input interface 260-3 recognizes the sound or gesture through the sensor to receive the user input command.

Video signals displayed or played on the display 280.

By way of example, the video processor 260-1 includes a demultiplexing module, a video decoding module, an image synthesis module, a frame rate conversion module, a display formatting module, etc. (not shown).

Video signals displayed or played on the display 280.

The demultiplexing module is used for demultiplexing the input audio/video data stream, such as the input MPEG-2, and demultiplexes the input audio/video data stream into video signals, audio signals and the like.

And the video decoding module is used for processing the demultiplexed video signal, including decoding, scaling and the like.

And an image synthesis module, such as an image synthesizer, for performing superposition mixing processing on the graphic generator and the video picture after the scaling processing according to the GUI signal input by the user or generated by the graphic generator, so as to generate an image signal for display.

A frame rate conversion module, configured to convert a frame rate of an input video, such as converting a frame rate of an input 24Hz, 25Hz, 30Hz, 60Hz video to a frame rate of 60Hz, 120Hz, or 240Hz, where the input frame rate may be related to a source video stream and the output frame rate may be related to a refresh rate of a display. And a display formatting module for converting the signal output by the frame rate conversion module into a signal conforming to a display format such as a display, for example, format converting the signal output by the frame rate conversion module to output an RGB data signal.

A display 280 for receiving image signals from the video processor 260-1 for displaying video content and images and a menu manipulation interface. The display 280 includes a display assembly for presenting pictures and a drive assembly for driving the display of images. The video content may be displayed from a video in a broadcast signal received by the modem 220 or may be displayed from a video input from a communicator or an external device interface. And a display 280 for simultaneously displaying a user manipulation interface UI generated in the display device 200 and used to control the display device 200.

And, depending on the type of display 280, a drive assembly for driving the display. Alternatively, if the display 280 is a projection display, a projection device and projection screen may be included.

The audio processor 260-2 is configured to receive the audio signal, decompress and decode according to the standard codec protocol of the input signal, and perform audio data processing such as noise reduction, digital-to-analog conversion, and amplification processing, so as to obtain an audio signal that can be played in the speaker 272.

An audio output interface 270 for receiving an audio signal output by the audio processor 2602 under the control of the first controller 210, where the audio output interface may include a speaker 272, or an external audio output terminal 274 for outputting to a generating device of an external device, such as: external sound terminals or earphone output terminals, etc.

In other exemplary embodiments, video processor 260-1 may include one or more chip components. The audio processor 260-2 may also include one or more chip components.

And, in other exemplary embodiments, the video processor 260-1 and the audio processor 260-2 may be separate chips or integrated with the first controller 210 in one or more chips.

The power supply module 240 is configured to provide power supply support for the display device 200 with power input from an external power source under the control of the first controller 210. The power supply module 240 may include a built-in power circuit installed inside the display apparatus 200, or may be a power source installed outside the display apparatus 200, such as a power interface providing an external power source in the display apparatus 200.

As shown in fig. 5, the modules connected to the second controller 310 may include a communicator 330, a detector 340, and a memory 390. User input interfaces, video processor 360-1, audio processor 360-2, displays, audio output interfaces (not shown) may also be included in some embodiments. In some embodiments, there may also be a power module ((not shown in the figures)) that independently powers the second controller 310.

In some embodiments, the second controller 310 may also be connected to more or fewer modules.

In some embodiments, the second controller 310 may include any one or more of the modules described above. When any one or more of the above modules are included in the second controller 310, the first controller 210 may not be connected to the modules having the same function.

The communicator 330 is a component for communicating with external devices or external servers according to various communication protocol types. For example: the communicator 330 may include a WIFI module 331, a bluetooth communication protocol module 332, a wired ethernet communication protocol module 333, and other network communication protocol modules such as an infrared communication protocol module or a near field communication protocol module.

(not shown in the drawings)

The communicator 330 of the second controller 310 and the communicator 230 of the first controller 210 also interact with each other. For example, the WiFi module 231 in the hardware system of the first controller 210 is used to connect to an external network, and generate network communication with an external server, etc. The WiFi module 331 in the hardware system of the second controller 310 is used to connect to the WiFi module 231 of the first controller 210 without directly connecting to an external network or the like, and the second controller 310 is connected to the external network through the first controller 210. Thus, for the user, a display device as in the above embodiment displays a WiFi account to the outside.

The detector 340 is a component of the display device that the second controller 310 uses to collect signals of the external environment or interact with the outside. The detector 340 may include a light receiver 342, a sensor for capturing ambient light intensity, a display parameter change that may be adapted by capturing ambient light, etc.; the system can also comprise an image collector 341, such as a camera, a video camera and the like, which can be used for collecting external environment scenes, collecting attributes of a user or interacting gestures with the user, adaptively changing display parameters and identifying the gestures of the user so as to realize the interaction function with the user.

The external device interface 350 provides a component for data transfer between the second controller 310 and the first controller 210 or other external devices. The external device interface 350 may be connected to external devices such as a set-top box, a game device, a notebook computer, etc., in a wired/wireless manner.

The video processor 360-1 is used for processing the relevant video signals.

The second controller 310 controls the operation of the display device 200 and responds to user operations by running various software control programs stored on the memory 390 (e.g., with an installed third party application, etc.), as well as interactions with the first controller 210.

As shown in fig. 5, the second controller 310 includes a read only memory ROM313, a random access memory RAM314, a graphic processor 316, a CPU processor 312, a communication interface 318, and a communication bus. The ROM313 and RAM314, and the graphics processor 316, CPU processor 312, and communication interface 318 are connected by a bus.

A ROM313 for storing instructions for various system starts. The CPU processor 312 executes system boot instructions in the ROM313 and copies the operating system stored in the memory 390 to the RAM314 to begin running the boot operating system. When the operating system is started, the CPU processor 312 copies various applications in the memory 390 to the RAM314, and then starts running the various applications.

The CPU processor 312 is configured to execute operating system and application program instructions stored in the memory 390, and to communicate, signal, data, instruction, etc. with the first controller 210, and to execute various application programs, data, and contents according to various interactive instructions received from the outside, so as to finally display and play various audio and video contents.

The communication interfaces 318 are plural and may include a first interface 318-1 through an nth interface 318-n. These interfaces may be network interfaces connected to external devices via a network, or network interfaces connected to the first controller 210 via a network.

The second controller 310 may control the overall operation of the display apparatus 200. For example: in response to receiving a user command for selecting a UI object to be displayed on the display 280, the first controller 210 may perform an operation related to the object selected by the user command.

A graphics processor 316 for generating various graphical objects, such as: icons, operation menus, user input instruction display graphics, and the like. The device comprises an arithmetic unit, wherein the arithmetic unit is used for receiving various interaction instructions input by a user to carry out operation and displaying various objects according to display attributes. And a renderer that generates various objects based on the results of the operator, and displays the results of rendering on the display 280.

The graphics processor 316 of the second controller 310 and the graphics processor 216 of the first controller 210 are each capable of generating various graphics objects. By distinction, if application 1 is installed on the second controller 310, application 2 is installed on the first controller 210, and when a user inputs an instruction at the interface of application 1 and within application 1, a graphic object is generated by the graphic processor 316 of the second controller 310. When a user is at the interface of application 2 and a user entered instruction is made within application 2, a graphical object is generated by the graphics processor 216 of the first controller 210.

A functional configuration diagram of a display device according to an exemplary embodiment is exemplarily shown in fig. 6.

As shown in fig. 6, the memory 390 of the second controller and the memory 290 of the first controller are used to store an operating system, application programs, contents, user data, and the like, respectively, and perform system operations for driving the display device 200 and various operations in response to a user under the control of the second controller 310 and the first controller 210. The memory 390 of the second controller and the memory 290 of the first controller may include volatile and/or nonvolatile memory.

For the first controller 210, the memory 290 is specifically configured to store an operation program for driving the first controller 210 in the display device 200, and store various application programs built in the display device 200, various application programs downloaded by a user from an external device, various graphical user interfaces related to the application programs, various objects related to the graphical user interfaces, user data information, and various internal data supporting the application programs. The memory 290 is used to store system software such as an Operating System (OS) kernel, middleware and applications, and to store input video data and audio data, as well as other user data.

Memory 290 is specifically used to store drivers and related data for video processor 260-1 and audio processor 260-2, display 280, communicator 230, modem 220, input/output interfaces, and the like.

In some embodiments, memory 290 may store software and/or programs, the software programs used to represent an Operating System (OS) including, for example: a kernel, middleware, an Application Programming Interface (API), and/or an application program. For example, the kernel may control or manage system resources, or functions implemented by other programs (such as the middleware, APIs, or application programs), and the kernel may provide interfaces to allow the middleware and APIs, or applications to access the controller to implement control or management of system resources.

By way of example, the memory 290 includes a broadcast receiving module 2901, a channel control module 2902, a volume control module 2903, an image control module 2904, a display control module 2905, a first audio control module 2906, an external instruction recognition module 2907, a communication control module 2908, a light receiving module 2909, a power control module 2910, an operating system 2911, and application programs 2912, a browser module (not shown in the figure), and the like. The first controller 210 executes various software programs in the memory 290 such as: broadcast television signal receiving and demodulating functions, television channel selection control functions, volume selection control functions, image control functions, display control functions, audio control functions, external instruction recognition functions, communication control functions, optical signal receiving functions, power control functions, software control platforms supporting various functions, browser functions and other various functions.

Memory 390 includes storage for various software modules for driving and controlling display device 200. Such as: various software modules stored in memory 390, including: a base module, a detection module, a communication module, a display control module, a browser module, various service modules, and the like (not shown in the figure). Since the functions of the memory 390 and the memory 290 are similar, the relevant portions will be referred to as the memory 290, and will not be described herein.

By way of example, the memory 390 includes an image control module 3904, a second audio control module 3906, an external instruction recognition module 3907, a communication control module 3908, a light receiving module 3909, an operating system 3911, and application programs 3912, a browser module 3913, and the like. The second controller 310 performs functions such as: image control function, display control function, audio control function, external instruction recognition function, communication control function, optical signal receiving function, power control function, software control platform supporting various functions, browser function and other various functions.

Differentially, the external instruction recognition module 2907 of the first controller and the external instruction recognition module 3907 of the second controller may recognize different instructions.

For example, since the image receiving device such as a camera is connected to the second controller 310, the external command recognition module 3907 of the second controller may include a graphic recognition module 3907-1, where a graphic database is stored in the graphic recognition module 3907-1, and when the camera receives an external graphic command, the camera performs a correspondence with the command in the graphic database to perform command control on the display device. Since the voice receiving device and the remote controller are connected to the first controller 210, the external command recognition module 2907 of the first controller may include a voice recognition module 2907-2, where a voice database is stored in the voice recognition module 2907-2, and when the voice receiving device receives an external voice command or receives a corresponding relationship with a command in the voice database, the voice receiving device and the like can control the display device with commands. Similarly, the control device 100 such as a remote controller is connected to the first controller 210, and the key instruction recognition module 2907-3 performs instruction interaction with the control device 100.

A block diagram of the configuration of the software system in the display device 200 according to the exemplary embodiment is exemplarily shown in fig. 7 (a).

For the first controller 210, as shown in fig. 7 (a), an operating system 2911, including execution operating software for processing various basic system services and for performing hardware-related tasks, serves as a medium for completing data processing between application programs and hardware components.

In some embodiments, portions of the operating system kernel may contain a series of software to manage display device hardware resources and to serve other programs or software code.

In other embodiments, portions of the operating system kernel may contain one or more device drivers, which may be a set of software code in the operating system that helps operate or control the devices or hardware associated with the display device. The driver may contain code to operate video, audio and/or other multimedia components. Examples include a display, camera, flash, wiFi, and audio drivers.

Wherein, accessibility module 2911-1 is configured to modify or access an application program to realize accessibility of the application program and operability of display content thereof.

The communication module 2911-2 is used for connecting with other peripheral devices via related communication interfaces and communication networks.

User interface module 2911-3 is configured to provide an object for displaying a user interface for access by each application program, so as to implement user operability.

Control applications 2911-4 are used to control process management, including runtime applications, and the like.

The event delivery system 2914 may be implemented within the operating system 2911 or in the application 2912. In some embodiments, one aspect is implemented within the operating system 2911, while the application 2912 is implemented to monitor various user input events, and to refer to a process program that implements one or more sets of predefined operations in response to recognition results of various events or sub-events, based on the various events.

The event monitoring module 2914-1 is configured to monitor a user input interface to input an event or a sub-event.

The event recognition module 2914-2 is configured to input definitions of various events to various user input interfaces, recognize various events or sub-events, and transmit them to a process for executing one or more corresponding sets of processes.

The event or sub-event refers to an input detected by one or more sensors in the display device 200, and an input of an external control device (such as the control apparatus 100). Such as: various sub-events are input through voice, gesture input sub-events of gesture recognition, sub-events of remote control key instruction input of a control device and the like. By way of example, one or more sub-events in the remote control may include a variety of forms including, but not limited to, one or a combination of key press up/down/left/right/, ok key, key press, etc. And operations of non-physical keys, such as movement, holding, releasing, etc.

The interface layout management module 2913 directly or indirectly receives the user input events or sub-events from the event transmission system 2914, and is used for updating the layout of the user interface, including but not limited to the positions of the controls or sub-controls in the interface, and various execution operations related to the interface layout, such as the size or position of the container, the level, and the like.

Since the operating system 3911 of the second controller is similar to the operating system 2911 of the first controller, the relevant portions only need to be referred to the operating system 2911, and the detailed description is omitted herein.

As shown in fig. 7 (b), the application of the display device includes various application programs executable on the display device 200.

The application 2912 of the first controller may include, but is not limited to, one or more applications such as: video on demand applications, application centers, gaming applications, etc. The applications 3912 of the second controller may include, but are not limited to, one or more applications such as: live television applications, media center applications, etc. It should be noted that what application program is contained in the second controller and the first controller is determined according to the operating system and other designs, and specific limitation and division of the application programs contained in the second controller and the first controller are not required in the present invention.

Live television applications can provide live television through different signal sources. For example, a live television application may provide television signals using inputs from cable television, radio broadcast, satellite services, or other types of live television services. And, the live television application may display video of the live television signal on the display device 200.

Video on demand applications may provide video from different storage sources. Unlike live television applications, video-on-demand provides video displays from some storage sources. For example, video-on-demand may come from the server side of cloud storage, from a local hard disk storage containing stored video programs.

The media center application may provide various applications for playing multimedia content. For example, a media center may be a different service than live television or video on demand, and a user may access various images or audio through a media center application.

An application center may be provided to store various applications. The application may be a game, an application, or some other application associated with a computer system or other device but operable on a display device. The application center may obtain these applications from different sources, store them in local storage, and then be run on the display device 200.

The modules included in the hardware in the above embodiments are examples of a real-time manner that may be implemented, and are not necessarily shown as a display device or a method of the present application, and those skilled in the art may appropriately increase or decrease the modules based on the implementation of the hardware, so long as implementation of the scheme is not affected.

A schematic diagram of a user interface in a display device 200 according to an exemplary embodiment is schematically shown in fig. 8. As shown in fig. 8, the user interface includes a plurality of view display areas, illustratively, a first view display area 201 and a play screen 202, wherein the play screen includes a layout of one or more different items. And a selector in the user interface indicating that an item is selected, the position of the selector being movable by user input to change selection of a different item.

It should be noted that the multiple view display areas may present different levels of display images. For example, the first view display area may present Video chat item content and the second view display area may present application layer item content (e.g., web page Video, VOD (Video on Demand) presentation, application screen, etc.).

Optionally, the presentation of different view display areas has priority difference, and the display priorities of the view display areas are different among the view display areas with different priorities. If the priority of the system layer is higher than that of the application layer, when the user uses the acquisition selector and the picture switching in the application layer, the picture display of the view display area of the system layer is not blocked; and when the size and the position of the view display area of the application layer are changed according to the selection of the user, the size and the position of the view display area of the system layer are not affected.

The same level of display may be presented, in which case the selector may switch between the first view display region and the second view display region, and the size and position of the second view display region may change as the size and position of the first view display region changes.

Since the second controller 310 and the first controller 210 may have separate operating systems installed therein, there are two independent but interrelated subsystems in the display device 200. For example, the second controller 310 and the first controller 210 may each be independently provided with an Android (Android) and various types of APP, so that each chip may implement a certain function, and the second controller 310 and the first controller 210 cooperatively implement a certain function.

Based on the dual system display device, as shown in fig. 9, an application scenario diagram is shown for an embodiment of the present application, where at least a server 300 and at least two display devices 200 are involved, where a plurality of display devices 200 may perform communication interaction through the server 300, and a Local Area Network (LAN), a Wireless Local Area Network (WLAN), and other internet networks may be used between the display devices 200 and the server 300. When the display device 200 is two, it corresponds to one-to-one video call. At least one of the plurality of display devices 200 is configured with the dual system structure as described above.

When the number of the accessed display devices 200 exceeds two, the multi-person video call can be entered, in a specific implementation, a member of a party initiating the multi-person video call can construct a virtual room for the video call, the user of the initiator can invite other members, the other members can join the virtual room after answering the incoming call of the video call, and each member in the virtual room has a user ID unique to the member for identifying and distinguishing each member. The audio of the member A can be input through a voice device such as a microphone in the display device, and the video image of the member A is collected through a camera arranged on the display device, namely, the member A can release audio and video through hardware in the local display device, so that other members in the virtual room can subscribe the audio and video released by the member A automatically or manually, further receive and display the video of the member A, and play the audio of the member A. The video call may be in one-to-one, one-to-many, or many-to-many, etc.

The server 300 may maintain a signaling session information supporting the unified use of the whole member, including, but not limited to, a current session ID, a user ID of each call member, and a call state, a camera state, a microphone state, etc. of each call member, through communication with the display device 200 corresponding to each member in the virtual room. The display device monitors the call state of the local user and the state information such as the working states of the microphone and the camera, and sends the state information to the server, and the server collects the state information of each call member and gathers the state information to form the signaling session information, so that the signaling session information is updated along with the change of the state information sent by each display device.

The current session ID is used for identifying different video calls, namely the ID equivalent to the virtual room, so that a user can conveniently join in the corresponding virtual room to carry out the video call; the user ID is an ID value obtained by combining and transforming an account ID and a display device ID, and the user ID of each call member in the virtual room is different so as to accurately process the situations that a single user logs in an application in multiple devices and the first controller and the second controller enter a call at the same time, and the account ID can be an account or a user name registered by the user in the video call application; the call state of each call member comprises new joining, unanswered, answered, in-call, hung-up and the like; the camera states include camera on (on) and camera off (off), and the microphone states include microphone on (on) and microphone off (off). The call state, the camera state and the microphone state of each call member are changed, so that a video call interface is changed, for example, if the member B closes the camera, the real portrait of the member B cannot be seen on the page, and for example, a virtual head portrait is displayed; for example, if the member C hangs up the video call, that is, if the member C exits the virtual room, the member C is not displayed on the video call interface, and the audio of the member C is not played. The server 300 communicates with the second controller 310 in the display device 200, and the server 300 continuously synchronizes the signaling session information to the second controller in the display device corresponding to each call member in the virtual room, and each time the signaling session information is changed, the updated signaling session information needs to be synchronized to each member side, so that the display device corresponding to each call member changes the audio/video playing state and the UI adaptively.

The second controller is preset with a second application, wherein the second application is a user-oriented application program with functions of processing video call and video call, namely the second application is a main application of video call, such as some chat and social applications, for example, weChat, QQ and the like, a user can click on a homepage or an icon of the second application displayed by an application center to enter the second application, and video call is initiated or answered in the second application, and the second application is a play source for the first controller to realize video call.

When the first controller plays a play source which is built-in or externally connected with the first controller, a first application can be preset in the first controller for realizing the simultaneous video call, but the first application is an application which is not oriented to a user and is invisible, the first application supports the display and the answering of the incoming call, the same audio and video access capability as the second application is added, and video call processing logic is configured. When the current playing source is a second application or other application program in the second controller, the second application processes the video call logic, and the first application stops receiving the audio and video data from the virtual room (i.e. stops streaming); when the current playing source is a playing source built in or externally connected to the first controller, i.e. when the current playing source is not a playing source in the second controller, for example, when a cable television is played, a first application needs to be started and controlled to pull a stream from a virtual space, in this case, the first application is taken as a main part, the second application is taken as an auxiliary part, the second application sends signaling session information received from a server to the first application, the first application adds a video call in a virtual room according to the signaling session information and generates a first thumbnail window, a small interface of the video call is displayed on the first thumbnail window, and meanwhile, the sound player plays the audio of the video call.

The first thumbnail window is an interface window obtained by shrinking a certain multiple relative to a full-screen call interface, and the reduced multiple is selected according to actual conditions, so that the first thumbnail window is ensured not to excessively block the picture of the current playing source, and the first thumbnail window is ensured to have a good and clear visual effect. The first thumbnail window is displayed on the upper layer of the interface of the built-in or externally connected playing source of the first controller, specifically, the first thumbnail window can be displayed at a designated position, for example, in fig. 8, the first thumbnail window corresponding to the video call can be displayed at the position of the first view display area 201 in the upper right corner, and the part of the playing screen 202 plays the cable television, so that the user can watch the cable television and perform the video call through the first thumbnail window, and realize "watch while chat".

Because the first application and the second application can access the video call, the first application and the second application can be regarded as two independent call members, if the current playing source is from the second controller, the second application can publish and subscribe (i.e. pull) the audio and video in the virtual room, and at the moment, the first application unsubscribes any audio and video in the virtual room, i.e. the first application stops pulling when the second application pulls; if the current playing source is not the playing source in the second controller, subscribing and receiving the audio and video by the first application, and unsubscribing the audio and video by the second application, namely stopping the streaming by the second application when the first application pulls the streaming, and not publishing the audio and video stream when the first application accesses the virtual room, but subscribing and receiving the audio and video stream only. When switching from the play source of the first controller to the play source of the second controller, the audio stream of the first application needs to be unsubscribed, otherwise, when switching from the play source of the second controller to the play source of the first controller, the audio stream of the second application needs to be unsubscribed, so that the local sound is ensured not to collide.

Since the first application and the second application belong to the same display device but are regarded as two independent call members, there is also a need to accurately distinguish between video call applications in the first controller and the second controller, and in one implementation, the ID value, i.e. the user ID in the foregoing signaling session information, may be calculated by the account ID and the display device ID, and each user ID has a uniqueness, i.e. the user IDs of the first controller and the second controller of different display devices are different.

According to the application, in the process of processing the video call by the first controller or the second controller, the user can be supported to switch the display mode of the video call interface, including a full-screen call interface and a thumbnail window mode. When a first application processes video call, a first thumbnail window created and generated by the first application is displayed; when the second application processes the video call, if the user selects the thumbnail window mode to perform the video call, the second thumbnail window is displayed. Some operation controls, such as a hang-up control, a camera switch control, a microphone switch control, a window switch control and the like, can be arranged in the full-screen call interface, the second thumbnail window and the second thumbnail window, and can be set according to practical application. It should be noted that, when the second application processes the video call, the video call interface may be a full-screen call interface or a mode of the second thumbnail window; when the first application processes the video call, the video call interface does not support the form of a full-screen call interface, but can only be displayed in the mode of the first thumbnail window. The video processing method and logic of the dual system display device are specifically described below.

With reference to the dual-system structure, the first controller and the second controller can simultaneously perform video call when watching the playing source through certain interactive logic, so that the function of watching and chatting is realized in the display device. The display device provided in the first embodiment of the present application includes a first controller 210, a second controller 310, a communicator 330, a display, a sound player and a user input interface 260-3, where the second controller 310 is respectively connected with the server 300 and the first controller 210 in a communication manner, the first controller 210 is respectively connected with the display and the sound player, the display is used for displaying video images of a playing source (including video call) in the first controller or the second controller, and the sound player is used for playing audio, and the sound player is a speaker 272 in fig. 5, or an audio device externally connected with an audio output terminal 274; the user input interface 260-3 is used for receiving operations input by a user, such as an operation of switching a play source, an operation of switching a window size of a video call interface, and the like. Specifically, as shown in fig. 10:

wherein the second controller is configured to perform: controlling the second application to carry out video call in the form of a full-screen call interface according to the signaling session information received from the server; the signaling session information comprises the current session ID of the virtual room, the call state of each call member, the camera state and the microphone state; responding to the operation of switching the playing source, controlling a first controller to start the playing source, and enabling the first controller to control a display to display a video call interface in a thumbnail window mode on the upper layer of the interface of the playing source; and responding to the operation of switching the thumbnail window to the full-screen call interface, and recording the ID of the playing source, wherein the ID of the playing source is used for enabling the display to redisplay the playing source when the user switches the full-screen call interface to the thumbnail window mode again or when the user hangs up the video call on the full-screen call interface.

When a user initiates a video call or joins a video call after accepting an incoming call invitation, the relevant logic for handling the video call by a second application preset in the second controller is generally defaulted. When the second application initiates a video call or accepts an incoming call invitation to join in the video call, signaling session information is received from the server, the second application correspondingly processes the video call according to the call state, the camera state and the microphone state of each call member in the virtual room indicated in the signaling session information, and can preferentially default to a full-screen call interface, such as counting a local end user, wherein 3 call members are in the virtual room, member 1 (initiator) calls, the camera and the microphone are all on, member 2 is to be answered, the microphone and the camera are all off, member 3 (local end user) has been answered, the microphone is turned on but the camera is turned off, and then the video call interface needs to be displayed in a matching manner and matched audio needs to be played according to the relevant state information of each member. When the second application processes the video call, the interface and/or audio playing of the video call need to be adaptively updated according to the updated signaling session information, since the signaling session information may be changed. The server may send the latest signaling session information to the second controller of the display device in real time; or the server may detect whether the signaling session information is changed, and after detecting that the signaling session information is changed, send the updated signaling session information to the second controller of the display device.

In the process that the second application performs video call in the form of a full-screen call interface, if the operation of switching the signal source is received, which means that the user wants to watch other play sources except the video call application (namely the second application), the first controller needs to be controlled to start the play source so as to ensure that the video call can be normally switched to the play source for play, and before the video call is not hung up, the video call is displayed in a thumbnail window mode, the user can see the switched play source on a display, and meanwhile, the video call is continuously kept in the thumbnail window mode, so that the video call is watched while chat is realized. The second controller may implement control of the first controller and audio/video playback by transmitting processing instructions/processing instructions or the like to the first controller in response to a user operation.

When the video call is just started, a full-screen call interface is presented, after a user switches a play source, the video call presents a display form of a thumbnail window, controls such as a hang-up control, a window switching control and the like can be arranged on the thumbnail window, when the hang-up control is triggered, the local end user actively hangs up the current video call, when the window switching control is triggered, the display form of the video call interface can be switched, and the thumbnail window is switched to the full-screen call interface. When the second controller responds to the operation that the user switches the thumbnail window to the full-screen call interface, the ID of the current playing source needs to be recorded, namely the ID of the playing source before the non-switched window form needs to be recorded, so that when the user switches the full-screen call interface to the thumbnail window mode again, or the user hangs up the video call on the full-screen call interface, the display can redisplay the playing source corresponding to the ID, namely the playing state before the thumbnail window switches the full screen is restored, the accurate control of the playing is ensured, and the user experience is improved.

On the basis of the foregoing embodiment one, in a second embodiment of the present application, as shown in fig. 11, before the second controller receives the operation of switching the play source, the second controller is further configured to perform: responding to clicking operation of a window switching control on the full-screen call interface by a user, and sending a fifth processing instruction to the first controller, wherein the fifth processing instruction is used for instructing the first controller to control a display to display a preset page and controlling the display to display a second thumbnail window on the upper layer of the preset page; the second thumbnail window is a call window obtained by reducing the full-screen call interface to a preset size; the preset page is a homepage of the display device or a homepage of the second application.

In the state defined by the embodiment, when the user clicks the control to switch the widget on the full-screen call interface, the second controller sends certain processing instructions to the first controller to indirectly control the working states of the display and the sound player, because the display and the sound player are connected with the first controller, namely, the display and the sound player are controlled objects directly controlled by the first controller. When the first controller responds to the fifth processing instruction, the display can be controlled to switch the full-screen call interface into the second thumbnail window, the second thumbnail window can be directly reduced to a preset size and displayed at a designated position without creating and generating the second thumbnail window, and of course, the size and the display position of the second thumbnail window are not fixed, and a user can adjust the second thumbnail window according to personal habits and actual requirements. The portion of the display interface other than the second thumbnail window may then display a preset page, such as the home page of the display device or the home page of the second application, i.e., the "preset page + second thumbnail window" mode.

In the second embodiment, when the hang-up control is triggered, the local end user actively hangs up the current video call, and the display does not display the second thumbnail window any more, but only displays the preset page; when the window switching control is triggered, the second thumbnail window can be switched back to the full-screen call interface again.

For the first embodiment, in the process of processing a video call in a full-screen call interface form by the second application, switching a play source includes two cases (a) and (B), where (a) is switching to a play source in the first controller, and the play source in the first controller includes a signal source (such as a set top box, etc.) connected to itself through interfaces such as HDMI, etc. and an application program built in itself, and is then reflected as the play source+the first thumbnail window of the first controller after switching; (B) The method comprises the steps of switching to other application programs except a second application program in a second controller, enabling a user to click a homepage key on a remote controller to enable a display device to display a homepage, clicking icons of the other application programs from the homepage, and starting the other application programs, wherein when the other application programs run, a video call interface is displayed in a second thumbnail window mode, for example, the user switches to a certain video playing application, and when a network video is watched through the video playing application, video call can be simultaneously carried out through the second thumbnail window, namely, the video call interface is reflected into the other application programs and the second thumbnail window after switching. The video processing logic in each of these two cases is described in detail below.

In the third embodiment of the present application, as shown in fig. 12, when the playback source switched in the first embodiment is a signal source connected to the first controller or an application program installed in the first controller, that is, corresponds to the case (a), the second controller is specifically configured to execute: in response to switching the play source operation, controlling the second application to stop receiving video data from the virtual room, and sending a first processing instruction carrying the signaling session information to the first controller, wherein the first processing instruction is configured to instruct the first controller to execute: starting the playing source and the first application; the method comprises the steps that a first application is controlled to join a virtual room of a video call according to a current session ID, and audio and video data are received from the virtual room; generating a first thumbnail window according to the call state and the camera state of each call member and the video data received from the virtual room, controlling a display to display the first thumbnail window on the upper layer of the interface of the playing source, and controlling a sound player to play the audio data received from the virtual room according to the microphone state of each call member. When the second application should stop receiving video data from the virtual room in response to switching to the play source in the first controller, the second controller simultaneously sends a first processing instruction to the first controller, so that the first controller side starts the first application to process the video call logic and correspondingly control the play state of the display and the sound player.

After receiving the first processing instruction, the first controller needs to start the playing source and the first application, wherein the first processing instruction should carry signaling session information, so that the first application joins in a virtual room of the video call according to the current session ID, receives audio and video data of the current video call from the virtual room, and creates a first thumbnail window according to the call states of all call members, the states of cameras and the video data, namely, a video call interface displayed by the first thumbnail window should be matched with the signaling session information, and the first thumbnail window only reduces the size of the video call interface. The first thumbnail window is displayed at a designated position on the upper layer of the interface of the switched playing source, such as displayed at the upper right corner by default, and of course, the position and the size of the first thumbnail window are not fixed, a user can slide the first thumbnail window at will on the upper layer of the playing interface of the target signal source, namely, the display position of the first thumbnail window can be adjusted at any time according to personal viewing habits, such as moving the first thumbnail window to the lower right corner, and the size of the first thumbnail window can be adjusted as required. Meanwhile, according to the microphone states of the call members indicated in the signaling session information, the sound player is controlled to play the audio data received from the virtual room, so that the local end user can hear the sound of the call member who opens the microphone and cannot hear the sound of the call member who closes the microphone. When a camera/microphone of a user in a virtual room is turned off, a UI change is required to the first thumbnail window and the video/audio stream of the user is turned off. According to the embodiment, when the playing source in the first controller is played, the user can talk in the form of the first thumbnail window, so that chat while watching is realized, and the actual application experience of the user is improved.

Communication interaction between the first controller and the second controller can be realized through the form of RPC (Remote Procedure Call ), namely, the second controller and the first controller can transmit information such as signaling session information, processing instructions and the like through the RPC. When the playing source of the first controller and the first thumbnail window are displayed, the second controller can synchronize the updated signaling session information to the first controller, so that the first controller can update the display state of the first thumbnail window according to the updated signaling session information, and/or update the playing state of the sound player, namely, the UI display state and the audio playing state of the first thumbnail window are adaptively changed according to the updated signaling session information.

In the "play source of first controller+first thumbnail window" mode shown in the third embodiment, the following three situations may occur according to the user operation: the first one (A-1), the user switches the first thumbnail window to the full screen call interface; the second type (A-2), the user switches the current playing source to the target application program, wherein the target application program is other application programs except the second application in the second controller; third, the current video call is cancelled, which includes a call hang-up instruction input by the local user by clicking a hang-up control on the first thumbnail window, that is, the local user actively hangs up the video call, exits the virtual room, and the local user does not actively hang up the video call, but all call objects with the local user video call hang up the video call, and the call objects are other call members in the virtual room except the local user, so that the current video call does not need to be continuously maintained, which is equivalent to that the local user passively hangs up the video call, and the call state of the call objects in the virtual room can be obtained through signaling session information to determine whether the passive hang-up condition exists.

On the basis of the third embodiment, as shown in fig. 13, in the fourth embodiment of the present application, the second controller is further configured to execute: receiving first indication information sent by a first controller, wherein the first indication information is sent by the first controller when responding to the operation of switching a first thumbnail window into a full-screen call interface, controlling a first application to stop receiving video data from a virtual room, and switching the playing source into a second application; controlling a second application to receive video data from the virtual room in response to the first indication information, and sending a third processing instruction to a first controller; the third processing instruction is used for instructing the first controller to control the display to hide the first thumbnail window and controlling the display to display a full-screen call interface.

In the mode of the play source of the first controller and the first thumbnail window, the user does not actively switch the play source, but switches the first thumbnail window to a full-screen call interface, and because the first application does not support the video call in the full-screen mode, the first application needs to automatically trigger the switch of the play source of the first controller to the second application, and the second application establishes the full-screen call interface.

The user can input the operation of switching the first thumbnail window to the full-screen call interface by clicking a window switching control on the first thumbnail window, the first controller responds to the operation and switches the playing source to the second application, controls the first application to stop receiving video data from the virtual room, and sends first indication information to the second controller, wherein the first indication information is used for indicating the user to switch the first thumbnail window to the full-screen call interface to the second controller. The second controller receives the first indication information, controls the second application to receive video data from the virtual room, so that the second application starts a full-screen mode video call according to the audio and video data, correspondingly generates a third processing instruction, and then sends the third processing instruction to the first controller; the first controller receives and responds to the third processing instruction, controls the display to display the full-screen call interface, and simultaneously controls the display to hide the first thumbnail window so that the first thumbnail window is not displayed any more, and the full-screen call interface is prevented from being blocked.

In this embodiment, the user does not actively switch the playing source, but automatically triggers the switching of the playing source of the first controller to the second application only when the user enlarges the first thumbnail window to the full-screen call interface, and when the user actively/passively hangs up the video call (a-1-1) or switches to the thumbnail window mode (a-1-2) on the full-screen call interface, the user needs to automatically return to the playing state of the playing source before the user switches to the full-screen call interface.

Referring to fig. 13, in the fourth embodiment of the present application, corresponding to the case (a-1-1), the second controller is further configured to perform: when the signaling session information indicates that the call state of each call object is hung up, or a call hanging up instruction input by a user is responded, controlling the second application to exit the current video call, and sending a sixth processing instruction to the first controller; the sixth processing instruction is used for instructing the first controller to control the display to display the picture of the playing source according to the ID of the playing source, and controlling the first application to destroy the first thumbnail window.

When the second controller responds to the first indication information, namely, responds to the operation of switching the first thumbnail window to the full-screen call interface, the second controller needs to record the ID of the playing source before the first thumbnail window switches to the full screen, wherein the playing source is sourced from the first controller, the ID is used for identifying the playing source, the ID of the playing source is the equipment ID of a signal source or the ID of an application program, and the record corresponds to the playing source ID of the playing source switched to the non-second controller last time.

When the signaling session information indicates that the call state of each call object is hung up (i.e. the home terminal user is passively hung up), or in response to a call hang-up instruction input by the user (i.e. the home terminal user is actively hung up), the second application exits the video call in the current full-screen mode, so that the second controller side hangs up the video call, and then sends a sixth processing instruction to the first controller, wherein the sixth processing instruction should carry the ID of the playing source recorded by the second controller; the first controller receives and responds to the sixth processing instruction, and controls the display to display the play source corresponding to the ID according to the ID of the play source recorded by the second controller, and as the user hangs up the video call, only the play source corresponding to the recording ID is displayed on the display, and the first thumbnail window is not displayed, namely, the play source (recording ID) +the first thumbnail window→the second application play source+the full-screen call interface→the play source corresponding to the ID of the first controller is realized, so that the play state before the first thumbnail window switches the full-screen call interface is restored.

For example, while watching a cable television, a user performs a video call through a first thumbnail window, during which the user clicks a switching window control on the first thumbnail window to switch the first thumbnail window to a full-screen call interface, and then switches to a second application playing source, the second controller records the device ID of the cable television set-top box, then controls the display to display the full-screen call interface, and after the call is ended, the user actively/passively hangs up the video call, and then the first controller switches the playing source to a cable television signal output by the set-top box according to the previously recorded device ID, and returns to the cable television for playing.

In practical application, when the mode of the playing source of the first controller and the first thumbnail window is switched to the full-screen call interface, the first thumbnail window needs to be hidden during switching so as to avoid the first thumbnail window from shielding the full-screen call interface picture. When the user actively or passively hangs up the video call while displaying the second thumbnail window, the second application needs to destroy the second thumbnail window, namely the second application exits the current video call, at this time, the first thumbnail window generated by the first application is in a hidden state, but the first thumbnail window is invalid because the current video call is hung up, so that the first thumbnail window can be destroyed at the same time, so that the first controller and the second controller synchronously exit the current video call, and the reliability and the accuracy of the video call are ensured.

Alternatively, on the basis of the third embodiment, if the full-screen call interface is changed back to the thumbnail window mode, that is, corresponds to the situation (a-1-2), as shown in fig. 14, in the fifth embodiment of the present application, the second controller is further configured to execute: responding to clicking operation of a user on the full-screen call interface on a window switching control, controlling a second application to stop receiving video data from a virtual room, and sending a seventh processing instruction to a first controller; the seventh processing instructions are configured to instruct the first controller to perform: controlling the display to display the picture of the playing source according to the ID of the playing source; and controlling the first application to receive video data from the virtual room, and controlling the display to display a first thumbnail window on the upper layer of the interface of the playing source.

If the user clicks the window switching control on the full-screen call interface, namely, the full-screen call interface is switched to a thumbnail window mode, the second application shall stop receiving video data from the virtual room and send a seventh processing instruction to the first controller, wherein the seventh processing instruction shall carry the ID of the playing source recorded by the second controller; the first controller receives and responds to the seventh processing instruction, and controls the display to display a play source corresponding to the ID according to the play source ID recorded by the second controller, and as the video call is not hung up and the current play source is derived from the first controller, the first application pulls a stream from the virtual room, and then displays the video call interface in a first thumbnail window form, namely, the play source (record ID) +the first thumbnail window of the first controller, the play source of the second application+the full-screen call interface, the play source corresponding to the ID+the first thumbnail window are realized, so that the play state before the first thumbnail window is switched to the full-screen call interface is restored.

For the case (a-2), as shown in fig. 15, in the sixth embodiment of the present application, the second controller is further configured to perform: receiving second indication information sent by the first controller, wherein the second indication information is sent by the first controller when responding to the operation of switching the playing source into a target application program, and controlling the first application to stop receiving video data from the virtual room; the target application program is other application programs except the second application in the second controller; controlling a second application to receive video data from the virtual room in response to the second instruction information, and sending a fourth processing instruction to the first controller; the fourth processing instruction is used for instructing the first controller to control the display to hide the first thumbnail window and controlling the display to display a second thumbnail window on the upper layer of the interface of the target application program; the second thumbnail window is a call window obtained by reducing the full-screen call interface to a preset size.

In the "play source of the first controller+first thumbnail window" mode, if the user switches the current play source to the target application program, i.e., to another application program in the second controller except the second application, for example, the user may click a home key on the remote controller, so that the display device displays the home page, then clicks an icon of the target application program from the home page, starts the target application program, i.e., inputs an operation of switching the play source to the target application program, when the target application program runs, the video call interface is displayed in the form of the second thumbnail window, for example, the user switches to a certain video play application, and when the network video is viewed through the video play application, the video call can be simultaneously performed by using the second thumbnail window.

When a user switches a play source to a target application program, namely to other application programs in a second controller, the first application is required to stop streaming from the virtual room, and the first controller is required to send second instruction information to the second controller; the second controller receives and responds to the second instruction information to control the second application to pull streams from the virtual room, the second application starts second thumbnail window video call logic, and then a fourth processing instruction is sent to the first controller; the first controller receives and responds to the fourth processing instruction, controls the display to display in a mode of 'target application program plus second thumbnail window', and simultaneously controls the display to hide the first thumbnail window so as to prevent the first thumbnail window from shielding the interface of the target application program and ensure that the first thumbnail window and the second thumbnail window are not repeatedly displayed.

For (A-2), in the "target application+second thumbnail window" mode, several situations may occur:

(a-2-1) switching the target application program to another application program N in the second controller other than the second application. For (a-2-1), since the play source is still derived from the second controller, only the target application needs to be displayed as application N instead, and the other display forms are unchanged, i.e. the second thumbnail window is still displayed.

(a-2-2) the user switching the target application to the play source in the first controller. For (A-2-2), similar to the logic and control procedure of (A) above, please refer to the related description of the case (A) and the sub-cases included in (A), which will not be repeated here.

(a-2-3) the user switching the second thumbnail window to a full screen call interface. In this case, only the display needs to be controlled to display the full-screen call interface, and the current playing source remains as the target application. Specifically, as shown in fig. 16, the second controller is configured to: and responding to the operation of switching the second thumbnail window to the full-screen call interface, recording the ID of the playing source before switching (namely the ID of the target application program), and sending an eleventh processing instruction to the first controller, wherein the eleventh processing instruction is used for enabling the first controller to control the display to display the full-screen call interface.

(a-2-4), the current video call is cancelled, including a call hang-up instruction input by the local end user by clicking a hang-up control on the second thumbnail window, that is, the local end user actively hangs up the video call, exits the virtual room, and the local end user does not actively hang up the video call, but all other call objects except the local end user in the virtual room hang up the video call, so that the current video call does not need to be maintained, which is equivalent to that the local end user passively hangs up the video call, and the call state of the call object in the virtual room can be obtained through signaling session information to determine whether the passive hang-up condition exists.

For (A-2-4), as shown in FIG. 15, the second controller is further configured to perform: when the signaling session information indicates that the call state of each call object is hung up, or a call hanging up instruction input by a user is responded, controlling a second application to exit the current video call, and sending a tenth processing instruction to a first controller; the tenth processing instruction is used for instructing the first controller to control the first application to destroy the first thumbnail window in the hidden state, and the current playing source of the first controller is not switched. When the user actively/passively hangs up the video call, the second application destroys the second thumbnail window, namely the second controller side exits the current video call, and then a tenth processing instruction is sent to the first controller; the first controller receives and responds to the tenth processing instruction, controls the first application to destroy the hidden first thumbnail window, namely the first controller side synchronously exits the video call, and does not switch the playing source when closing the thumbnail window in the thumbnail window mode, namely the display still displays the picture of the target application program. Specifically, for (a-2-3), when switching to the full-screen call interface in the "target application+second thumbnail window" mode, the second controller also needs to record the ID of the current playing source, that is, the ID of the target application, so that the user can actively/passively hang up the video call (a-2-3-1) or switch to the thumbnail window mode (a-2-3-2) on the full-screen call interface, and can automatically return to the playing state before the user switches to the full-screen call interface.

Referring to fig. 16, in the seventh embodiment of the present application, corresponding to the case (a-2-3-1), after switching the second thumbnail window to the full-screen call interface, the second controller is further configured to perform: when the signaling session information indicates that the call state of each call object is hung up, or a call hanging up instruction input by a user is responded, controlling the second application to exit the current video call, and sending an eighth processing instruction to the first controller; the eighth processing instruction is configured to instruct the first controller to control the display to display a picture of the play source according to the ID of the play source.

When the second controller responds to the operation of switching the second thumbnail window to the full-screen call interface, the second controller needs to record the ID of the playing source before the second thumbnail window switches to the full screen, wherein the playing source is derived from the second controller, and in the case, the playing source is the target application program, and the ID of the playing source is the target application program.

When the signaling session information indicates that the call state of each call object is hung up (i.e. the home terminal user is passively hung up), or in response to a call hang-up instruction input by the user (i.e. the home terminal user is actively hung up), the second application destroys the full-screen call interface, i.e. the video call in the current full-screen mode is exited, so that the second controller side hangs up the video call, then an eighth processing instruction is sent to the first controller, and the eighth processing instruction should carry the ID of the playing source recorded by the second controller; the first controller receives and responds to the eighth processing instruction, and controls the display to display the play source corresponding to the ID according to the ID of the play source recorded by the second controller, and as the user hangs up the video call, only the play source corresponding to the recording ID is displayed on the display, and the first thumbnail window and the second thumbnail window are not displayed, so that the play source corresponding to the target application program (recording ID) +the second thumbnail window- & gtfull screen call interface- & gtID is realized, and the play state before the second thumbnail window switches the full screen call interface is restored. Meanwhile, the first thumbnail window in the hidden state can be destroyed, so that the first controller and the second controller synchronously withdraw the current video call, and the reliability and the accuracy of the video call are ensured.

Alternatively, for (a-2-3-2), if the full-screen call interface is changed back to the second thumbnail window mode, as shown in fig. 17, in the eighth embodiment of the present application, the second controller is further configured to perform: responding to clicking operation of a user on the full-screen call interface on a window switching control, and sending a ninth processing instruction to a first controller; the ninth processing instruction is configured to instruct the first controller to perform: and controlling the display to display the picture of the playing source according to the ID of the playing source (namely the ID of the target application program), and controlling the display to display a second thumbnail window on the upper layer of the interface of the playing source.

If the user clicks the window switching control on the full-screen call interface, namely the full-screen call interface is switched to a thumbnail window mode, the second application still processes the video call because the current signal source is derived from the second controller, namely the second application should keep receiving video data from the virtual room, and send a ninth processing instruction to the first controller, wherein the ninth processing instruction should carry the ID of the playing source recorded by the second controller; the first controller receives and responds to the ninth processing instruction, and controls the display to display the play source corresponding to the ID according to the play source ID recorded by the second controller, and the video call interface is displayed in a second thumbnail window form because the video call is not hung up, namely, the play source corresponding to the target application program (recording ID) +the second thumbnail window- & gt the full-screen call interface- & gt the ID- & gt the second thumbnail window is realized, so that the play state before the second thumbnail window is switched to the full-screen call interface is restored.

For (a-3), referring to fig. 12, when the display shows the first thumbnail window, if the user clicks the hang-up control or passively hangs up on the first thumbnail window, the second controller is further configured to perform: receiving third indication information sent by the first controller, wherein the third indication information is sent when the signaling session information received by the first controller indicates that the call state of each call object is hung up, or the first application is controlled to exit the current video call in response to a call hanging-up instruction input by a user; and responding to the third indication information, and controlling the second application to synchronously withdraw the video call.

When the display displays the first thumbnail window, if the user actively/passively hangs up the video call, the second application destroys the first thumbnail window, namely the first controller side exits the current video call, and then sends third indication information to the second controller; the second controller receives and responds to the third indication information to control the second application to synchronously withdraw the video call, namely the second application destroys a video call interface (which can be a full-screen call interface or a second thumbnail window) created by the second application, so that the first controller synchronously withdraws from the video call, and in the thumbnail window mode, a playing source is not switched when the thumbnail window is closed, and the current playing source is maintained unchanged.

It should be noted that, when the play source is switched from the second controller to the first controller, the full-screen call interface or the second thumbnail created by the second application cannot be displayed on the display, so that only the first thumbnail created by the first application is displayed on the display; when the playing source is switched from the first controller to the second controller, the first thumbnail window is displayed on the display, so that the first thumbnail window is needed to be hidden to avoid repeated display of the call interface.

For the case (B), in the process of processing the video call by the second application in the form of a full-screen call interface, if the user inputs an operation of switching the play source, when the play source is an application program installed in the second controller except for the second application, as shown in fig. 18, in the ninth embodiment of the present application, the second controller is specifically configured to execute: in response to the play source switching operation, sending a second processing instruction to the first controller, wherein the second processing instruction is used for instructing the first controller to execute: starting the playing source; and controlling a display to display a second thumbnail window on the upper layer of the interface of the playing source, wherein the second thumbnail window is a call window obtained by reducing the full-screen call interface to a preset size.

The original playing source is the second application, and the second application program is switched to other application programs except the second application program in the second controller, and the second controller sends a second processing instruction to the first controller; the first controller receives and responds to the second processing instruction, switches the playing source, and simultaneously displays a second thumbnail window to realize chat while watching.

For the case (B), that is, "the application H in the second controller except the second application+the second thumbnail window", there may be a case (B-1) of switching the play source to another application M in the second controller. For (B-1), since the play source is still derived from the second controller, only the application H needs to be displayed as the application M instead, and the other display forms are unchanged, i.e. the second thumbnail window is still displayed.

(B-2) the user switching the application H to the play source in the first controller. For (B-2), similar to some logic and control procedures described in (A), please refer to the description of (A) and the sub-cases included in (A), and the description is omitted here.

And (B-3) canceling the current video call, wherein the current video call comprises a call hang-up instruction input by the local end user by clicking a hang-up control on the second thumbnail window, namely the local end user actively hangs up the video call and exits the virtual room, and the local end user does not actively hang up the video call, but all other call objects except the local end user in the virtual room hang up the video call, so that the current video call does not need to be continuously maintained, which is equivalent to that the local end user passively hangs up the video call, and the call state of the call object in the virtual room can be obtained through signaling session information so as to determine whether the passive hang-up condition exists. Because (B-3) is not switched to the playing source of the first controller, the first controller is not started, and only the second application directly destroys the second thumbnail window, the current video call can be exited, and the current playing source application program H is kept unchanged.

(B-4) the user switching the second thumbnail window to a full screen call interface. In this case, only the display needs to be controlled to display the full-screen call interface, and the current playing source remains as the target application.

Specifically, for (B-4), when switching to the full-screen call interface in the "application h+second thumbnail window except for the second application" mode in the second controller, the second controller also needs to record the ID of the play source before switching, that is, the ID of the application H, and then send an eleventh processing instruction to the first controller, where the eleventh processing instruction is used to cause the first controller to display the full-screen call interface. The purpose of recording the ID of the play source before switching is to automatically return to the play state before the user switches the full-screen call interface when the user actively/passively hangs up the video call (B-4-1) or switches to the thumbnail window mode (B-4-2) on the full-screen call interface.

Referring to fig. 18, in embodiment ten of the present application, corresponding to the case (B-4-1), after switching the second thumbnail window to the full-screen call interface, the second controller is further configured to perform: when the signaling session information indicates that the call state of each call object is hung up, or a call hanging up instruction input by a user is responded, controlling the second application to exit the current video call, and sending an eighth processing instruction to the first controller; the eighth processing instruction is configured to instruct the first controller to control the display to display a picture of the play source according to the ID of the play source.

When the second controller responds to the operation of switching the second thumbnail window to the full-screen call interface, the second controller needs to record the ID of the playing source before the second thumbnail window switches to the full screen, wherein the playing source is derived from the second controller, and in the case, the ID of the playing source is the ID of the application program H.

When the signaling session information indicates that the call state of each call object is hung up (i.e. the home terminal user is passively hung up), or in response to a call hang-up instruction input by the user (i.e. the home terminal user is actively hung up), the second application destroys the full-screen call interface, i.e. the video call in the current full-screen mode is exited, so that the second controller side hangs up the video call, then an eighth processing instruction is sent to the first controller, and the eighth processing instruction should carry the ID of the playing source recorded by the second controller; the first controller receives and responds to the eighth processing instruction, and controls the display to display the play source corresponding to the ID according to the ID of the play source recorded by the second controller, and as the user hangs up the video call, only the play source corresponding to the recording ID is displayed on the display, and the second thumbnail window is not displayed, namely, the play source (application program H) corresponding to the application program H (recording ID) +the second thumbnail window- & gtfull screen call interface- & gtID is realized, so that the play state before the second thumbnail window switches the full screen call interface is restored. And (B-4-1) is not switched to the playing source of the first controller, so that the first controller is not started up, and only the second application directly destroys the second thumbnail window and exits the current video call.

Alternatively, for (B-4-2), if the full-screen call interface is changed back to the second thumbnail window mode, as shown in fig. 19, in the eleventh embodiment of the present application, the second controller is further configured to perform: responding to clicking operation of a user on the full-screen call interface on a window switching control, and sending a ninth processing instruction to a first controller; the ninth processing instruction is configured to instruct the first controller to perform: and controlling the display to display the picture of the playing source according to the ID of the playing source, and controlling the display to display a second thumbnail window on the upper layer of the interface of the playing source.

If the user clicks the window switching control on the full-screen call interface, namely the full-screen call interface is switched to a thumbnail window mode, the second application still processes the video call because the current signal source is derived from the second controller, namely the second application should keep receiving video data from the virtual room, and send a ninth processing instruction to the first controller, wherein the ninth processing instruction should carry the ID of the playing source recorded by the second controller; the first controller receives and responds to the ninth processing instruction, and controls the display to display the play source corresponding to the ID according to the play source ID recorded by the second controller, and the video call interface is displayed in a second thumbnail window form because the video call is not hung up, namely, the play state before the second thumbnail window switches the full-screen call interface is restored by realizing the application program H (recording ID) +the second thumbnail window- & gt the full-screen call interface- & gt the play source corresponding to the ID (application program H) +the second thumbnail window.

After (B-4-2), the video call may be actively/passively suspended in the second thumbnail state, or the second thumbnail may be switched to a full-screen call interface, or the playing source may be switched, which will not be described in detail herein.

For the interactive flow charts of several video call logics shown in the embodiments of the present application, in these charts, a first display device is used as an initiator and a second display device is used as an invitee to perform an exemplary illustration, the first display device initiates a video call, that is, a current session ID can be generated, the server adds the current session ID to the signaling session information, and after receiving the signaling session information, the second display device receives the signaling session information, the invitee user answers and adds to the video call corresponding to the current session ID, that is, enters a virtual room corresponding to the current session ID. The virtual room described in the specification is not a room with a physical structure, but is a virtual conversation space constructed by a video conversation application, and a user can join or exit to realize multi-person video chat.

It should be noted that, in the video call logic in the above embodiments, the display state of the first thumbnail window, the second thumbnail window or the full-screen call interface and/or the playing state of the audio player need to be dynamically matched and adjusted according to the updated signaling session information at any time. The second controller is further configured to perform:

Receiving updated signaling session information sent by the server;

if the video call is processed by the second application, that is, when the playing source is from the second controller, updating the display state of the full-screen call interface or the second thumbnail window and/or updating the playing state of the sound player according to the updated signaling session information;

and if the video call is processed by the first application, namely, the playing source is from the first controller, sending the updated signaling session information to the first controller, so that the first controller updates the display state of the first thumbnail window and/or updates the playing state of the sound player according to the updated signaling session information.

When the video call is displayed in the form of the thumbnail window, the user can switch the thumbnail window into a full-screen call interface at any time, and when the window switching operation is responded, the ID of the playing source before the window is not switched needs to be recorded, so that when the user switches the full-screen call interface into the thumbnail window mode again, or the user hangs up the video call on the full-screen call interface, the display can redisplay the playing source corresponding to the ID, namely, the playing state before the thumbnail window is switched to the full screen is restored, the accurate control of playing is ensured, and the user experience is improved. The application sets the adaptive video call processing logic based on the user operation type, improves the efficiency, accuracy and reliability of the dual-system display equipment for video call processing, and improves the user experience.

It should be noted that, the present application is based on adjustability of the window mode/size of the video call interface, and the user may modify the play source at any time in the display device according to the viewing desire, so as to enumerate application situations that may be encountered in the foregoing embodiments, and because the user may perform different operations under different call states, call phases and moments, adaptive combination may be performed on portions of the application situations, so as to obtain an interaction flow of the video call processing logic that is not limited to the foregoing embodiments. However, in practical application, the method is not limited to the above application cases, and a person skilled in the art may adaptively set the video call processing logic in other application situations according to the application cases and the video call processing logic listed in the embodiments of the present application.

It should be noted that, because the present application relates to the interaction between the first controller and the second controller, that is, to the problem of which controller the user operation command is to be responded by, the first controller needs to record the controller to which the real-time recording play source belongs, and then forwards the received user operation to the controller corresponding to the ID recorded currently to respond. For example, when the second application starts full-screen call, the first controller records the ID of the second controller, and when the first controller receives the operation of switching the thumbnail window input by the user, the first controller forwards the operation instruction to the second controller to respond; for another example, when the mode is currently "play source of first controller+first thumbnail window", the first controller records ID of the first controller, and when the first controller receives the operation of switching to the target application program input by the user, the first controller queries that the current ID indicates the first controller, and then the first controller responds to the operation instruction.

The embodiments are written from the perspective of the second controller, that is, the execution body is the second controller, and correspondingly, the video call processing logic may also be described from the perspective of the first controller, so that the logic of the first controller may refer to the same or similar description, and will not be repeated herein.

In some embodiments, the present application also provides a display apparatus including: a display; a sound player; the communicator is used for being in communication connection with the server; a user input interface for receiving an operation input by a user; a first controller connected with the display and the sound player, and a second controller connected with the first controller and the server in a communication way respectively;

In some embodiments, when the play source is a signal source connected to the first controller or an application installed in the first controller, the first controller is specifically configured to perform:

receiving a first processing instruction carrying the signaling session information sent by a second controller, wherein the first processing instruction is sent by the second controller when the second controller responds to the operation of switching the playing source to control the second application to stop receiving video data from the virtual room;

responding to the first processing instruction, and starting the playing source and the first application; the method comprises the steps that a first application is controlled to join a virtual room of a video call according to a current session ID, and audio and video data are received from the virtual room; generating a first thumbnail window according to the call state and the camera state of each call member and the video data received from the virtual room, controlling a display to display the first thumbnail window on the upper layer of the interface of the playing source, and controlling a sound player to play the audio data received from the virtual room according to the microphone state of each call member.

In some embodiments, when the play source is an application installed in the second controller other than the second application, the first controller is specifically configured to perform:

receiving a second processing instruction sent by a second controller, wherein the second processing instruction is sent by the second controller in response to the operation of switching the playing source;

responding to the second processing instruction, and starting the playing source; and controlling a display to display a second thumbnail window on the upper layer of the interface of the playing source, wherein the second thumbnail window is a call window obtained by reducing the full-screen call interface to a preset size.

In some embodiments, the first controller is further configured to perform:

responding to the operation of switching the first thumbnail window into a full-screen call interface, controlling a first application to stop receiving audio and video data from a virtual room, switching the playing source into a second application, and sending first indication information to the second controller;

receiving a third processing instruction sent by a second controller, wherein the third processing instruction is sent by the second controller when responding to the first indication information and controlling a second application to receive video data from the virtual room;

and responding to the third processing instruction, controlling the display to hide the first thumbnail window, and controlling the display to display a full-screen call interface.

In some embodiments, the first controller is further configured to perform:

controlling a first application to stop receiving video data from a virtual room and sending second instruction information to a second controller in response to the operation of switching the play source to the target application; the target application program is other application programs except the second application in the second controller;

receiving a fourth processing instruction sent by a second controller, wherein the fourth processing instruction is sent by the second controller when responding to the second instruction information and controlling a second application to receive video data from the virtual room;

controlling a display to hide the first thumbnail window and controlling the display to display a second thumbnail window on the upper layer of the interface of the target application program in response to the fourth processing instruction; the second thumbnail window is a call window obtained by reducing the full-screen call interface to a preset size.

In some embodiments, before the second controller receives the operation of switching the play source, the first controller is further configured to perform:

receiving a fifth processing instruction sent by a second controller, wherein the fifth processing instruction is sent by the second controller when responding to clicking operation of a window switching control on the full-screen call interface by a user;

Responding to the fifth processing instruction, controlling a display to display a preset page, and controlling the display to display a second thumbnail window on the upper layer of the preset page; the second thumbnail window is a call window obtained by reducing the full-screen call interface to a preset size; the preset page is a homepage of the display device or a homepage of the second application.

In some embodiments, after the first thumbnail window is switched to the full-screen call interface, the first controller is specifically configured to perform:

receiving a sixth processing instruction sent by the second controller, wherein the sixth processing instruction is sent by the second controller when the signaling session information indicates that the call state of each call object is hung up, or responding to a call hanging up instruction input by a user, and controlling the second application to exit the current video call; responding to the sixth processing instruction, controlling the display to display the picture of the playing source according to the ID of the playing source, and controlling the first application to destroy the first thumbnail window;

or receiving a seventh processing instruction sent by the second controller, wherein the seventh processing instruction is sent when the second controller responds to the click operation of a window switching control on the full-screen call interface by a user and controls the second application to stop receiving video data from the virtual room; and responding to the seventh processing instruction, controlling the display to display the picture of the playing source according to the ID of the playing source, controlling the first application to receive video data from the virtual room, and controlling the display to display a first thumbnail window on the upper layer of the interface of the playing source.

In some embodiments, after the second thumbnail window is switched to the full-screen call interface, the first controller is specifically configured to perform:

receiving an eighth processing instruction sent by the second controller, wherein the eighth processing instruction is sent by the second controller when the signaling session information indicates that the call state of each call object is hung up, or responding to a call hanging up instruction input by a user, and controlling the second application to exit the current video call; responding to the eighth processing instruction, and controlling the display to display the picture of the playing source according to the ID of the playing source;

or receiving a ninth processing instruction sent by the second controller, wherein the ninth processing instruction is sent by the second controller in response to clicking operation of a window switching control on the full-screen call interface by a user; and responding to the ninth processing instruction, controlling the display to display the picture of the playing source according to the ID of the playing source, and controlling the display to display a second thumbnail window on the upper layer of the interface of the playing source.

In some embodiments, the first controller is further configured to perform:

receiving a tenth processing instruction sent by a second controller, wherein the tenth processing instruction is sent by the second controller when the signaling session information indicates that the call state of each call object is hung up, or responding to a call hanging up instruction input by a user, and controlling a second application to exit from the current video call; and responding to the tenth processing instruction, controlling a first application to destroy the first thumbnail window in a hidden state, and not switching the current playing source of the first controller.

In some embodiments, when the display displays the first thumbnail window, the first controller is further to perform:

when the signaling session information indicates that the call state of each call object is hung up, or a call hanging up instruction input by a user is responded, the first application is controlled to exit the current video call, and third indication information is sent to the second controller; the third indication information is used for indicating the second controller to control the second application to synchronously withdraw the video call.

The application also provides an embodiment of a video call processing method, which is used for a second controller and comprises the following steps:

The application also provides an embodiment of a video call processing method, which is used for the first controller and comprises the following steps:

The above two method embodiments may refer to description and expansion of the display device embodiments, and are not repeated herein.

The same or similar content may be referred to each other in each embodiment of the present application, and the related embodiments will not be described in detail.

It will be apparent to those skilled in the art that the techniques of embodiments of the present application may be implemented in software plus a necessary general purpose hardware platform. In a specific implementation, the present application also provides a computer storage medium, where the computer storage medium may store a program, where the program may include program steps of video call processing logic configured to be executed by a first controller and a second controller in a display device. The computer storage medium may be a magnetic disk, an optical disk, a read-only memory (ROM), a random-access memory (random access memory, RAM), or the like.

Other embodiments of the application will be apparent to those skilled in the art from consideration of the specification and practice of the application disclosed herein. This application is intended to cover any variations, uses, or adaptations of the application following, in general, the principles of the application and including such departures from the present disclosure as come within known or customary practice within the art to which the application pertains. It is intended that the specification and examples be considered as exemplary only, with a true scope of the application being indicated by the following claims. The true scope and spirit of the application is indicated by the following claims.

Claims

1. A display device, characterized by comprising:

a display;

a sound player;

the communicator is used for being in communication connection with the server;

a user input interface for receiving an operation input by a user;

the second controller is configured to perform:

in response to the operation of switching the playing source, controlling the first controller to start the playing source, and enabling the first controller to control the display to display a video call interface in a thumbnail window mode on the upper layer of the interface of the playing source, wherein when the playing source is an application program installed in a second controller except for a second application, a second processing instruction is sent to the first controller, and the second processing instruction is used for instructing the first controller to execute: starting the playing source; controlling a display to display a second thumbnail window on the upper layer of the interface of the playing source, wherein the second thumbnail window is a call window obtained by reducing the full-screen call interface to a preset size;

2. The display device according to claim 1, wherein when the play source is a signal source connected to the first controller or an application program installed in the first controller, the second controller is specifically configured to perform:

in response to switching the play source operation, controlling the second application to stop receiving video data from the virtual room, and sending a first processing instruction carrying the signaling session information to the first controller, wherein the first processing instruction is configured to instruct the first controller to execute: starting the playing source and the first application; the method comprises the steps that a first application is controlled to join a virtual room of a video call according to a current session ID, and audio and video data are received from the virtual room; generating a first thumbnail window according to the call state and the camera state of each call member and the video data received from the virtual room, controlling a display to display the first thumbnail window on the upper layer of the interface of the playing source, and controlling a sound player to play the audio data received from the virtual room according to the microphone state of each call member.

3. The display device of claim 2, wherein the second controller is further configured to perform:

receiving first indication information sent by a first controller, wherein the first indication information is sent by the first controller when responding to the operation of switching a first thumbnail window into a full-screen call interface, controlling a first application to stop receiving video data from a virtual room, and switching the playing source into a second application;

controlling a second application to receive video data from the virtual room in response to the first indication information, and sending a third processing instruction to a first controller; the third processing instruction is used for instructing the first controller to control the display to hide the first thumbnail window and controlling the display to display a full-screen call interface.

4. The display device of claim 2, wherein the second controller is further configured to perform:

receiving second indication information sent by the first controller, wherein the second indication information is sent by the first controller when responding to the operation of switching the playing source into a target application program, and controlling the first application to stop receiving video data from the virtual room; the target application program is other application programs except the second application in the second controller;

Controlling a second application to receive video data from the virtual room in response to the second instruction information, and sending a fourth processing instruction to the first controller; the fourth processing instruction is used for instructing the first controller to control the display to hide the first thumbnail window and controlling the display to display a second thumbnail window on the upper layer of the interface of the target application program; the second thumbnail window is a call window obtained by reducing the full-screen call interface to a preset size.

5. The display device of claim 1, wherein the second controller is further configured to, prior to receiving the switch play source operation, perform:

responding to clicking operation of a window switching control on the full-screen call interface by a user, and sending a fifth processing instruction to the first controller, wherein the fifth processing instruction is used for instructing the first controller to control a display to display a preset page and controlling the display to display a second thumbnail window on the upper layer of the preset page; the second thumbnail window is a call window obtained by reducing the full-screen call interface to a preset size; the preset page is a homepage of the display device or a homepage of the second application.

6. The display device of claim 3, wherein after switching the first thumbnail window to a full screen talk interface, the second controller is further configured to perform:

When the signaling session information indicates that the call state of each call object is hung up, or a call hanging up instruction input by a user is responded, controlling the second application to exit the current video call, and sending a sixth processing instruction to the first controller; the sixth processing instruction is used for instructing the first controller to control the display to display the picture of the playing source according to the ID of the playing source, and controlling the first application to destroy the first thumbnail window;

or, in response to the clicking operation of the window switching control on the full-screen call interface by the user, controlling the second application to stop receiving video data from the virtual room, and sending a seventh processing instruction to the first controller; the seventh processing instructions are configured to instruct the first controller to perform: controlling the display to display the picture of the playing source according to the ID of the playing source; and controlling the first application to receive video data from the virtual room, and controlling the display to display a first thumbnail window on the upper layer of the interface of the playing source.

7. The display device of claim 1, wherein after switching the second thumbnail window to a full screen talk interface, the second controller is further configured to perform:

When the signaling session information indicates that the call state of each call object is hung up, or a call hanging up instruction input by a user is responded, controlling the second application to exit the current video call, and sending an eighth processing instruction to the first controller; the eighth processing instruction is configured to instruct the first controller to control the display to display a picture of the play source according to the ID of the play source;

or, responding to the click operation of the user on the full-screen call interface on the window switching control, and sending a ninth processing instruction to the first controller; the ninth processing instruction is configured to instruct the first controller to perform: and controlling the display to display the picture of the playing source according to the ID of the playing source, and controlling the display to display a second thumbnail window on the upper layer of the interface of the playing source.

8. The display device of claim 4, wherein the second controller is further configured to perform:

when the signaling session information indicates that the call state of each call object is hung up, or a call hanging up instruction input by a user is responded, controlling a second application to exit the current video call, and sending a tenth processing instruction to a first controller; the tenth processing instruction is used for instructing the first controller to control the first application to destroy the first thumbnail window in the hidden state, and the current playing source of the first controller is not switched.

9. The display device of claim 2 or 6, wherein when the display displays the first thumbnail window, the second controller is further configured to perform:

receiving third indication information sent by the first controller, wherein the third indication information is sent when the signaling session information received by the first controller indicates that the call state of each call object is hung up, or the first application is controlled to exit the current video call in response to a call hanging-up instruction input by a user; and responding to the third indication information, and controlling the second application to synchronously withdraw the video call.

10. A display device, characterized by comprising:

a display;

a sound player;

the communicator is used for being in communication connection with the server;

a user input interface for receiving an operation input by a user;

the first controller is configured to perform: in the process that a second controller controls a second application to carry out video call in the form of a full-screen call interface according to signaling session information received from a server, starting a play source according to a processing instruction sent by the second controller after responding to a play source switching operation, and controlling a display to display the video call interface in a thumbnail window mode on the upper layer of the interface of the play source, wherein when the play source is an application program which is installed in the second controller and is except for the second application, a second processing instruction sent by the second controller is received, and the second processing instruction is sent by the second controller when responding to the play source switching operation; responding to the second processing instruction, and starting the playing source; controlling a display to display a second thumbnail window on the upper layer of the interface of the playing source, wherein the second thumbnail window is a call window obtained by reducing the full-screen call interface to a preset size; according to a processing instruction sent by the second controller after responding to the operation of switching the thumbnail window into the full-screen call interface, controlling the display to display the full-screen call interface;

11. The display device according to claim 10, wherein when the play source is a signal source connected to a first controller or an application program installed in the first controller, the first controller is specifically configured to perform:

12. The display device of claim 11, wherein the first controller is further configured to perform:

13. The display device of claim 11, wherein the first controller is further configured to perform:

14. The display device of claim 10, wherein the first controller is further configured to perform, prior to the second controller receiving the switch play source operation:

15. The display device of claim 12, wherein after switching the first thumbnail window to a full-screen call interface, the first controller is specifically configured to perform:

16. The display device of claim 10, wherein after switching the second thumbnail window to a full screen call interface, the first controller is specifically configured to perform:

17. The display device of claim 13, wherein the first controller is further configured to perform:

18. The display device of claim 11 or 15, wherein when the display displays the first thumbnail window, the first controller is further configured to perform:

19. A video call processing method of a display device, the display device including a first controller, a second controller communicatively connected to the first controller and a server, respectively, a display and a sound player connected to the first controller, the method being for the second controller, comprising:

20. A video call processing method of a display device, the display device including a first controller, a second controller communicatively connected to the first controller and a server, respectively, a display and a sound player connected to the first controller, the method being for the first controller, comprising:

in the process that a second controller controls a second application to carry out video call in the form of a full-screen call interface according to signaling session information received from a server, starting a play source according to a processing instruction sent by the second controller after responding to a play source switching operation, and controlling a display to display the video call interface in a thumbnail window mode on the upper layer of the interface of the play source, wherein when the play source is an application program which is installed in the second controller and is except for the second application, a second processing instruction sent by the second controller is received, and the second processing instruction is sent by the second controller when responding to the play source switching operation; responding to the second processing instruction, and starting the playing source; controlling a display to display a second thumbnail window on the upper layer of the interface of the playing source, wherein the second thumbnail window is a call window obtained by reducing the full-screen call interface to a preset size; according to a processing instruction sent by the second controller after responding to the operation of switching the thumbnail window into the full-screen call interface, controlling the display to display the full-screen call interface;