WO2021088308A1

WO2021088308A1 - Display device and music recommendation method

Info

Publication number: WO2021088308A1
Application number: PCT/CN2020/083023
Authority: WO
Inventors: 吴超; 何宗承; 卢可敬
Original assignee: 海信视像科技股份有限公司
Priority date: 2019-11-04
Filing date: 2020-04-02
Publication date: 2021-05-14
Also published as: CN112788376A

Abstract

A display device and a music recommendation method. The method comprises: recording audio output by a first display when the first display carries out playing to obtain at least one segment of audio data; identifying, according to audio features of each segment of audio data, whether the content of the audio data is music; and acquiring music information of the music if the content of the audio data is music, so as to recommend, to a user and when the user views content played by the first display, the music information of music in the content being played.

Description

Display device and music recommendation method

This application is required to be submitted to the Chinese Patent Office on November 04, 2019, with the application number 2019110673937, The priority of the Chinese patent application titled "A display device and music recommendation method", all of which The contents of this part are incorporated into this application by reference.

Technical field

This application relates to the technical field of smart TVs, and in particular to a display device and a music recommendation method.

Background technique

With the continuous development of the smart TV technology field, the display device can provide users with the playback and display of audio, video, and pictures.

In the display process of the above-mentioned display device, the prior art generally displays a single content to be played to the user through the display device, or, when multiple displayed content is played, the content being played by the user is often blocked or the user's operation of the content being played is affected, and , It is impossible to recommend music related to the playback content based on the playback content, which reduces the user experience and is less interactive.

Summary of the invention

An exemplary embodiment of the present application provides a display device and a music recommendation method, which are used to recommend music related to the playback content to the user while watching the playback content.

According to one aspect of the exemplary embodiment, there is provided a display device, including:

A display, a controller, a communicator, and an audio output interface; the display includes a first display and a second display;

The communicator is configured to communicate and interact with the server;

The controller is configured to:

When the first display plays an audio and video picture, recording the audio corresponding to the audio and video picture output through the audio output interface to obtain at least one piece of audio data;

Identifying whether the content of the audio data is music according to the audio characteristics of the audio data;

If the content of the audio data is music, the music information of the music is obtained from the server, and the second display is controlled to display the music information; the music information includes: lyrics, song title, composer, At least one of lyricist, singer, and music dynamics.

In a second aspect, the present application provides a music recommendation method, which is applied to a display device with a dual-display structure, and is characterized in that it includes:

When the first display plays the audio and video images, periodically recording the audio corresponding to the audio and video images to obtain at least one piece of audio data;

According to the audio characteristics of each piece of audio data, identifying whether the content of the audio data is music;

If the content of the audio data is music, the music information of the music is acquired, and the second display is controlled to display the music information; the music information includes: lyrics, song title, composer, lyricist, singer At least one of the dynamic effects of music.

Description of the drawings

In order to more clearly describe the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings needed in the description of the embodiments. Obviously, the drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, without creative work, other drawings can be obtained based on these drawings.

Fig. 1 exemplarily shows a schematic diagram of an operation scenario between a display device and a control device according to an embodiment;

FIG. 2 exemplarily shows a configuration block diagram of the control device 100 according to an exemplary embodiment;

FIG. 3 exemplarily shows a schematic diagram of the hardware structure of the hardware system in the display device 200 according to an exemplary embodiment;

Figure 4 shows a schematic diagram of the connection relationship between the power board and the load;

FIG. 5 exemplarily shows a block diagram of the hardware architecture of the display device 200 shown in FIG. 3;

FIG. 6 exemplarily shows a schematic diagram of a functional configuration of a display device according to an exemplary embodiment;

FIG. 7 exemplarily shows a configuration block diagram of a software system in the display device 200 according to an exemplary embodiment;

FIG. 8 exemplarily shows that the application layer of the display device according to the exemplary embodiment includes various application programs executed on the display device 200;

FIG. 9 exemplarily shows a schematic diagram of a user interface in the display device 200 according to an exemplary embodiment;

[Corrected 12.06.2020 according to Rule 91]
FIG. 10 exemplarily shows a schematic diagram of a user interface in a display device 200 according to an exemplary embodiment;
FIG. 11 exemplarily shows a schematic diagram of a music push interface in the display device 200 according to an exemplary embodiment;

[Corrected 12.06.2020 according to Rule 91]
FIGS. 12a to 12d exemplarily show schematic diagrams of a music push interface in the display device 200 according to an exemplary embodiment;

FIG. 13 exemplarily shows a schematic diagram of a music push interface in the display device 200 according to an exemplary embodiment;

FIG. 14 exemplarily shows a schematic flowchart of a music recommendation method according to an exemplary embodiment;

FIG. 15 exemplarily shows a schematic flowchart of a music recommendation method according to an exemplary embodiment;

Fig. 16 exemplarily shows a song recommendation device according to an exemplary embodiment;

Fig. 17 also exemplarily shows a song recommendation device according to an exemplary embodiment.

Detailed ways

The following describes the technical solutions in the embodiments of the present application clearly and completely with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in this application, all other embodiments obtained by those of ordinary skill in the art without creative work shall fall within the protection scope of this application.

This application is mainly aimed at display devices with dual-system and dual-display structures, that is, display devices with a first controller (first hardware system), a second controller (second hardware system), a first display, and a second display For audio-visual synchronization processing, the following first describes the structure, function, and implementation of a display device with a dual-system hardware structure in detail.

For the convenience of users, various external device interfaces are usually provided on the display device to facilitate the connection of different peripheral devices or cables to achieve corresponding functions. When a high-resolution camera is connected to the interface of the display device, if the hardware system of the display device does not have the hardware interface of the high-pixel camera that receives the source code, it will cause the data received by the camera to be unable to present the data received by the camera to the display of the display device. On the screen.

Moreover, due to the hardware structure, the hardware system of traditional display devices only supports one hard decoding resource, and usually only supports 4K resolution video decoding. Therefore, when you want to realize the video chat while watching Internet TV, in order not to reduce The definition of the network video picture requires the use of hard decoding resources (usually the GPU in the hardware system) to decode the network video. In this case, the general-purpose processor (such as CPU) in the hardware system can only be used to decode the video. The video chat screen is processed by soft decoding.

Using soft decoding to process the video chat screen will greatly increase the data processing burden on the CPU. When the data processing burden on the CPU is too heavy, the picture may freeze or become unsmooth. Further, subject to the data processing capabilities of the CPU, when the CPU soft decoding is used to process the video chat screen, it is usually impossible to achieve multi-channel video calls. When the user wants to simultaneously video chat with multiple other users in the same chat scene, it will There is a situation where access is blocked.

Based on the above considerations, in order to overcome the above shortcomings, this application discloses a dual hardware system architecture to implement multiple channels of video chat data (at least one local video).

The concepts involved in the present application will be described below in conjunction with the drawings. It should be pointed out here that the following description of each concept is only to make the content of this application easier to understand, and does not mean to limit the scope of protection of this application.

The term "module" used in the various embodiments of this application can refer to any known or later developed hardware, software, firmware, artificial intelligence, fuzzy logic, or combination of hardware or/and software code that can execute related components Function.

The term "remote control" used in the various embodiments of this application refers to a component of an electronic device (such as the display device disclosed in this application), which can generally control the electronic device wirelessly within a short distance. This component can generally use infrared and/or radio frequency (RF) signals and/or Bluetooth to connect with electronic devices, and can also include functional modules such as WiFi, wireless USB, Bluetooth, and motion sensors. For example, a handheld touch remote control replaces most of the physical built-in hard keys in general remote control devices with the user interface in the touch screen.

The term "gesture" used in the embodiments of the present application refers to a user's behavior through a change of hand shape or hand movement to express expected ideas, actions, goals, and/or results.

The term "hardware system" used in the various embodiments of this application may refer to an integrated circuit (IC), printed circuit board (Printed circuit board, PCB) and other mechanical, optical, electrical, and magnetic devices with computing , Control, storage, input and output functions of the physical components. In the various embodiments of the present application, the hardware system is usually also referred to as a motherboard or a chip.

Fig. 1 exemplarily shows a schematic diagram of an operation scene between the display device and the control device according to the embodiment. As shown in FIG. 1, the user can operate the display device 200 by controlling the device 100.

Wherein, the control device 100 may be a remote controller 100A, which can communicate with the display device 200 through infrared protocol communication, Bluetooth protocol communication, ZigBee protocol communication or other short-distance communication methods for wireless or other short-distance communication. The display device 200 is controlled in a wired manner. The user can control the display device 200 by inputting user instructions through keys, voice input, control panel input, etc. on the remote controller 100A. For example: the user can control the display device by inputting corresponding control commands through the volume up and down keys on the remote control 100A, channel control keys, up/down/left/right movement keys, voice input keys, menu keys, switch machine keys, etc. 200 features.

The control device 100 may also be a smart device, such as a mobile terminal 100B, a tablet computer, a computer, a notebook computer, etc., which may be connected through a local area network (LAN, Wide Area Network), a wide area network (WAN, Wide Area Network), and a wireless local area network (WLAN, Wireless Local Area Network) or other networks communicate with the display device 200, and control the display device 200 through an application program corresponding to the display device 200. For example, an application program running on a smart device is used to control the display device 200. The application can provide users with various controls through an intuitive user interface (UI, User Interface) on the screen associated with the smart device.

For example, both the mobile terminal 100B and the display device 200 can be installed with software applications, so that the connection and communication between the two can be realized through a network communication protocol, and the purpose of one-to-one control operation and data communication can be realized. For example, the mobile terminal 100B can establish a control command protocol with the display device 200, synchronize the remote control keyboard to the mobile terminal 100B, and control the user interface of the mobile terminal 100B to achieve the function of controlling the display device 200; or the mobile terminal 100B The audio and video content displayed on the screen is transmitted to the display device 200 to realize the synchronous display function.

As shown in Fig. 1, the display device 200 can also communicate with the server 300 through a variety of communication methods. In various embodiments of the present application, the display device 200 may be allowed to perform a wired communication connection or a wireless communication connection with the server 300 through a local area network, a wireless local area network, or other networks. The server 300 may provide various contents and interactions to the display device 200.

Illustratively, the display device 200 transmits and receives information, interacts with an Electronic Program Guide (EPG, Electronic Program Guide), receives software program updates, or accesses a remotely stored digital media library. The server 300 may be a group or multiple groups, and may be one or more types of servers. The server 300 provides other network service content such as video-on-demand and advertising services.

The display device 200 includes a first display 201 and a second display 202, wherein the first display 201 and the second display 202 are independent of each other, and a dual hardware control system is adopted between the first display 201 and the second display 202.

Among them, the first display 201 and the second display 202 can be used to display different display images. For example, the first display 201 can be used for screen display of traditional television programs, and the second display 202 can be used for screen display of auxiliary information such as notification messages and voice assistants.

Optionally, the content displayed on the first display 201 and the content displayed on the second display 202 may be independent of each other without affecting each other. For example, when the first display 201 is playing a TV program, the second display 202 may display information such as time, weather, temperature, and reminder messages that are not related to the TV program.

Optionally, there may also be an association relationship between the content displayed on the first display 201 and the content displayed on the second display 202. For example, when the first display 201 plays the main screen of the video chat, the second display 202 may display information such as the avatar and the chat duration of the user currently accessing the video chat.

Optionally, part or all of the content displayed on the second display 202 can be adjusted to be displayed on the first display 201. For example, the time, weather, temperature, reminder message and other information displayed on the first display 201 may be adjusted to be displayed on the first display 201, and the second display 202 may be used to display other information.

In addition, the first display 201 displays a multi-party interactive screen while displaying a traditional TV program screen, and the multi-party interactive screen does not block the traditional TV program screen. Among them, the present application does not limit the display mode of the traditional TV program screen and the multi-party interactive screen. For example, this application can set the position and size of the traditional TV program screen and the multi-party interactive screen according to the priority of the traditional TV program screen and the multi-party interactive screen.

Taking the priority of traditional TV program screens higher than the priority of multi-party interactive screens as an example, the area of traditional TV program screens is larger than that of multi-party interactive screens, and the multi-party interactive screens can be located on one side of the traditional TV program screen or can be set floating In the corner of the multi-party interactive screen.

The display device 200, on the one hand, may be a liquid crystal display, an OLED (Organic Light Emitting Diode) display, or a projection display device; on the other hand, the display device may be a smart TV or a display system composed of a display and a set-top box. The specific display device type, size, resolution, etc. are not limited, and those skilled in the art can understand that the display device 200 can make some changes in performance and configuration as required.

In addition to providing the broadcast receiving TV function, the display device 200 may additionally provide a smart network TV function that provides a computer support function. Examples include Internet TV, Smart TV, Internet Protocol TV (IPTV), and so on. In some embodiments, the display device may not have the function of broadcasting and receiving TV.

As shown in FIG. 1, a camera may be connected or provided on the display device 200 for presenting a picture captured by the camera on the display interface of the display device or other display devices, so as to realize interactive chats between users. Specifically, the image captured by the camera can be displayed on the display device in full screen, half screen, or in any selectable area.

As an optional connection method, the camera is connected to the rear shell of the display device through a connecting plate, and is fixedly installed on the upper middle of the rear shell of the display device. As an installable method, it can be fixedly installed on the rear shell of the display device. Any position can ensure that the image acquisition area is not blocked by the rear shell. For example, the image acquisition area and the display device have the same orientation.

As another optional connection method, the camera can be connected to the rear shell of the display device through a connecting plate or other conceivable connectors. The connector is equipped with a lifting motor. When the user wants to use the camera or has an application to use The camera is raised above the display device. When the camera is not needed, it can be embedded behind the back shell to protect the camera from damage and protect the privacy of the user.

As an embodiment, the camera used in this application may have 16 million pixels to achieve the purpose of ultra-high-definition display. In actual use, a camera with higher or lower than 16 million pixels can also be used.

When a camera is installed on the display device, the content displayed in different application scenarios of the display device can be merged in a variety of different ways, so as to achieve functions that cannot be achieved by traditional display devices.

Exemplarily, the user may have a video chat with at least one other user while watching a video program. The presentation of the video program can be used as the background screen, and the video chat window is displayed on the background screen. Visually, you can call this function "watch and chat".

Optionally, in the "watching and chatting" scenario, while watching live video or online video, at least one video chat is conducted across terminals.

In another example, the user can video chat with at least one other user while entering the education application for learning. For example, students can realize remote interaction with teachers while learning content in educational applications. Visually, you can call this function "learning and chatting".

In another example, when the user is playing a card game, a video chat is conducted with players entering the game. For example, when a player enters a game application to participate in a game, it can realize remote interaction with other players. Visually, you can call this function "watch and play".

Optionally, the game scene and the video picture are integrated, and the portrait in the video picture is cut out and displayed on the game picture to improve the user experience.

Optionally, in somatosensory games (such as ball games, boxing games, running games, dancing games, etc.), human body postures and movements are acquired through the camera, body detection and tracking, and detection of key points of human skeleton data, and then interact with the game Animations are integrated to realize games such as sports, dance and other scenes.

In another example, the user can interact with at least one other user in video and voice in the K song application. Visually, you can call this function "watch and sing". Optionally, when at least one user enters the application in a chat scene, multiple users can jointly complete the recording of a song.

In another example, the user can turn on the camera locally to obtain pictures and videos, which is vivid, and this function can be called "look in the mirror".

In other examples, more functions can be added or the aforementioned functions can be reduced. This application does not specifically limit the function of the display device.

Fig. 2 exemplarily shows a configuration block diagram of the control device 100 according to an exemplary embodiment. As shown in FIG. 2, the control device 100 includes a controller 110, a communicator 130, a user input/output interface 140, a memory 190, and a power supply 180.

The control device 100 is configured to control the display device 200, and can receive user input operation instructions, and convert the operation instructions into instructions that can be recognized and responded to by the display device 200, acting as an intermediary for the interaction between the user and the display device 200 effect. For example, the user operates the channel plus and minus key on the control device 100, and the display device 200 responds to the channel plus and minus operation.

In some embodiments, the control device 100 may be a smart device. For example, the control device 100 can install various applications for controlling the display device 200 according to user requirements.

In some embodiments, as shown in FIG. 1, the mobile terminal 100B or other smart electronic devices can perform a similar function to the control device 100 after installing an application for controlling the display device 200. For example, the user can install various function keys or virtual buttons of the graphical user interface that can be provided on the mobile terminal 100B or other smart electronic devices by installing applications to realize the function of the physical keys of the control device 100.

The controller 110 includes a processor 112, a RAM 113 and a ROM 114, a communication interface, and a communication bus. The controller 110 is used to control the operation and operation of the control device 100, as well as communication and cooperation between internal components, and external and internal data processing functions.

The communicator 130 realizes the communication of control signals and data signals with the display device 200 under the control of the controller 110. For example, the received user input signal is sent to the display device 200. The communicator 130 may include at least one of communication modules such as a WIFI module 131, a Bluetooth module 132, and an NFC module 133.

The user input/output interface 140, where the input interface includes at least one of input interfaces such as a microphone 141, a touch panel 142, a sensor 143, a button 144, and a camera 145. For example, the user can implement the user instruction input function through voice, touch, gesture, pressing and other actions. The input interface converts the received analog signal into a digital signal and the digital signal into a corresponding instruction signal, and sends it to the display device 200.

The output interface includes an interface for sending the received user instruction to the display device 200. In some embodiments, it may be an infrared interface or a radio frequency interface. For example, in the case of an infrared signal interface, the user input instruction needs to be converted into an infrared control signal according to the infrared control protocol, and sent to the display device 200 via the infrared sending module. For another example: in the case of a radio frequency signal interface, a user input instruction needs to be converted into a digital signal, which is then modulated according to the radio frequency control signal modulation protocol, and then sent to the display device 200 by the radio frequency sending terminal.

In some embodiments, the control device 100 includes at least one of a communicator 130 and an output interface. The control device 100 is equipped with a communicator 130, such as WIFI, Bluetooth, NFC and other modules, which can encode the user input command through the WIFI protocol, or the Bluetooth protocol, or the NFC protocol, and send it to the display device 200.

The memory 190 is used to store various operating programs, data, and applications for driving and controlling the control device 100 under the control of the controller 110. The memory 190 can store various control signal instructions input by the user.

The power supply 180 is used to provide operating power support for the electrical components of the control device 100 under the control of the controller 110. The power supply 180 can be powered by a battery and related control circuits.

FIG. 3 exemplarily shows a schematic diagram of the hardware structure of the hardware system in the display device 200 according to an exemplary embodiment. For ease of description, the display device 200 in FIG. 3 uses a liquid crystal display as an example for illustration.

As shown in FIG. 3, the display device 200 includes: a first panel 11, a first backlight assembly 21, a main board 31, an interactive board 32, a first display driver board 33, a second panel 12, a second backlight assembly 22, and a second display driver The board 34, the power supply board 4, the first rear housing 51, the second rear housing 52 and the base 6.

Among them, the first panel 11 is used to present a picture of the first display 201 to the user. The first backlight assembly 21 is located below the first panel 11, and is usually some optical components for supplying sufficient brightness and uniformly distributed light sources, so that the first panel 11 can display images normally. The first backlight assembly 21 also includes a first back plate (not shown in the figure). The main board 31, the interactive board 32, the first display driving board 33, and the power supply board 4 are arranged on the first back plate, and some convex hull structures are usually stamped on the first back plate. The main board 31, the interactive board 32, the first display driving board 33 and the power supply board 4 are fixed on the convex hull by screws or hooks. The main board 31, the interactive board 32, the first display driver board 3, and the power board 4 may be arranged on one board, or may be arranged on different boards. The first rear shell 51 is covered on the first panel 11 to hide the components of the display device 200 such as the first backlight assembly 21, the main board 31, the interactive board 32, the first display driving board 33, and the power supply board 4, which is beautiful Effect.

Among them, the main function of the first display driver board 33 is: to perform a thousand-level backlight partition control through the PWM signal and the Lcaldimming signal transmitted by the first controller on the main board 31. This part of the control is changed according to the image content, and when communicating with the main board After the handshake is established between the first controller on the main board 31, the VbyOne display signal sent by the first controller on the main board 31 is received, and the VbyOne display signal is converted into an LVDS signal to realize the image display of the first display 201. The base 6 is used to support the display device 200. It is worth noting that only one form of base design is shown in the figure, and those skilled in the art can design different forms of bases according to product requirements.

Among them, the second panel 12 is used to present the screen of the second display 202 to the user. The second backlight assembly 22 is located below the second panel 12 and is usually some optical components for supplying sufficient brightness and uniformly distributed light sources so that the second panel 12 can display images normally. The second backlight assembly 22 also includes a second back plate (not shown in the figure). The second display driving board 34 is disposed on the second backplane, and usually some convex hull structures are stamped on the second backplane. The second display driving board 34 is fixed on the convex hull by screws or hooks. The second display driving board 34 may be arranged on one board, or may be arranged on different boards. The second rear shell 52 is covered on the second panel 12 to hide the components of the display device 200 such as the second backlight assembly 22, the redrive board 35, the second TCON board 34b and the key board 36, which has a beautiful effect .

Optionally, FIG. 3 also includes a key board 35. The key board 35 may be arranged on the first back board or the second back board, which is not limited in this application.

In addition, the display device 200 also includes a sound reproduction device (not shown in the figure), such as an audio component, such as an I2S interface including a power amplifier (AMP) and a speaker (Speaker), for realizing sound reproduction. Usually the audio component can achieve at least two channels of sound output; when the panoramic sound surround effect is to be achieved, multiple audio components need to be set to output multiple channels of sound, which will not be described in detail here.

It should be noted that the display device 200 may also adopt an OLED display screen. In this way, the template included in the display device 200 will be changed accordingly, and no further description will be given here.

Figure 4 shows a schematic diagram of the connection relationship between the power board and the load. As shown in Figure 4, the power board 4 includes an input terminal IN and an output terminal OUT (the figure shows a first output terminal OUT1, a second output terminal OUT2, and a third output terminal. The output terminal OUT3, the fourth output terminal OUT4 and the fifth output terminal OUT5), wherein the input terminal IN is connected to the mains, the output terminal OUT is connected to the load, for example, the first output terminal OUT1 and the light emitting element (such as a light bar or a self Light-emitting devices, etc.) are connected, the second output terminal OUT2 is connected to the audio component, the third output terminal OUT3 is connected to the main board 31, the fourth output terminal OUT4 is connected to the first display driver board 33, and the fifth output terminal OUT5 is connected to the first backlight assembly. 21 is connected. The power board 4 needs to convert the AC mains power into the DC power required by the load, and the DC power usually has different specifications, for example, the audio component requires 18V, the panel requires 12V/18V, and so on.

For ease of description, in the following, one hardware system in the dual hardware system architecture is referred to as the first hardware system or the first controller, and the other hardware system is referred to as the second hardware system or the second controller. The first controller includes various processors of the first controller, various interfaces, and various modules connected to the first controller through various interfaces, and the second controller includes various processors of the second controller , Various interfaces, and various modules connected to the second controller through various interfaces. A relatively independent operating system may be installed in the first controller and the second controller, and the operating system of the first controller and the operating system of the second controller may communicate with each other through a communication protocol, for example: the first controller The framework layer of the operating system of the second controller and the framework layer of the operating system of the second controller can communicate for command and data transmission, so that there are two independent but interrelated subsystems in the display device 200.

The dual hardware system architecture of the present application will be further described below in conjunction with FIG. 5. It should be noted that FIG. 5 is only an exemplary description of the dual hardware system architecture of the present application, and does not represent a limitation to the present application. In practical applications, both hardware systems can contain more or less hardware or interfaces as required.

FIG. 5 exemplarily shows a block diagram of the hardware architecture of the display device 200 shown in FIG. 3. As shown in FIG. 5, the hardware system of the display device 200 includes a first controller 210 and a second controller 310, and modules connected to the first controller 210 or the second controller 310 through various interfaces.

Wherein, the first controller 210 may be arranged on the main board 31 shown in FIG. 3. Optionally, the first controller 210: mainly implements traditional TV functions (for example, an external set-top box can be connected). The second controller 310 may be disposed on the second display driving board 34 shown in FIG. 3. Optionally: the second controller 310 may be used to receive instructions sent by the first controller 210 and control the second display 380 to display corresponding images.

The modules connected to the first controller 210 may include a tuner and demodulator 220, a communicator 230, an external device interface 25, a memory 290, a user input interface 260-3, a video processor 260-1, an audio processor 260-2, The first display 280 (ie, the first display 201 in FIG. 1), the audio output interface 270, and the power supply module 240. In other embodiments, the first controller 210 may also be connected with more or fewer modules.

Among them, the tuner and demodulator 220 is used to perform modulation and demodulation processing such as amplification, mixing, and resonance on the broadcast and television signals received through wired or wireless methods, so as to demodulate the user’s information from multiple wireless or cable broadcast and television signals. Select the audio and video signals carried in the frequency of the TV channel, as well as additional information (such as EPG data signals). According to different television signal broadcasting systems, the signal path of the tuner and demodulator 220 can have many kinds, such as: terrestrial broadcasting, cable broadcasting, satellite broadcasting or Internet broadcasting, etc.; and according to different modulation types, the signal adjustment method can be digitally modulated The method may also be an analog modulation method; and according to different types of received television signals, the tuner demodulator 220 may demodulate analog signals and/or digital signals.

The tuner and demodulator 220 is also used to respond to the TV channel frequency selected by the user and the TV signal carried by the frequency according to the user's selection and control by the first controller 210.

In some other exemplary embodiments, the tuner demodulator 220 may also be in an external device, such as an external set-top box. In this way, the set-top box outputs TV audio and video signals through modulation and demodulation, and inputs them to the display device 200 through the external device interface 250.

The communicator 230 is a component for communicating with external devices or external servers according to various communication protocol types. For example, the communicator 230 may include a WIFI module 231, a Bluetooth communication protocol module 232, a wired Ethernet communication protocol module 233, and an infrared communication protocol module and other network communication protocol modules or near field communication protocol modules (not shown in the figure).

The display device 200 may establish a control signal and a data signal connection with an external control device or content providing device through the communicator 230. For example, the communicator may receive a control signal of the remote controller 100 according to the control of the first controller 210.

The external device interface 250 is a component that provides data transmission between the first controller 210 and other external devices. The external device interface 250 can be connected to external devices such as set-top boxes, game devices, notebook computers, etc. in a wired/wireless manner, and can receive external devices such as video signals (such as moving images), audio signals (such as music), and additional information (such as EPG) and other data.

Among them, the external device interface 250 may include: a high-definition multimedia interface (HDMI) terminal is also called HDMI 251, a composite video blanking synchronization (CVBS) terminal is also called AV 252, and an analog or digital component terminal is also called component 253. , Any one or more of universal serial bus (USB) terminals 254, red, green and blue (RGB) terminals (not shown in the figure), etc. This application does not limit the number and type of external device interfaces.

The first controller 210 controls the work of the display device 200 and responds to user operations by running various software control programs (such as an operating system and/or various application programs) stored on the memory 290.

As shown in FIG. 5, the first controller 210 includes a read-only memory RAM 213, a random access memory ROM 214, a graphics processor 216, a CPU processor 212, a communication interface 218, and a communication bus. Among them, the RAM 213 and the ROM 214, the graphics processor 216, the CPU processor 212, and the communication interface 218 are connected by a bus.

ROM 213, used to store various system startup instructions. For example, when the power-on signal is received, the power of the display device 200 starts to start, and the CPU processor 212 runs the system start-up instruction in the ROM, and copies the operating system stored in the memory 290 to the RAM 214 to start the operating system. After the operating system is started, the CPU processor 212 copies various application programs in the memory 290 to the RAM 214, and then starts to run and start various application programs.

The graphics processor 216 is used to generate various graphics objects, such as icons, operation menus, and user input instructions to display graphics. Including an arithmetic unit, which performs operations by receiving various interactive commands input by the user, and displays various objects according to the display attributes. It also includes a renderer, generates various objects obtained based on the arithmetic unit, and displays the result of the rendering on the first display 280.

The CPU processor 212 is configured to execute operating system and application program instructions stored in the memory 290. And according to receiving various interactive instructions input from the outside, to execute various application programs, data and content, so as to finally display and play various audio and video content.

In some exemplary embodiments, the CPU processor 212 may include multiple processors. The multiple processors may include one main processor and multiple or one sub-processors. The main processor is used to perform some operations of the display device 200 in the pre-power-on mode, and/or to display images in the normal mode. Multiple or one sub-processor, used to perform an operation in the standby mode and other states.

The communication interface 218 may include a first interface 218-1 to an nth interface 218-n. These interfaces may be network interfaces connected to external devices via a network.

The first controller 210 may control operations of the display device 200 related to the first display 280. For example, in response to receiving a user command for selecting a UI object to be displayed on the first display 280, the first controller 210 may perform an operation related to the object selected by the user command.

The first controller 210 may control the operation of the display device 200 related to the second display 380. For example, in response to receiving a user command for selecting a UI object to be displayed on the second display 380, the first controller 210 may perform an operation related to the object selected by the user command.

Wherein, the object may be any one of the selectable objects, such as a hyperlink or an icon. Operations related to the selected object, for example: display operations connected to hyperlink pages, documents, images, etc., or perform operations corresponding to the icon. The user command for selecting the UI object may be a command input through various input devices (for example, a mouse, a keyboard, a touch pad, etc.) connected to the display device 200 or a voice command corresponding to the voice spoken by the user.

The memory 290 includes storing various software modules used to drive and control the display device 200. For example, various software modules stored in the memory 290 include: a basic module, a detection module, a communication module, a display control module, a browser module, and various service modules (not shown in the figure).

Among them, the basic module is a bottom-level software module used for signal communication between various hardware in the display device 200 and sending processing and control signals to the upper-level module. The detection module is a management module used to collect various information from various sensors or user input interfaces, and perform digital-to-analog conversion, analysis and management. The voice recognition module includes a voice analysis module and a voice command database module. The display control module is a module for controlling the first display 280 to display image content, and can be used to play information such as multimedia image content and UI interfaces. The communication module is a module used for control and data communication with external devices. The browser module is a module used to perform data communication between browsing servers. The service module is a module used to provide various services and various applications.

At the same time, the memory 290 is also used to store and receive external data and user data, images of various items in various user interfaces, and visual effect diagrams of focus objects, and the like.

The user input interface 260-3 is used to send a user's input signal to the first controller 210, or to transmit a signal output from the first controller 210 to the user. Exemplarily, the control device (such as a mobile terminal or a remote control) can send input signals input by the user, such as a power switch signal, a channel selection signal, and a volume adjustment signal, to the user input interface, and then forward the input signal to the user input interface 260-3. The first controller 210; or, the control device may receive output signals such as audio, video, or data output from the user input interface 260-3 processed by the first controller 210, and display the received output signal or output the received output signal It is in the form of audio or vibration.

In some embodiments, the user may input a user command on a graphical user interface (GUI) displayed on the first display 280, and the user input interface 260-3 receives the user input command through the graphical user interface (GUI). Alternatively, the user may input a user command by inputting a specific sound or gesture, and the user input interface 260-3 recognizes the sound or gesture through a sensor to receive the user input command.

Exemplarily, when the user input interface 260-3 is used to receive a voice command input by the user (that is, to input a user command through a specific voice), the user input interface 260-3 may specifically be a voice receiver.

The video processor 260-1 is used to receive video signals, and perform video data processing such as decompression, decoding, scaling, noise reduction, frame rate conversion, resolution conversion, and image synthesis according to the standard codec protocol of the input signal. The video signal is directly displayed or played on the first display 280.

Illustratively, the video processor 260-1 includes a demultiplexing module, a video decoding module, an image synthesis module, a frame rate conversion module, a display formatting module, etc. (not shown in the figure).

Among them, the demultiplexing module is used to demultiplex the input audio and video data stream. For example, if MPEG-2 is input, the demultiplexing module will demultiplex into a video signal and an audio signal.

The video decoding module is used to process the demultiplexed video signal, including decoding and scaling.

An image synthesis module, such as an image synthesizer, is used for superimposing and mixing the GUI signal generated by the graphics generator with the zoomed video image according to user input or by itself, so as to generate an image signal for display.

The frame rate conversion module is used to convert the frame rate of the input video, such as converting the frame rate of the input 24Hz, 25Hz, 30Hz, 60Hz video to the frame rate of 60Hz, 120Hz or 240Hz, where the input frame rate can be the same as the source The video stream is related, and the output frame rate can be related to the refresh rate of the display device. The display formatting module is used to change the signal output by the frame rate conversion module into a signal conforming to the display format of a display device, for example, format the signal output by the frame rate conversion module to output RGB data signals.

The first display 280 is used to receive the image signal input from the video processor 260-1, display video content and images, and the menu control interface. The first display 280 includes a display component for presenting images and a driving component for driving the image display . The displayed video content can be from the video in the broadcast signal received by the tuner and demodulator 220, or from the video content input by the communicator or the interface of an external device. The first display 280 simultaneously displays a user manipulation interface UI generated in the display device 200 and used to control the display device 200.

And, depending on the type of the first display 280, it further includes a driving component for driving the display. Alternatively, if the first display 280 is a projection display, it may also include a projection device and a projection screen.

The audio processor 260-2 is used to receive audio signals, and perform decompression and decoding according to the standard codec protocol of the input signal, as well as audio data processing such as noise reduction, digital-to-analog conversion, and amplification processing, so that it can be played in the speaker 272 Audio signal.

The audio output interface 270 is used to receive the audio signal output by the audio processor 260-2 under the control of the first controller 210. The audio output interface may include a speaker 272, or output to an external audio output terminal 274 of a generator of an external device , Such as: external audio terminal or headphone output terminal, etc.

In some other exemplary embodiments, the video processor 260-1 may include one or more chips. The audio processor 260-2 may also include one or more chips.

And, in some other exemplary embodiments, the video processor 260-1 and the audio processor 260-2 may be separate chips, or may be integrated with the first controller 210 in one or more chips.

The power supply module 240 is configured to provide power supply support for the display device 200 with power input from an external power supply under the control of the first controller 210. The power supply module 240 may include a built-in power supply circuit installed inside the display device 200, or may be a power supply installed outside the display device 200, such as a power interface for providing an external power supply in the display device 200.

Similar to the first controller 210, as shown in FIG. 5, the modules connected to the second controller 310 may include a communicator 330, a detector 340, a memory 390, and a second display 380 (ie, the second display in FIG. 1 202). In some embodiments, it may also include a user input interface, a video processor, an audio processor, a display, and an audio output interface (not shown in the figure). In some embodiments, there may also be a power supply module (not shown in the figure) that independently supplies power to the second controller 310.

The communicator 330 is a component for communicating with external devices or external servers according to various communication protocol types. For example, the communicator 330 may include a WIFI module 331, a Bluetooth communication protocol module 332, a wired Ethernet communication protocol module 333, and an infrared communication protocol module and other network communication protocol modules or near field communication protocol modules (not shown in the figure).

The communicator 330 and the communicator 230 of the first controller 210 also interact with each other. For example, the WiFi module 231 in the hardware system of the first controller 210 is used to connect to an external network, and to generate network communication with an external server or the like. The WiFi module 331 in the hardware system of the second controller 310 is used to connect to the WiFi module 231 of the first controller 210 without direct connection with the external network, etc., and the second controller 310 connects to the external network through the first controller 210 . Therefore, for the user, a display device as in the above embodiment may display a WiFi account to the outside.

The detector 340 is a component used by the second controller 310 to collect signals from the external environment or interact with the outside. The detector 340 may include a light receiver 342, a sensor used to collect the intensity of ambient light, which can adaptively display parameter changes by collecting ambient light, etc.; it may also include an image collector 341, such as a camera, a camera, etc., which can be used to collect external The environment scene, as well as the user's attributes or gestures used to interact with the user, can adaptively change the display parameters, and can also recognize the user's gestures to achieve the function of interaction with the user.

The external device interface 350 provides a component for data transmission between the second controller 310 and the first controller 210 or other external devices. The external device interface can be connected to external devices such as set-top boxes, game devices, notebook computers, etc., in a wired/wireless manner.

The video processor 360 is used to process related video signals.

The second controller 310 controls the work of the display device 200 and responds to the user by running various software control programs (such as installed third-party applications, etc.) stored on the memory 390 and interacting with the first controller 210. Operation.

As shown in FIG. 5, the second controller 310 includes a read-only memory ROM 313, a random access memory RAM 314, a graphics processor 316, a CPU processor 312, a communication interface 318, and a communication bus. Among them, the ROM 313 and the RAM 314, the graphics processor 316, the CPU processor 312, and the communication interface 318 are connected by a bus.

ROM 313, used to store various system startup instructions. The CPU processor 312 runs the system startup instruction in the ROM, and copies the operating system stored in the memory 390 to the RAM 314 to start the operating system. After the operating system is started, the CPU processor 312 copies the various application programs in the memory 390 to the RAM 314, and then starts to run and start the various application programs.

The CPU processor 312 is used to execute operating system and application instructions stored in the memory 390, communicate with the first controller 210, transmit and interact with signals, data, instructions, etc., and receive various interactive instructions from external inputs. , To execute a variety of applications, data and content, in order to finally display and play a variety of audio and video content.

There are multiple communication interfaces 318, which may include a first interface 318-1 to an nth interface 318-n. These interfaces may be network interfaces connected to external devices via a network, or network interfaces connected to the first controller 210 via a network.

The second controller 310 may control operations of the display device 200 related to the second display 380. For example, in response to receiving a user command for selecting a UI object to be displayed on the second display 380, the second controller 310 may perform an operation related to the object selected by the user command.

The second controller 310 may control operations of the display device 200 related to the first display 280. For example, in response to receiving a user command for selecting a UI object to be displayed on the first display 280, the first controller 210 may perform an operation related to the object selected by the user command.

The graphics processor 316 is used to generate various graphics objects, such as icons, operation menus, and user input instructions to display graphics. Including an arithmetic unit, which performs operations by receiving various interactive commands input by the user, and displays various objects according to the display attributes. It also includes a renderer, which generates various objects based on the arithmetic unit, and displays the result of the rendering on the second display 380.

Both the graphics processor 316 of the second controller 310 and the graphics processor 216 of the first controller 210 can generate various graphics objects. Differentily, if application 1 is installed in the second controller 310 and application 2 is installed in the first controller 210, when the user is on the interface of application 1, and the user inputs instructions in the application 1, the second controller The graphics processor 316 of 310 generates graphics objects. When the user is on the interface of the application 2 and performs the user-input instruction in the application 2, the graphics processor 216 of the first controller 210 generates a graphic object.

Fig. 6 exemplarily shows a schematic diagram of a functional configuration of a display device according to an exemplary embodiment.

As shown in FIG. 6, the memory 390 of the second controller 310 and the memory 290 of the first controller 210 are respectively used to store an operating system, application programs, content, and user data. Under the control of 210, the system operation of driving the first display 280 and the second display 380 and responding to various operations of the user are executed. The memory 390 and the memory 290 may include volatile and/or nonvolatile memory.

The memory 290 is specifically used to store the operating program that drives the first controller 210 in the display device 200, and store various application programs built in the display device 200, various application programs downloaded by the user from an external device, and application related programs. A variety of graphical user interfaces, and various objects related to the graphical user interface, user data information, and various internal data supporting applications. The memory 290 is used to store system software such as an operating system (OS) kernel, middleware, and applications, as well as to store input video data and audio data, and other user data.

The memory 290 is specifically used to store driver programs and related data such as the video processor 260-1 and the audio processor 260-2, the first display 280, the communicator 230, the tuner and demodulator 220, and the input/output interface.

In some embodiments, the memory 290 may store software and/or programs. The software programs used to represent an operating system (OS) include, for example, kernels, middleware, application programming interfaces (APIs), and/or application programs. Exemplarily, the kernel may control or manage system resources, or functions implemented by other programs (such as the middleware, API, or application program), and the kernel may provide interfaces to allow middleware and APIs, or applications to access the controller , In order to achieve control or management of system resources.

For example, the memory 290 includes a broadcast receiving module 2901, a channel control module 2902, a volume control module 2903, an image control module 2904, a display control module 2905, a first audio control module 2906, an external command recognition module 2907, a communication control module 2908, The light receiving module 2909, the power control module 2910, the operating system 2911, and other application programs 2912, the browser module 2913, and so on. The first controller 210 executes various software programs in the memory 290 such as: broadcast and television signal reception and demodulation function, TV channel selection control function, volume selection control function, image control function, display control function, audio control function, Various functions such as external command recognition function, communication control function, optical signal receiving function, power control function, software control platform supporting various functions, and browser function.

The memory 390 includes storing various software modules used to drive and control the display device 200. For example, various software modules stored in the memory 390 include: a basic module, a detection module, a communication module, a display control module, a browser module, and various service modules (not shown in the figure). Since the functions of the memory 390 and the memory 290 are relatively similar, please refer to the memory 290 for related parts, which will not be repeated here.

For example, the memory 390 includes an image control module 3904, a second audio control module 3906, an external command recognition module 3907, a communication control module 3908, an optical receiving module 3909, an operating system 3911, and other application programs 3912, a browser module 3913, etc. . The first controller 210 executes various software programs in the memory 290 such as: image control function, display control function, audio control function, external command recognition function, communication control function, light signal receiving function, power control function, support Various functions such as software control platform and browser functions.

Differently, the external command recognition module 2907 of the first controller 210 and the external command recognition module 3907 of the second controller 310 can recognize different commands.

Exemplarily, when an image receiving device such as a camera is connected to the second controller 310, the external command recognition module 3907 of the second controller 310 may include a graphic recognition module 2907-1, and the graphic recognition module 3907-1 stores a graphic database. , When the camera receives the graphics instruction from the outside world, it will correspond to the instruction in the graphics database to control the display device. Since the voice receiving device and the remote controller are connected to the first controller 210, the external command recognition module 2907 of the first controller 210 may include a voice recognition module 2907-2, and the voice recognition module 2907-2 stores a voice database. When the voice receiving device or the like receives a voice instruction from the outside world, it corresponds to the instruction in the voice database to control the display device. Similarly, the control device 100 such as a remote controller is connected to the first controller 210, and the key instruction recognition module 2907-3 interacts with the control device 100 in instructions.

FIG. 7 exemplarily shows a configuration block diagram of a software system in the display device 200 according to an exemplary embodiment.

For the first controller 210, as shown in FIG. 7, the operating system 2911 includes operating software for processing various basic system services and for implementing hardware-related tasks, acting as an application program and completing data processing between hardware components Medium.

In some embodiments, part of the operating system kernel may include a series of software to manage the hardware resources of the display device and provide services for other programs or software codes.

In some other embodiments, part of the operating system kernel may include one or more device drivers. The device driver may be a set of software codes in the operating system to help operate or control the device or hardware associated with the display device. The drive may contain code to manipulate video, audio, and/or other multimedia components. Examples include displays, cameras, Flash, WiFi, and audio drivers.

Among them, the accessibility module 2911-1 is used to modify or access the application program to realize the accessibility of the application program and the operability of its display content.

The communication module 2911-2 is used to connect to other peripherals via related communication interfaces and communication networks.

The user interface module 2911-3 is used to provide objects that display user interfaces for access by various applications, and can realize user operability.

The control application 2911-4 is used to control process management, including runtime applications.

The event transmission system 2914 can be implemented in the operating system 2911 or in the application 2912. In some embodiments, it is implemented in the operating system 2911 on the one hand, and implemented in the application program 2912 at the same time, for monitoring various user input events, and responding to the recognition results of various events or sub-events according to various events. And implement one or more sets of pre-defined operation procedures.

Among them, the event monitoring module 2914-1 is used to monitor input events or sub-events of the user input interface.

The event recognition module 2914-2 is used to input the definition of various events to various user input interfaces, recognize various events or sub-events, and transmit them to the processing to execute their corresponding one or more sets of processing procedures .

The event or sub-event refers to the input detected by one or more sensors in the display device 200 and the input of an external control device (such as the control device 100, etc.). Such as: various sub-events of voice input, gesture input sub-events of gesture recognition, and sub-events of remote control button command input of control devices. For example, one or more sub-events in the remote control include multiple forms, including but not limited to one or a combination of pressing up/down/left/right/, the OK key, and pressing the key. And the operations of non-physical keys, such as moving, pressing, and releasing.

The interface layout management module 2913, which directly or indirectly receives various user input events or sub-events monitored by the event transmission system 2914, is used to update the layout of the user interface, including but not limited to the position of each control or sub-control in the interface, and the container The size, position, level, etc. of the interface are related to the various execution operations of the interface layout.

Since the functions of the operating system 3911 of the second controller 310 and the operating system 2911 of the first controller 210 are relatively similar, please refer to the operating system 2911 for related details, which will not be repeated here.

As shown in FIG. 8, the application layer of the display device includes various application programs that can be executed on the display device 200.

The application layer 2912 of the first controller 210 may include, but is not limited to, one or more applications, such as: video-on-demand application, application center, game application, and so on. The application layer 3912 of the second controller 310 may include but is not limited to one or more applications, such as a live TV application, a media center application, and so on. It should be noted that the application programs included in the second controller 310 and the first controller 210 are determined according to the operating system and other designs. This application does not need to determine the application programs included in the second controller 310 and the first controller 210. The application program is specifically defined and divided.

Live TV applications can provide live TV through different sources. For example, a live TV application can provide a TV signal using input from cable TV, wireless broadcasting, satellite services, or other types of live TV services. And, the live TV application can display the video of the live TV signal on the display device 200.

Video-on-demand applications can provide videos from different storage sources. Unlike live TV applications, VOD provides video display from certain storage sources. For example, video on demand can come from the server side of cloud storage, and from the local hard disk storage that contains stored video programs.

Media center applications can provide various multimedia content playback applications. For example, the media center can provide services that are different from live TV or video on demand, and users can access various images or audio through the media center application.

Application center, can provide storage of various applications. The application program can be a game, an application program, or some other application program that is related to a computer system or other device but can be run on a display device. The application center can obtain these applications from different sources, store them in the local storage, and then run on the display device 200.

Since the second controller 310 and the first controller 210 may be respectively installed with independent operating systems, there are two independent but related sub-systems in the display device 200. For example, both the second controller 310 and the first controller 210 can be independently installed with Android (Android) and various APPs, both of which can realize certain functions, and the second controller 310 and the first controller 210 can cooperate to realize certain functions. Item function.

FIG. 9 exemplarily shows a schematic diagram of a user interface in the display device 200 according to an exemplary embodiment. As shown in FIG. 9, the user interface includes a first image display area 2011 and a second image display area 2021. The function realization of the display area 2011 in the first image and the display area 2021 in the second image are basically the same. The following only focuses on the display area 2011 in the first image. For example, the first view display area 2011 includes one or more different items in the layout. And, the user interface also includes a selector indicating that the item is selected, and the position of the selector can be moved through user input to change the selection of different items.

In some embodiments, the first view display area 2011 is a zoomable view display. "Scalable" may indicate that the size or proportion of the first view display area 2011 on the screen is scalable, or that the size or proportion of the items in the view display 201 on the screen is scalable.

"Items" refer to visual objects displayed in the view display area of the user interface of the display device 200 to represent corresponding content, such as icons, thumbnails, and video clips. For example, an item can represent image content or video clips of movies, TV series, audio content of music, applications, or other user access content history information.

In addition, the item may indicate an interface or a set display of interfaces connected to the display device 200 and an external device, or may indicate the name of an external device connected to the display device, or the like. Such as: signal source input interface collection, or high-definition multimedia interface (High Definition Multimedia Interface, HDMI), USB interface, PC terminal interface, etc.

It should be noted that: the view display area can present the content of the video chat project or the content of the application layer project (for example, webpage video, video on demand (VOD) display, application program screen, etc.).

"Selector" is used to indicate that any item has been selected, such as cursor or focus object. Positioning the selection information input according to the icon or menu position touched by the user in the display 200 can enable the movement of the focus object displayed in the display device 200 to select the control item, and select or control one or more items.

The focus object refers to the object that moves between items based on user input. For example, draw a thick line on the edge of the item to realize or identify the position of the focus object. In other embodiments, the focus form is not limited to examples, it can be a tangible or intangible form that can be recognized by the user such as a cursor, such as a 3D deformation of the item, and the border line and size of the text or image of the focused item can also be changed. , Color, transparency and outline and/or font.

The event transmission system 2914 can monitor the user input of each predefined event or sub-event, and directly or indirectly provide the interface layout management module 2913 with the control of the recognized event or sub-event.

The interface layout management module 2913 is used to monitor the state of the user interface (including the position and/or size, change process, etc. of the view partition, item, focus or cursor object, etc.), and according to the event or sub-event, the view display area can be modified The size, position, and level of the layout, and/or, adjust or modify the size or/and position, quantity, type, content, etc. of the layout of various items in the view display area. In some embodiments, modifying and adjusting the layout includes displaying or not displaying each view partition or the content of items in the view partition on the screen.

The user input interface is used to send the user's input signal to the controller, or to transmit the signal output from the controller to the user. Exemplarily, the control device (such as a mobile terminal or a remote control) can send input signals input by the user, such as a power switch signal, a channel selection signal, and a volume adjustment signal, to the user input interface, and then forward the user input interface to the controller; Alternatively, the control device may receive output signals such as audio, video, or data output from the user input interface processed by the controller, and display the received output signal or output the received output signal in the form of audio or vibration.

In some embodiments, the user may input a user command on a graphical user interface (GUI) displayed on the display 200, and the user input interface receives the user input command through the graphical user interface (GUI). Alternatively, the user may input a user command by inputting a specific voice or gesture, and the user input interface recognizes the voice or gesture through the sensor to receive the user input command.

FIG. 10 exemplarily shows a schematic diagram of an initial interface in the display device 200 according to an exemplary embodiment. Exemplarily, as shown in FIG. 10, the display 200 includes a first display 201 and a second display 202. When the user uses the display device 200 to watch the playback content, the first view display area 2011 of the first display 201 displays that the user is The audio and video screen of the watched content, the audio and video screen can be a TV interface, video, picture, or display the webpage that the user is browsing, the application used, etc., displayed in the second view display area 2021 of the second display 202 The initial screen, the initial screen can be set to display one or more combinations of time, date, weather, air quality, product cartoon image as shown in Figure 10, the initial screen can be performed according to user preferences or different scenes Set and modify settings.

Exemplarily, when an audio and video screen is displayed in the first view display area 2011 of the first display 201, the second display 202 may be in a standby state, or no content may be displayed in the second view display area 2021.

On the basis of the above-mentioned embodiments, the playback content that the user is watching is often accompanied by the playback of music, such as background music in film and television works, music sung in singing videos, etc. When the audio output of music is detected, it will pass the first The second view display area 2021 of the second display 202 pushes the acquired music information to the user.

In a specific implementation manner, this solution determines whether the user has turned on the music information push function before displaying the music information in the second view display area 2021 through the second display 202. If the user has set to enable the music information push function, The obtained music information is directly displayed to the user, and if the music information push function is not enabled, the user is prompted to enable the music information push function.

FIG. 11 exemplarily shows a schematic diagram of a music push interface in the display device 200 according to an exemplary embodiment. As shown in FIG. 11, if the music information push function is turned on, a floating frame is displayed through the first view display area 2011 of the first display 201 to remind the user that background music is detected, and to ask the user whether the music information push function needs to be turned on , The floating box can also include function descriptions, such as telling the user to push the music information to the small screen after it is turned on, without affecting the large-screen movie viewing experience. Optionally, the floating frame can be set as a floating frame with transparency to avoid affecting the user's viewing of the content being played on the first display.

Figures 12a to 12d exemplarily show schematic diagrams of the music push interface in the display device 200 according to an exemplary embodiment; exemplarily, as shown in Figure 12, the display 200 includes a first display 201 and a second display 202. The first view display area 2011 of a display 201 displays the audio and video images of the playback content that the user is watching. The playback content can be TV programs, videos, pictures, or display the webpages the user is browsing, the applications used, etc. The second view display area 2021 of the second display 202 displays the music information in the currently played content, for example, displays the lyrics, song title, composer, lyricist, singer, music dynamics, etc. of the background music in the currently played content. Examples Sexually, music-related pictures can also be displayed, such as photos of singers, promotional photos of music, etc. Figures 12a to 12d respectively show one of the preview screens of the playback scene.

In some exemplary embodiments, when the audio and video images of the playing content are displayed in the first view display area 2011 of the first display 201, the audio output interface 270 synchronously outputs the audio of the playing content, and the controller outputs the audio of the playing content to the audio output interface 270. The audio is recorded to obtain at least one piece of audio data. According to the audio characteristics of the audio data, it is recognized whether the content of the audio data is music. When the audio data is music, the controller obtains the music information corresponding to the audio from the server, and then uses the second display The music information is displayed in the second view display area 2021 of 202. Optionally, the controller is generally the first controller 210, but may also be the second controller 310, which is not limited in this solution.

Exemplarily, the controller obtains the music information corresponding to the audio of the playing content from the server by controlling the communicator. Optionally, if the controller is the first controller 210, the communicator is the communicator 230 connected to the first controller 210; if the controller is the second controller 310, the communicator is the communicator with the second controller 310 Connected communicator 330.

As shown in Figure 12a, the controller controls the second display to display music information including lyrics, song titles, composers, lyricists, singers, and music promotional photos; as shown in Figures 12b to 12d, the controller controls the first The music information displayed on the second display is music dynamic effects with different dynamic effects. The music dynamic effects follow the music to display dynamic effects, and truly reflect the music beat, pitch, volume, and other information of the first display. Illustratively, Figure 12b The image of the music animation is composed of columnar graphics, the image of the music animation of Figure 12c is composed of dot graphics, and the image of the music animation of Figure 12d is composed of gradient colors. It should be understood that the music information displayed on the second display in this solution is not limited to the display interface shown in Figure 12a to Figure 12d, and any music information in Figure 12a to Figure 12d can be displayed independently or in any form according to the setting. The form of combination is displayed.

In some exemplary embodiments, the second display scrolls to display music information according to the control of the controller, for example, scrolls to display lyrics, or scrolls to display multiple pieces of music information.

FIG. 13 exemplarily shows a schematic diagram of a music push interface in the display device 200 according to an exemplary embodiment. The controller receives a voice instruction input by the user through a voice receiver. The voice instruction is used to instruct to play the audio data of the music, and then control The device controls the audio output interface 270 to stop outputting the audio data corresponding to the audio and video images played by the first display, and selects to output the audio data corresponding to the music information displayed on the second display, that is, to turn on the mute mode of the first display. The one-view interface displays the mute sign as shown in the figure, and controls the playback of the audio data of the music displayed on the second display.

In some exemplary embodiments, after playing the audio data of the music ends, the controller controls the audio output interface 270 to play the audio corresponding to the audio and video images displayed on the first display, that is, the mute mode of the first display is turned off.

The present application provides a music recommendation method, which is applied to the display device in any of the foregoing embodiments.

FIG. 14 exemplarily shows a schematic flowchart of a music recommendation method according to an exemplary embodiment. As shown in FIG. 14, the music recommendation method includes:

S101: When playing on the first display, periodically record audio corresponding to the audio and video images to obtain at least one piece of audio data.

In a specific application scenario, when the user watches the content played on the first display, the played content is often accompanied by the playing of music, such as background music in film and television works, music sung in singing videos, and so on.

In this step, when the user is watching the audio and video screen played through the first display, the audio corresponding to the audio and video screen, that is, the currently played audio, is periodically recorded to obtain at least one piece of audio data.

Optionally, audio of a preset length of time can be recorded continuously, for example, the preset length of time can be 10s.

In a specific implementation manner, for each piece of recorded audio data, the audio feature of each piece of audio data is separately extracted, and the audio feature includes, but is not limited to, sound peaks and/or sound beats. It should be understood that after a piece of audio data is recorded, audio feature extraction may be performed on the audio data, or after multiple pieces of audio data are recorded, audio feature extraction may be performed on the multiple pieces of audio data in sequence.

S102: Identify whether the content of the audio data is music according to the audio characteristics of each piece of audio data.

In a specific implementation manner, the sound beat in the audio feature can be used to determine whether the audio data is music. For example, if the sound beat is an ordered beat, the audio data is music, and if the sound beat is none Sequence beats (such as human voices or other sounds), the video data is not music; alternatively, the audio data can be input into a pre-trained music recognition model to determine whether the audio data is music.

S103: If the content of the audio data is music, obtain music information of the music.

The music information includes at least one of lyrics, song titles, composers, lyrics writers, singers, and music dynamics. It should be understood that the music can be pure music, and the music information does not include the lyrics and the lyricist.

In a specific implementation manner, if the content of the audio data is music, the controller sends the audio data to the server, and the server obtains the music information by identifying the audio data, and sends the music information to the controller.

In another specific implementation manner, if the content of the audio data is music, the controller obtains the media asset information of the music from the server, and obtains the music information according to the media asset information. Among them, the media asset information includes a playback address and/or a playback location.

Exemplarily, for audio data confirmed as music, the audio data may be uploaded to a server, so that the server obtains media asset information by identifying the audio data, and receives the media asset information sent by the server. Optionally, read and/or download the music information according to the play address of the media asset information, or obtain the music information according to the play address and play position in the media asset information. The music information is related to the music played on the first display. The progress is consistent. Illustratively, the current music playback progress is determined according to the playback position, and the music information corresponding to the playback progress is read from the playback address, or the music information is downloaded from the playback address, and the current music playback progress is determined according to the playback position , Acquire the music information corresponding to the current music playback progress, for example, the lyrics and audio data in the music information need to be displayed according to the playback progress.

S104: Control the second display to display music information.

Further, according to the music information acquired in real time, the controller controls the second display to display the music information without interrupting the content being played on the first display.

In the music recommendation method provided by the embodiments of the present application, at least one piece of audio data is obtained by periodically recording the audio output by the first display while playing on the first display, and the audio data is identified according to the audio characteristics of each piece of audio data. Whether the content is music, if the content of the audio data is music, the music information of the music is obtained and the second display is controlled to display the music information. The music information includes: lyrics, song title, composer, lyricist, singer, music action At least one of the effects, when the user is watching the content played on the first display, the user is recommended to the user the music information of the music in the content being played (for example, background music or the song sung or the music played), so that the user is While watching the playback content without interruption, the music information corresponding to the playback content can be obtained in time through the second display, which improves user experience and increases entertainment.

In a specific implementation, before identifying whether the content of the audio data is music according to the audio characteristics of each piece of audio data, according to the audio characteristics of the audio data, identify whether there is text content in the audio data, for example, according to the audio characteristics The change of the sound peak value identifies whether there is text content. If there is a change in the sound peak value, it means that there is text content, otherwise there is no sound content. Further, if there is text content in the audio data, determine whether the content of the audio data is based on the audio characteristics of the audio data. It is music, otherwise, the process of identifying audio features ends.

In a specific implementation manner, the manner of controlling the second display to display the music information may be scrolling display, for example, scrolling display of lyrics in the music information.

In a specific implementation, the user watches the music information displayed on the second display screen, and when he needs to listen to the music, he can issue a voice command, such as "play music", mute the first display, and make the second display To play the music, specifically, obtain the audio data of the music according to the play address in the media information, and control the mute mode of the first display to be turned on according to the voice command input by the user through the microphone, and control the second display to play the audio of the music data.

Optionally, after playing the audio data of the music played through the second display ends, the controller controls the mute mode of the first display to be turned off.

On the basis of the foregoing embodiment, FIG. 15 exemplarily shows a flow chart of a music recommendation method according to an exemplary embodiment. As shown in FIG. 15, when the media resource is played on the first display, the first display is periodically recorded. Obtain audio data from the audio output by a display, and identify the change of the sound peak in the recorded audio data. If there is a change in the sound peak, for example, through the audio interface of the framework layer, record 10 seconds of audio data at regular intervals, and pass the sound peak. Change the recognition of whether there is a human voice output. If there is, it means that there is text content to proceed to the next step. If not, it means that there is no text content in this time period, and the song recommendation process ends.

If the sound peak changes, the sound beat extraction is performed to determine whether the sound beats are in order, if the sound beats are in order, proceed to the next step, and if the sound beats are out of order, the song recommendation process ends.

If the sound beats are in order, it indicates that the audio data is a song, and the text content in the audio data is recognized. For example, the voice assistant cloud recognizes the song content, and then searches for media information corresponding to the recognized text.

In some embodiments, the voice assistant cloud is a server. If the corresponding media asset information can be searched, the server sends the media asset information to the first controller, and the first controller sends the media asset information to the cloud based on the received media asset information. Acquire music information, and if the corresponding media asset information is not found, the song recommendation process ends. Exemplarily, the lyrics content is searched through the media resource interface of the voice assistant cloud, and information such as the matching playback location and playback address is obtained. The first controller reads and/or downloads the music information through the playback location and playback address.

The first controller converts the music information into characters and sends them to the second controller through the serial port. The second controller inverts the received characters to obtain the music information, and controls the second display to display the music information. Exemplarily, after obtaining the music information, the first controller converts the information containing the action command (action) and the parameter key and value into binary bytecode, and schedules low pressure according to the busyness of the serial port and the network. The byte code composed of 0 and 1 transmitted by the method is sent from the first controller to the second controller system, and the second controller reverses the music information after receiving the music information to obtain the music information. Exemplarily, the second controller sends the action in the information in the form of broadcast, and adds key and value information in the broadcast. After receiving the corresponding action, the second controller reads the corresponding key and value values to obtain Music information and layout information delivered to the main screen.

The second controller controls the second display to display music information in the form of views. At this time, the operating focus of the entire system is always on the first display, and the second display only displays the music information without interrupting the operation of the first display. In this process, when the user inputs a voice command by voice, the second controller downloads the audio data of the song through the playback address for playback, and at the same time transmits the mute command to the first display through the serial port and the network in the same way as the media information. , The user’s operating focus is transferred to the second display. Further, when the music on the second display is played and the user does not issue an instruction by voice, the second controller automatically transmits an unmute instruction to the first display through the serial port, and the user The focus of operation is restored to the first display.

Correspondingly, on the device side, FIG. 16 exemplarily shows a song recommendation device according to an exemplary embodiment, including:

The audio processing module 11 is configured to periodically record the audio corresponding to the audio and video screen when the audio and video screen is played on the first display to obtain at least one piece of audio data;

The recognition module 12 is configured to recognize whether the content of the audio data is music according to the audio characteristics of each piece of audio data;

The obtaining module 13 is configured to obtain music information of the music if the content of the audio data is music; the music information includes at least one of lyrics, song titles, composers, lyrics, and singers.

The display control module 14 is configured to control the second display to display the music information.

A song recommendation device provided by an embodiment of the present application includes an audio processing module 11, an identification module 12, an acquisition module 13, and a display control module 14. By periodically recording the audio output by the first display when playing on the first display, Obtain at least one piece of audio data, and identify whether the content of the audio data is music according to the audio characteristics of each piece of audio data. If the content of the audio data is music, obtain the music information of the music and control the second display to display the music information. The information includes at least one of lyrics, song title, composer, lyricist, singer, and music dynamics, so that when the user watches the content played on the first display, the user is recommended to the music in the content being played (for example, The music information of the background music or the sung or played music) enables the user to obtain the music information corresponding to the playback content in time through the second display without interrupting the viewing of the playback content, which improves the user experience and increases the entertainment .

In a possible design, the recognition module 12 is specifically configured to determine whether the content of the audio data is music according to the sound beat.

In one possible design, the identification module 12 is also used to:

Identifying whether there is text content in the audio data according to the audio characteristics of the audio data;

If there is text content in the audio data, determining whether the content of the audio data is music according to the audio characteristics of the audio data;

Otherwise, the process of recognizing the audio feature is ended.

In a possible design, the acquisition module 13 is specifically used to:

If the content of the audio data is music, acquiring media asset information of the music from the server, where the media asset information includes a playback address and/or a playback location;

Acquire music information from the server according to the media asset information.

In a possible design, the acquiring module 13 is configured to acquire the music information according to the play address and play position in the media asset information; the music information and the progress of the music played on the first display Unanimous.

In a possible design, the display control module 14 is used to control the second display to scroll and display the music information.

In a possible design, the obtaining module 13 is configured to obtain the audio data of the music according to the play address in the media asset information;

The display control module 14 is configured to control the mute mode of the first display to be turned on and control the second display to play the audio data of the music according to the voice instruction input by the user through the microphone.

In a possible design, after the audio data of the music played through the second display ends, the display control module 14 is used to control the mute mode of the first display to be turned off.

In a possible design, the recognition module 12 is used to separately extract the audio features of each piece of audio data.

Fig. 17 also exemplarily shows a song recommendation device according to an exemplary embodiment, including:

The processor 600 is configured to execute computer program instructions in the memory 610, so that the processor executes the song recommendation method in any of the foregoing embodiments.

Wherein, in FIG. 17, the bus architecture may include any number of interconnected buses and bridges. Specifically, one or more processors represented by the processor 600 and various circuits of the memory represented by the memory 610 are linked together. The bus architecture can also link various other circuits such as peripheral devices, voltage regulators, power management circuits, etc., which are all known in the art, and therefore, will not be further described herein. The bus interface provides the interface.

An exemplary embodiment of the present application provides a display terminal, and the display terminal may specifically be a desktop computer, a portable computer, a smart phone, a tablet computer, a personal digital assistant (Personal Digital Assistant, PDA), etc. The display terminal may include a central processing unit (CPU), a memory, an input/output device, etc. The input device may include a keyboard, a mouse, a touch screen, etc., and an output device may include a display device, such as a liquid crystal display (Liquid Crystal Display, LCD), Cathode Ray Tube (CRT), etc.

For different display terminals, in some exemplary implementations, the user interfaces 620 and 820 may be interfaces that can externally and internally connect the required equipment. The connected equipment includes but not limited to a keypad, a display, a speaker, a microphone, a joystick, etc. .

The processor is responsible for managing the bus architecture and general processing, and the memory can store data used by the processor 600 when performing operations.

In some exemplary embodiments, the processor may be a CPU (central embedded device), an ASIC (Application Specific Integrated Circuit, application specific integrated circuit), an FPGA (Field-Programmable Gate Array, field programmable gate array) or a CPLD ( Complex Programmable Logic Device, complex programmable logic device).

The memory may include read only memory (ROM) and random access memory (RAM), and provides the processor with program instructions and data stored in the memory. In the embodiment of the present application, the memory may be used to store the program of any of the methods provided in the exemplary implementation of the present application.

The processor calls the program instructions stored in the memory, and the processor is configured to execute any of the methods provided in the exemplary implementation manners of the present application according to the obtained program instructions.

The exemplary embodiment of the present application provides a computer storage medium for storing computer program instructions used by the device provided in the foregoing embodiment of the present application, which includes a computer program instruction used to execute any method provided in the foregoing embodiment of the present application. program.

The computer storage medium may be any available medium or data storage device that the computer can access, including but not limited to magnetic storage (such as floppy disk, hard disk, magnetic tape, magneto-optical disk (MO), etc.), optical storage (such as CD, DVD, BD, HVD, etc.), and semiconductor memory (such as ROM, EPROM, EEPROM, non-volatile memory (NAND FLASH), solid state drive (SSD)), etc.

Those skilled in the art should understand that the exemplary embodiments of the present application may be provided as a method, a system, or a computer program product. Therefore, this application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, this application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, optical storage, etc.) containing computer-usable program codes.

This application is described with reference to flowcharts and/or block diagrams of methods, devices (systems), and computer program products according to embodiments of this application. It should be understood that each process and/or block in the flowchart and/or block diagram, and the combination of processes and/or blocks in the flowchart and/or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing equipment to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing equipment are generated It is a device that realizes the functions specified in one process or multiple processes in the flowchart and/or one block or multiple blocks in the block diagram.

These computer program instructions can also be stored in a computer-readable memory that can guide a computer or other programmable data processing equipment to work in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture including the instruction device. The device implements the functions specified in one process or multiple processes in the flowchart and/or one block or multiple blocks in the block diagram.

These computer program instructions can also be loaded on a computer or other programmable data processing equipment, so that a series of operation steps are executed on the computer or other programmable equipment to produce computer-implemented processing, so as to execute on the computer or other programmable equipment. The instructions provide steps for implementing the functions specified in one process or multiple processes in the flowchart and/or one block or multiple blocks in the block diagram.

Based on the exemplary embodiments shown in this application, all other embodiments obtained by those of ordinary skill in the art without creative work shall fall within the protection scope of this application. In addition, although the disclosure in this application is introduced in accordance with one or several exemplary examples, it should be understood that various aspects of these disclosures can also be individually constituted as a complete technical solution.

It should be understood that the terms "first", "second", "third", etc. in the specification and claims of this application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific sequence or Priority. It should be understood that the data used in this way can be interchanged under appropriate circumstances, for example, it can be implemented in a sequence other than those given in the illustration or description of the embodiments of the present application.

In addition, the terms "including" and "having" and any variations of them are intended to cover but not exclusively include. For example, a product or device including a series of components need not be limited to those clearly listed, but may include Other components that are not clearly listed or are inherent to these products or equipment.

Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the application, not to limit them; although the application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: It is still possible to modify the technical solutions described in the foregoing embodiments, or equivalently replace some or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the technical solutions of the embodiments of the present application. range.

Claims

A display device is characterized in that it comprises:

A display, a controller, a communicator, and an audio output interface; the display includes a first display and a second display;

The communicator is configured to communicate and interact with the server;

The controller is configured to:

When the first display plays an audio and video picture, recording the audio corresponding to the audio and video picture output through the audio output interface to obtain at least one piece of audio data;

Identifying whether the content of the audio data is music according to the audio characteristics of the audio data;

If the content of the audio data is music, obtain music information from the server;

The second display is controlled to display the music information; the music information includes at least one of lyrics, song titles, composers, lyricists, singers, and music dynamics.
The display device according to claim 1, wherein when the audio feature of the audio data includes a sound beat, the controller is specifically configured to determine whether the content of the audio data is music according to the sound beat .
The display device according to claim 1, wherein the controller is further configured to:

Identifying whether there is text content in the audio data according to the audio characteristics of the audio data;

If there is text content in the audio data, determining whether the content of the audio data is music according to the audio characteristics of the audio data;

Otherwise, the process of recognizing the audio feature is ended.
The display device according to claim 1, wherein the controller executes that if the content of the audio data is music, obtaining music information from the server specifically includes:

If the content of the audio data is music, acquiring media asset information of the music from the server, where the media asset information includes a playback address and/or a playback location;

Acquire music information from the server according to the media asset information.
The display device according to claim 1, wherein the controller is specifically configured to:

Acquire the music information according to the play address and play position in the media asset information; the music information is consistent with the progress of the music played by the first display.
The display device according to claim 1, wherein the display device further comprises a voice receiver;

The controller is specifically used for:

Acquiring audio data of the music according to the play address in the media asset information;

According to the voice instruction input by the user through the voice receiver, the mute mode of the first display is controlled to be turned on, and the second display is controlled to play the audio data of the music.
7. The display device according to claim 6, wherein the controller controls the mute mode of the first display to be turned off after the audio data of the music played through the second display ends.
The display device according to claim 1, wherein the controller is further configured to extract audio characteristics of each piece of audio data separately.
A music recommendation method, characterized in that it is applied to a display device with a dual-display structure, and is characterized in that it includes:

When the first display plays the audio and video images, periodically recording the audio corresponding to the audio and video images to obtain at least one piece of audio data;

According to the audio characteristics of each piece of audio data, identifying whether the content of the audio data is music;

If the content of the audio data is music, acquiring music information of the music;

The second display is controlled to display the music information; the music information includes at least one of lyrics, song titles, composers, lyricists, singers, and music dynamics.
The method according to claim 9, characterized in that, when the audio feature of the audio data includes a sound beat, the method further comprises:

According to the sound beat, it is determined whether the content of the audio data is music.