WO2023000950A1

WO2023000950A1 - Display device and media content recommendation method

Info

Publication number: WO2023000950A1
Application number: PCT/CN2022/103154
Authority: WO
Inventors: 王光强
Original assignee: 聚好看科技股份有限公司
Priority date: 2021-07-23
Filing date: 2022-06-30
Publication date: 2023-01-26
Also published as: CN117643061A

Abstract

Provided are a display device (200), a server (400) and a media content recommendation method. The display device comprises a display (260), which is configured to display a user interface; and a controller (250), which is configured to: acquire an image recognition instruction inputted by a user; in response to the image recognition instruction, send an image recognition request to a server, the request comprising an image to be recognized; receive data that is associated with the image and is fed back by the server, the associated data at least comprising hot search text, and the hot search text being text associated with a recognition result obtained by means of performing image recognition on the image to be recognized; and display a recommendation screen in the user interface according to the associated data, the recommendation screen comprising an option generated at least on the basis of the hot search text in order to request a search related to the image.

Description

Display device and media resource content recommendation method

Cross References to Related Applications

This application claims the priority of Chinese Patent Application No. 202110836063.0 filed on July 23, 2021, and Chinese Patent Application No. 202111120100.4 filed on September 24, 2021, the entire contents of which are incorporated herein by reference.

technical field

The present application relates to the technical field of display devices, in particular to a display device and a method for recommending media content.

Background technique

A display device refers to a terminal device capable of outputting specific display images, which may be terminal devices such as smart TVs, mobile terminals, smart advertising screens, and projectors. Taking smart TV as an example, smart TV is based on Internet application technology, has an open operating system and chip, has an open application platform, can realize two-way human-computer interaction, and integrates multiple functions such as audio-visual, entertainment, and data. Products are used to meet the diverse and individual needs of users.

In actual use, the display device can provide a user with various user interfaces, and the user can control the display device to perform different operations based on various types of interfaces to meet the needs of the user. The user interface provided by the display device may include a user interface for displaying links to multimedia resource resources, such as a media resource recommendation interface, a media resource list interface, and the like. When the user clicks any media resource link in the user interface, the display device may jump to a detailed interface or a playback interface of the selected media resource, so as to play the media resource selected by the user.

The media resource links displayed in the user interface are usually uniformly issued by the server connected to the display device, and can be adjusted individually according to the actual operation rules of the user. For example, the display device can adjust the arrangement order of the links of the media assets in the media asset recommendation interface according to the viewing history of the user, so that the media assets matching the viewing type of the user are arranged in the front area. However, this media asset recommendation method relies too much on the user's viewing habits, which is not conducive to the user's exposure to new media asset types, making the content on the media asset recommendation page too single, and reducing user experience.

In addition, the display device can also support the image recognition function, that is, the user can control the display device to perform image processing on the target image (hereinafter also referred to as "image to be recognized") to recognize the characters, features, text, etc. in the target image information. Then, according to the image recognition result, match the associated media asset content in the media asset database, and display the media asset items through a specific content display window, so that the user can select the media asset items that may be of interest to play.

Since the associated media content is obtained through matching in the media database based on the image recognition results, when the resources in the media database are large and the image recognition results tend to be associated with popular resources, you can A large number of media assets are matched. However, when there are a large number of media asset items, a larger content display window is required for display, which not only blocks the user interface, but also hinders users from selecting the media asset items they are interested in, reducing user experience.

Contents of the invention

The present application provides a display device and a method for recommending media asset content to solve the problems of single media asset content and too many matching results of traditional image recognition functions in the user interface of traditional display devices.

In one aspect, an embodiment of the present application provides a display device, including:

a display configured to display an image;

a user input interface configured to receive user instructions;

remote control,

a controller coupled to the display, user input interface and configured to:

Obtaining a picture recognition instruction input by a user while the display is displaying an image;

In response to the image recognition instruction, send an image recognition request to the server, the request includes the image to be recognized; and

receiving data associated with the image fed back by the server, the associated data at least including hot search text, where the hot search text is text associated with a recognition result obtained by performing image recognition on the image to be recognized ;

A recommendation screen is displayed on the display according to the associated data, the recommendation screen including an option generated based at least on the trending text to request a search related to the image.

On the other hand, an embodiment of the present application provides a method for recommending media content on a display device, including:

Obtain the image recognition instruction input by the user through the display device,

And in response to the image recognition instruction, send an image recognition request to the server, the request includes the image to be recognized;

receiving data associated with the image obtained by performing image recognition on the server, the associated data includes at least hot search text, and the hot search text is the same as the recognition result obtained by performing image recognition on the image to be recognized associated text;

A recommendation screen is displayed in a user interface based on the associated data, the recommendation screen including an option generated based at least on the trending text to request a search related to the image.

Description of drawings

FIG. 1 is a usage scenario of a display device in an embodiment of the present application;

Fig. 2 is a block diagram of the hardware configuration of the control device in the embodiment of the present application;

FIG. 3 is a hardware configuration diagram of a display device in an embodiment of the present application;

FIG. 4 is a software configuration diagram of a display device in an embodiment of the present application;

Figure 5A is a schematic diagram of the home page in the embodiment of the present application;

FIG. 5B is a schematic diagram of media asset recommendation in the embodiment of the present application;

FIG. 6 is a schematic diagram of a recommended screen in an embodiment of the present application;

FIG. 7 is a schematic structural diagram of a display device in an embodiment of the present application;

FIG. 8 is a time series relationship diagram of a method for recommending media asset content in an embodiment of the present application;

FIG. 9A is a schematic diagram of hot search text options in the embodiment of the present application;

FIG. 9B is a schematic diagram of search results based on the current media asset platform in the embodiment of the present application;

FIG. 9C is a schematic diagram of search results in the entire network in the embodiment of the present application;

FIG. 9D is a schematic diagram of the search input interface in the embodiment of the present application;

Fig. 10 is a schematic diagram of the image recognition tab in the embodiment of the present application;

Fig. 11A is a schematic diagram of a search tab in the embodiment of the present application;

FIG. 11B is a schematic diagram of another search tab in the embodiment of the present application;

Figure 12 is a schematic diagram of the search operation process in the embodiment of the present application;

Figure 13 is a schematic diagram of the selection operation process in the embodiment of the present application;

FIG. 14 is a schematic flow diagram of cutting a screenshot in an embodiment of the present application;

FIG. 15 is a sequence diagram of the operation flow of the image recognition application in the embodiment of the present application;

FIG. 16 is a schematic flowchart of a method for displaying a recommendation window in an embodiment of the present application;

Fig. 17 is a schematic diagram of the mode switching window in the embodiment of the present application;

Figure 18 is a schematic diagram of the children's mode interface in the embodiment of the present application;

FIG. 19 is a schematic flow diagram of the server generating associated data in the embodiment of the present application;

FIG. 20A is a schematic diagram of a recommendation window in the video mode in the embodiment of the present application;

FIG. 20B is a schematic diagram of the recommendation window in the children's mode in the embodiment of the present application;

FIG. 20C is a schematic diagram of the recommendation window in the game mode in the embodiment of the present application;

Figure 21 is a schematic diagram of the search result interface in the embodiment of the present application;

Fig. 22 is a schematic diagram of the process of generating a target image in the embodiment of the present application;

FIG. 23 is a schematic diagram of the server structure in the embodiment of the present application;

FIG. 24 is a schematic diagram of the process of extracting associated data by the server in the embodiment of the present application; and

Fig. 25 is a sequence diagram of the display process of the recommendation window in the embodiment of the present application.

detailed description

The embodiments will be described in detail hereinafter, examples of which are illustrated in the accompanying drawings. When the following description refers to the accompanying drawings, the same numerals in different drawings refer to the same or similar elements unless otherwise indicated. The implementations described in the following examples do not represent all implementations consistent with this application. These are merely examples of systems and methods consistent with aspects of the present application as recited in the claims.

Fig. 1 is a schematic diagram of a usage scenario of a display device according to an embodiment. As shown in FIG. 1 , the display device 200 also performs data communication with the server 400 , and the user can operate the display device 200 through the smart device 300 or the control device 100 .

In some embodiments, the control device 100 may be a remote controller, and the communication between the remote controller and the display device includes at least one of infrared protocol communication, Bluetooth protocol communication, and other short-distance communication methods, and the display device is controlled wirelessly or wiredly. Device 200. The user can control the display device 200 by inputting user instructions through at least one of buttons on the remote control, voice input, and control panel input.

In some embodiments, the display device 200 also performs data communication with the server 400 . The display device 200 may be allowed to communicate via a local area network (LAN), a wireless local area network (WLAN), and other networks. The server 400 may provide various contents and interactions to the display device 200 .

FIG. 2 is a configuration block diagram of the control device 100 according to some embodiments. As shown in FIG. 2 , the control device 100 includes a controller 110 , a communication interface 130 , a user input/output interface 140 , a memory, and a power supply. The control device 100 can receive the user's input operation instruction, and convert the operation instruction into an instruction that the display device 200 can recognize and respond to, and play an intermediary role between the user and the display device 200 .

FIG. 3 is a block diagram of a hardware configuration of a display device 200 according to some embodiments.

In some embodiments, the display device 200 includes a tuner and demodulator 210, a communicator 220, a detector 230, an external device interface 240, a controller 250, a display 260, an audio output interface 270, a memory, a power supply, and a user interface 280. at least one of .

In some embodiments, the communicator 220 is a component for communicating with external devices or servers according to various communication protocol types. For example, the communicator may include at least one of a Wifi module, a Bluetooth module, a wired Ethernet module and other network communication protocol chips or near field communication protocol chips, and an infrared receiver. The display device 200 can establish transmission and reception of control signals and data signals with the control device 100 or the server 400 through the communicator 220 .

In some embodiments, the external device interface 240 may include but not limited to the following: high-definition multimedia interface (HDMI), analog or data high-definition component input interface (component), composite video input interface (CVBS), USB input interface ( Any one or more interfaces such as USB), RGB port, etc.

In some embodiments, the controller 250 and the tuner-demodulator 210 may be located in different split devices, that is, the tuner-demodulator 210 may also be located in an external device of the main device where the controller 250 is located, such as an external set-top box Wait.

In some embodiments, the controller 250 controls the operation of the display device and responds to user operations through various software control programs stored in the memory. The controller 250 controls the overall operations of the display device 200 . For example, in response to receiving a user command for selecting a UI object to be displayed on the display 260, the controller 250 may perform an operation related to the object selected by the user command.

In some embodiments, the user can input user commands through a graphical user interface (GUI) displayed on the display 260, and the user input interface receives user input commands through the graphical user interface (GUI). Alternatively, the user may input a user command by inputting a specific sound or gesture, and the user input interface recognizes the sound or gesture through a sensor to receive the user input command.

Referring to Fig. 4, in some embodiments, the system is divided into four layers, from top to bottom are respectively the application (Applications) layer (abbreviated as "application layer"), application framework (Application Framework) layer (abbreviated as "framework layer") "), Android runtime (Android runtime) and system library layer (referred to as "system runtime layer"), and the kernel layer.

In some embodiments, there is at least one application program running in the application program layer, and these application programs can be window (Window) program, system setting program or clock program etc. that come with the operating system; they can also be developed by third-party developers. s application. During specific implementation, the application program packages in the application program layer are not limited to the above examples.

The framework layer provides an application programming interface (application programming interface, API) and a programming framework for applications in the application layer. The application framework layer includes some predefined functions. The application framework layer is equivalent to a processing center, which decides to make the applications in the application layer take actions. Through the API interface, the application program can access the resources in the system and obtain the services of the system during execution.

As shown in Figure 4, in the embodiment of the present application, the application framework layer includes managers (Managers), content providers (Content Provider) etc., wherein the manager includes at least one of the following modules: activity manager (Activity Manager) Interact with all activities running in the system; the Location Manager is used to provide system services or applications with access to the system location service; the Package Manager is used to retrieve the information currently installed on the device Various information related to the application package; Notification Manager (Notification Manager) is used to control the display and clearing of notification messages; Window Manager (Window Manager) is used to manage icons, windows, toolbars, wallpapers on the user interface and desktop widgets.

In some embodiments, the kernel layer is a layer between hardware and software. As shown in Figure 4, the kernel layer at least includes at least one of the following drivers: audio driver, display driver, bluetooth driver, camera driver, WIFI driver, USB driver, HDMI driver, sensor driver (such as fingerprint sensor, temperature sensor, pressure sensors, etc.), and power drives, etc.

Based on the above display device 200 , the user can control the display device 200 to display various user interfaces during the process of using the display device 200 . In the user interface displayed by the display device 200, recommended resources may be displayed according to the viewing habits of the user.

In some embodiments, as shown in FIG. 5A , the display device can display homepages of different channels. Through operation configuration, different media resources can be displayed on the homepage column for media resource recommendation. Users can select the column position to expand or view the next page. Users can also search for media resources by entering the search page through the search page set on the display page. Users can also directly obtain feedback results through voice search.

In some embodiments, for example, as shown in FIG. 5B , on the media resource details page of watching movie A, the user can display the information related to movie A in the "guess you like" area according to the type, content, author and other information of movie A. Movie B of the same type, movie C with similar content to movie A, and movie D with the same author as movie A are available for the user to choose.

In order to realize the above content recommendation process, the display device 200 may have a built-in control program for recording the user's viewing habits. As the user watches, the display device 200 can record the content watched by the user by starting the control program to form historical record information. When the recommended content needs to be displayed, the display device 200 can use the historical record information to perform matching in the resource library, so as to obtain the recommended content that matches the content watched by the user.

However, the method of determining recommended content based on historical records can only obtain part of the content associated with the media assets of the historical records, which is not conducive to the user's understanding of new content. For example, when a user sees a poster or a screenshot of a certain movie, he or she may be interested in the movie or actors, or want to learn more about the movie. To this end, in some embodiments, the display device 200 can perform image recognition processing on the image through image recognition technology, so as to recognize relevant information such as movie titles and actor names from the image, thereby providing a more convenient search experience for users.

For example, as shown in FIG. 6 , the user may control the display device 200 to perform an image recognition operation, and in this operation, a screenshot operation may be performed to obtain a screenshot image. An image recognition algorithm may be executed on the screenshot image, so as to identify the person target in the screenshot image according to the pixel point distribution law in the screenshot image. Then, according to the character type to which the character target belongs, the recommended media resource content suitable for the character type is determined. For example, for a character of the film and television star type, different film and television works of the same character can be determined; for a task of the sports star type, the sports-related video resources can be determined. After the recommended content is determined, the display device 200 may display the corresponding recommended media asset item at a specific position on the user interface for the user to select.

In some embodiments, when the screenshot recognition page is displayed, the position of movie B and/or movie C in FIG. 6 can be used to display the person recognition result.

In some embodiments, when the number of people identified in the person recognition result is less than the number of display slots, the remaining display slots may be used to display recommended media asset items.

In some embodiments, the display content is displayed on the video layer, the screenshot recognition result is displayed on the OSD layer above the video layer, and the person recognition result and recommended media asset items are all displayed through the display position control on the OSD layer.

In some embodiments, when displaying the screenshot recognition result, the focus is set to any control position on the OSD layer, and at this time, the control of the video layer cannot obtain the focus, that is, the focus will not move to the video when the OSD layer is displayed. The position of the control on the layer. After the user chooses to cancel the display of the OSD layer, the control of the video layer starts to get the normal focus.

In some embodiments, the recommended content determined through image recognition can enable the user to obtain recommended content related to the image content without viewing specific media content. However, the method of recommending media asset items only based on image recognition results is far from meeting the needs of users for richer content. For example, some users use the image recognition method not to obtain other works of the characters in the image, but to understand the characters in the image.

In order to meet the needs of more users for recommended content, in some embodiments of the present application, a display device 200 is provided. The display device 200 includes: a display 260 , a communicator 220 and a controller 250 . Wherein, the display 260 can be used to display images. The communicator 220 may connect to the server 400 through remote communication such as a network connection, so as to realize data interaction with the server 400 . The user input interface is configured to receive user instructions. The controller 250 may be configured to execute an image recognition-based content recommendation method for displaying various types of recommended content in all or part of the image, and it may be configured to: acquire user An input map recognition instruction; in response to the map recognition instruction, send an image recognition request to the server, the request includes an image to be recognized; and receive data associated with the image fed back by the server, the associated data At least including hot search text, the hot search text is text associated with the recognition result obtained by performing image recognition on the image to be recognized; displaying a recommendation screen on the display according to the associated data, the recommendation A screen includes an option generated based at least on the trending text to request a search related to the image.

As shown in Figure 7, it specifically includes the following steps:

Obtain the image recognition instruction input by the user. During the running process, the display device 200 can monitor the user's interactive actions in real time. Different interactive actions can be used to control the display device 200 to implement different functions. Wherein, when the user inputs an interactive action for controlling the display device 200 to perform image recognition, it means that the user inputs an image recognition instruction. According to different specific interaction modes of the display device 200, the user may input a map recognition instruction through different interaction modes.

For some display devices 200 , the user may perform an interactive action through the control device 100 provided with the display device 200 to input a map recognition instruction. For example, a screenshot key may be provided on the control device 100, and when the user presses the screenshot key, the display device 200 may be triggered to perform a screenshot operation to obtain a screenshot image.

The display device 200 can also set screenshot rules, that is, in different user interfaces, image processing-related operations are automatically executed after taking screenshots to obtain images. For example, if the user presses the screenshot key when the display device 200 is displaying the playback interface, after the screenshot is taken to obtain a specific image, the image obtained by the screenshot is saved. When the user presses the screenshot key when the display device 200 displays the media asset list interface, the image can be automatically identified after the specific image is captured by screenshot. That is, the image recognition command can be input by the user by pressing a button on the control device 100 when the display device 200 displays the media asset list interface.

The display device 200 may also display controls for the image recognition operation in the user interface for the user to select. For example, the user may call out the status bar of the user interface through the control device 100, and the status bar may include a "screenshot/picture recognition" option. The user controls the movement of the focus cursor and selects the "screenshot/picture recognition" option through the direction keys and the confirmation key on the control device 100, and triggers the display device 200 to perform screen capture and picture recognition operations. The image recognition command can be input based on the option control in the user interface.

For part of the display device 200, an intelligent voice system may be built in, and the user may use the intelligent voice system to input voice input image recognition instructions. For example, the user can input voice content such as "recognize the content in the picture" and "recognize the person in the current picture", then the intelligent voice system can recognize the voice content and convert it into a specific control command to drive the display device 200 to perform the recognition. Figure operations.

It should be noted that the user can not only input an image recognition instruction for a specific screen in the user interface, but also input an image recognition instruction for a specific image file. For example, when the display device 200 opens a certain picture file, the user can control the display device 200 to perform a picture recognition operation on the opened picture file by long pressing the screenshot button on the control device 100 .

After receiving the image recognition instruction input by the user, the display device 200 may send the image to be recognized to the server in response to the image recognition instruction. Wherein, the image to be recognized may have different image forms according to different targets of the image recognition operation. For example, when the image recognition instruction received by the display device 200 is triggered by a screen capture operation, the image to be recognized is a screen capture image obtained by the screen capture operation. When the image recognition instruction received by the display device 200 is for a picture file that has been opened by the display device 200, the image to be recognized is the picture file.

After receiving the image recognition instruction, the display device 200 may extract the image to be recognized indicated by the image recognition instruction, and transcode and compress the extracted image file to form a data packet. Then according to the network connection relationship between the display device 200 and the server 400, the data packet is sent to the server 400 with a specific transmission protocol to perform an image recognition request.

As shown in FIG. 8 , after receiving the image recognition request, the server 400 obtains the data packet, and may decompress and decode the data packet, thereby parsing the image to be recognized in the data packet. Then the server 400 performs image recognition processing on the image to be recognized according to the image recognition algorithm, so as to identify a specific target from the image to be recognized. For example, the server 400 may have a built-in recognition model for recognizing human objects in images, and after the display device 200 sends the image to be recognized to the server 400, the server 400 may input the image to be recognized sent by the display device 200 into the recognition model. Then, after the calculation of the recognition model, it is possible to output whether the image to be recognized contains a person target, and the person information conforming to the characteristics of the person target. Wherein, the content output by the recognition model may specifically be the classification probability that the person object belongs to a certain person, and finally the person information of the characteristics of the person object is determined as the label type with the highest classification probability.

Recognition of different targets in the image to be recognized can be realized by presetting multiple types of recognition models in the server 400 . For example, in addition to the character target recognition model, the server 400 can also preset a character recognition model for recognizing specific text content contained in an image, a scene recognition model for recognizing scene content contained in an image, and a scene recognition model for recognizing image The product identification model of the product content in the product, etc. The specific input recognition model of the image to be recognized can be set through the display device 200 . That is, the more types of information that need to be recognized in the image, the more types of recognition models the server 400 will input the image to be recognized into. Obviously, when the server 400 inputs more types of recognition models for images to be recognized, the time required to obtain the recognition results will be longer. Therefore, in order to balance the types of recognition results and the recognition time, in some embodiments, the server 400 can A feature target recognition model and a character recognition model are respectively set up, which are respectively used to recognize and obtain graphic information in pictures.

After the server 400 performs recognition processing on the image to be recognized, associated data extracted based on the recognition result may be fed back to the display device 200 . That is, the display device 200 may receive associated data fed back by the server 400 . The associated data includes hot search text and/or recommended links. Wherein, the hot search text may be the text associated with the recognition result obtained by performing image recognition on the image to be recognized; the recommended link may be the media resource address and/or webpage address associated with the recognition result.

In some embodiments, when the server 400 performs character matching and recognition on the images received from the display device 200 , it also performs character recognition.

In some embodiments, during the image recognition process, the server 400 will obtain the relative position of the recognition object in the image, and then, after obtaining the result of the person recognition and the character recognition result, can recognize the person whose positional relationship meets the preset condition The results are associated with the text recognition results. Exemplarily, the positional relationship may be that the distance is less than a preset threshold, or that the object corresponding to the character recognition result is below the image corresponding to the person recognition result, or a combination of both.

In some embodiments, the server 400 may maintain a media asset item database, and the media asset item database may store a hot search term base based on network or local search engine statistics, that is, a set of words with high search frequency by users. All media asset items in the current platform may also be stored in the media asset item database. After the word "variety show" is recognized in the image through the image recognition operation, the server 400 can match the hot search text related to "variety show" in the hot search word database. Similarly, after the character "Zhang San" is identified in the image through the image recognition operation, the server 400 can match the film and television content related to "Zhang San" in the media asset project database, and extract the corresponding media asset address. Therefore, the server 400 can combine the hot search text related to "Variety Show" and the media resource address related to "Zhang San" into associated data, and feed it back to the display device 200 .

In some embodiments, the server 400 may recommend hot search texts only based on the recognized text results, and at the same time recommend media resources only based on the character recognition results. In some embodiments, it is also possible to recommend popular texts and/or media resources based on the recognized text results and character recognition results.

In some embodiments, there may be many text results contained in the screenshot. At this time, the server 400 can recommend hot search texts only based on the text recognition results that meet the preset positional relationship with the person recognition results, and ignore the texts that do not meet the positional relationship. Recognition results.

In some embodiments, when no text result is identified, the server 400 may also recommend hot search texts according to the hot searches of the current search function.

After receiving the associated data fed back by the server 400, the display device 200 may display a recommendation screen in the user interface according to the associated data. In the embodiment of the present application, the recommendation screen may be rendered for display. The recommendation screen includes options generated based on hot search text and/or recommended links. The recommendation screen may be a new interface that the display device 200 jumps to, or may be a floating window displayed in a specific area of the original interface.

For example, as shown in FIG. 9A , the recommendation screen may be located at the bottom area of the current user interface. In the long strip window of the recommendation screen, there may be a search area in the middle and recommended item areas on both sides. The display device 200 may display a search box and hot search text options below the search box in the search area for the user to select. When the user selects any hot search text option, the display device 200 may automatically perform a search operation for the hot search text.

After performing the search operation, the display device 200 may display a search result interface on the OSD layer of the current user interface. As shown in FIG. 9B , when performing a search, the display device 200 may search within the current media asset platform, so as to search the current media asset platform for media asset items associated with the selected hot search text. For example, after the user selects the hot search text "Zhang San", the display device 200 can send the search term containing the text "Zhang San" to the server 400, so that the server 400 can match the media related to "Zhang San" on the current media platform. and then feed it back to the display device 200 to render or present a media resource display window on the OSD layer of the display device 200 to display the associated items obtained by matching.

Similarly, when the display device 200 performs a search, it may also perform a network-wide search through a network search engine. For example, after the user selects any hot search text, the display device 200 can search the entire network through the search website by visiting a specified search website and using the selected hot search text as a keyword, and display the search results on the display device 200. The OSD layer displays the search result webpage, as shown in FIG. 9C.

In some embodiments, during the search process, the display device 200 may successively display the search results in the above two manners. That is, after the user selects any hot search text, two windows can be rendered or presented on the OSD layer, which are used to display the search results in the current media platform and the search results of the whole network respectively.

In some embodiments, during the search process, the display device 200 may first search in the current media asset platform and display the search results. When the search results in the current media asset platform are displayed for a certain period of time, or after the user enters an instruction to search the entire network, the selected hot search text is used as the search term to search the entire network, and the search results of the entire network are displayed.

Apparently, if the user selects the search box without selecting the hot search text, the display device 200 can start the normal search function. That is, as shown in FIG. 9D , after the user selects the search box, input controls, such as a keyboard, a tablet, and a voice assistant, may be displayed in the OSD layer. Users can input the text content they want to search based on the displayed input control.

The display device 200 can also display recommended links in the recommended item area. When the user selects any recommended link, the display device 200 can be controlled to jump to the specified media asset details interface or a specific web page interface to realize playback or access operations and satisfy the user's needs. specific needs.

It should be noted that during the data interaction process between the display device 200 and the server 400 , affected by network transmission delay, data encryption and decryption, etc., a part of the time will be consumed for data transmission. That is, when the display device 200 sends the image to be recognized to the server 400, it will consume a part of the time, and when the server 400 feeds back the associated data to the display device 200, it will also consume a part of the time. Combined with the time consumed by the image recognition process, it will lead to the entire recommendation process. for too long. Therefore, in order to improve the efficiency of content recommendation, in some embodiments, a recognition model with a high frequency of use can also be preset in the display device 200, so that when the user inputs an image recognition instruction, the recognition model built in the display device 200 can be used first. Image recognition is performed first, so as to obtain some types of image recognition results, so that the display device 200 can first render or present a recommended picture for the user to choose. When the user further inputs an indication to obtain other types of recommended content based on the previously rendered or presented recommendation screen, the display device 200 then sends the image to be recognized to the server 400 and receives associated data fed back by the server 400 .

It can be seen that, in the above embodiment, by presetting the recognition models in the display device 200 and the server 400 respectively, the efficiency of content recommendation can be improved, and the diversity of content recommendation types can be taken into account. In addition, since the image recognition operation can be completed by the display device 200 and the server 400 respectively, the data processing amount of the display device 200 or the server 400 can also be reduced, and excessive occupation of computing resources in the image recognition process can be avoided.

In some embodiments, the recommendation screen rendered by the display device 200 may include two tabs (tab pages), one of which is used to display image recognition results; the other tab is used to display searched and recommended results. For example, as shown in Fig. 10, Fig. 11A, and Fig. 11B, the recommendation screen may include a "××Picture Recognition" tab and a "Search" tab. Recognize the thumbnail of the image, and display the recognized results from the image, such as people, text, products, etc., in the areas on both sides. Therefore, after the user inputs the image recognition command, the display device 200 can first input the image to be recognized into the product recognition model and the person recognition model respectively, so as to obtain product recommendation content and character recommendation content, and display them in the "×× picture recognition" tab. Render or present as map recognition result options for users to view and select.

As shown in FIG. 11A and FIG. 11B , the user can also control the display device 200 to switch to this tab by selecting the "Search" option. In the "Search" tab, a search box can be displayed in the central area, and multiple hot search texts can be displayed below the search box. The media resource links and/or web page links recommended according to the image recognition results are displayed in the areas on both sides. For this reason, when the user clicks on the "Search" tab, the display device 200 can send the image to be recognized to the server 400, and further perform image recognition operations on the image to be recognized through the recognition model in the server 400, so as to obtain information including hot search text, Data associated with media links and/or web links.

The recommendation screen can have a specific layout. For example, as shown in FIG. 10 , the product identification result may be displayed on the left area of the recommendation screen, and the recognized image and name of the person may be displayed on the right area. Different tabs can maintain the same layout, as shown in Figure 11A, or change the layout after switching tabs, as shown in Figure 11B, the layout can be rearranged in the left display area to display new content. In the foregoing embodiments, the recommendation screen presented by the display device 200 may be displayed through an OSD layer located above the video layer. That is, in some embodiments, after the user inputs the image recognition instruction, the display device 200 may respond to the image recognition instruction and call the OSD layer to display or render the interface. During this process, the display device 200 may call the display template of the recommended screen, and obtain the image recognition result output by the image recognition model and the associated data fed back by the server 400 . Finally, a specific recommended picture is rendered or presented according to the display template and the image recognition result, so as to be displayed on the OSD layer.

Moreover, different tabs in the recommendation screen may also be displayed through different OSD layers. For example, when the recommendation screen includes two tabs of "XX Image Recognition" and "Search", two OSD layers can be called to render or present "XX Image Recognition" respectively when rendering or presenting the recommendation screen Image recognition result screen in the tab and recommended items in the "Search" tab. Correspondingly, when switching tabs, the user can display the content of each tab by controlling the display/hide state of the corresponding OSD layer.

For the content displayed in multiple tabs, the display device 200 can issue the content in advance through the server 400, that is, when the user inputs a picture recognition command, the server 400 directly completes the picture recognition operation and content recommendation, and combines the picture recognition result and the recommended item together. And send it to the display device 200, so as to perform rendering or presentation operations on different OSD layers. The content displayed in each tab can also be obtained after the user selects the tab label control, that is, after the display device 200 detects that the user clicks the "search" option, it obtains the recommended item from the server 400 for rendering Or render the Search tab.

By displaying the recommended screen through the OSD layer, the display process of the recommended screen will not affect the display process of the original user interface of the display device 200, thereby maintaining the normal display of the video layer screen and improving user experience. Moreover, displaying the recommendation screen through the OSD layer also facilitates the user to return to the original user interface through simple interactive operations. For example, when the search operation does not need to be performed, the display device 200 can be controlled to cancel displaying the recommendation screen and continue to display the original user interface by pressing the "exit" button on the control device 100 .

After the display device 200 displays the recommended screen, the user can also perform further interactive operations based on the displayed recommended screen, as shown in FIG. Obtain the network resource content corresponding to the hot search text. In order to realize fast search interactive operation, the display device 200 may first obtain a search instruction input by the user based on the hot search text option in the recommendation screen. For example, the user can use the direction keys on the control device 100 to control the focus cursor to move to the hot search text of interest. When the user controls the focus cursor to select any hot search text by pressing the OK key, the display device 200 can be controlled to search using the selected hot search text as a search term, that is, the display device 200 obtains the search command input by the user.

After the user inputs a search instruction, the display device 200 may send a search request to the server 400 in response to the search instruction. Wherein, the search request includes the hot search text selected in the search instruction. The server 400 can perform a search operation after starting a network resource search engine or a local search engine, that is, the server 400 can search and select the media resource associated with the hot search text in the resource item data, and feed back the search result to the display device 200 .

After the display device 200 receives the associated media asset link fed back by the server 400 for the search request, it may update the options in the recommendation screen according to the associated media asset link. For example, when multiple characters can be identified in the picture to be recognized through the image recognition operation, the server 400 extracts the name of the person who has been searched more frequently among the multiple characters as the hot search text, and feeds back the name of the person who has been searched more frequently to The device 200 is displayed, and at the same time, a resource link of a representative work of each character is fed back. The display device 200 then renders or presents a recommendation screen according to the received character name and representative works. That is, the names of multiple characters are displayed in the central region of the recommendation screen, and the representative works of each character are displayed in the regions on both sides of the recommendation screen.

When the user selects the character name "Li Si" in the middle area of the recommendation screen, the display device 200 can send a search request with the hot search text "Li Si" to the server 400, so that the server 400 can further search for media related to "Li Si". Links and/or web links. After the search, the server 400 feeds back the search results to the display device 200, and the display device 200 replaces the representative works in the areas on both sides according to the received search results, and replaces them with "Li Si" related media links or Web links.

Obviously, since the recommended screen can be displayed through the OSD layer, during the process of updating the recommended screen, the display device 200 can also only update the content of the OSD layer, and maintain the separate rendering process of the OSD layer. Therefore, the video layer interface, such as the playback interface, the control home page, etc., can be played or displayed normally to meet the user's viewing needs.

In some embodiments, the display device 200 may further display a specific recommendation interface after receiving the associated media resource link fed back by the server 400 for the search request. The recommendation interface may be displayed in the upper OSD layer of the OSD layer where the recommendation screen is located. Compared with the traditional display device 200 that directly jumps to the search interface when performing the search function, in this embodiment, the associated media resource link can be displayed through the OSD layer, that is, while ensuring the normal playback or display of the video layer interface, the associated media asset link can be displayed separately through the OSD. The recommended interface for media resource links reduces the interference of the search process on the user's viewing process.

In some embodiments, after the display device 200 displays the recommendation screen, if the user selects a recommendation link, the display device 200 may perform a page jump operation to access or play the selected recommendation content. That is, as shown in FIG. 13 , the display device 200 may obtain the selection instruction input by the user based on the recommended link option in the recommendation screen. Obviously, similar to the search command and the picture recognition command, the selection command can also be input through the buttons of the control device 100, the touch screen, the intelligent voice assistant, and the like.

After the user inputs the selection instruction, the display device 200 may respond to the selection instruction and detect the link type of the recommended link specified by the selection instruction. Wherein, the link type includes a media asset address and a web page address. If the link type is a media asset address, that is, when it is determined that the user controls to play the recommended media asset, the display device 200 may send a data acquisition request to the server 400, and jump to a playback interface to play the selected media asset. If the link type is a webpage address, that is, when it is determined that the user controls access to the recommended webpage, the display device 200 may send a webpage access request to the server 400 to jump to a webpage browsing interface to display the visited webpage.

For example, after acquiring the associated data fed back by the server 400, the display device 200 may add the web page link of the profile of "Zhang San" identified from the image and the link of the character's related media assets and works in the recommendation screen. Therefore, the "Zhang San" profile option and the movie A, movie B, and movie C options that Zhang San starred in can be displayed on both sides of the recommendation screen. When the user selects the profile option of "San Zhang", the display device 200 can detect that the link type is a web page address, so it can send an access request to the server 400, thereby obtaining the content of the web page corresponding to the profile web page of "San Zhang", and jumping to The browser displays the profile page of "Zhang San". When used to select the movie A option, the display device 200 can detect that the selected link type is a media asset address, so it can send a data acquisition request to the server 400 to obtain the playback file of the movie A, and jump to the playback interface at the same time. Movie A plays.

In some embodiments, after the user performs an operation based on the recommended interface and jumps to a new interface such as a web page interface or a playback interface, the display device 200 may be controlled to exit the new interface through an interactive operation. For example, the user may press the return key on the control device 100 to control the display device 200 to exit the playback interface when the display device 200 displays the playback interface of the movie A.

In response to the exit operation input by the user, the display device 200 can return to the user interface displaying the recommendation screen according to the recorded operation path, and after returning to the recommendation screen, the focus cursor can also be set on the OSD layer where the recommendation screen is located, so that Then the user continues to perform other interactive actions based on the recommendation screen, such as clicking on the option of movie B to jump to the playback interface of movie B.

In some embodiments, in response to an exit operation input by the user, the display device 200 may also return to the original user interface that does not include the recommendation screen. For example, when the user triggers the display of a recommendation screen on the control home page, and selects to play movie A in the recommendation screen, when the user inputs an exit operation, the display device 200 can return to the display control home page, that is, return to the video layer interface that does not include the OSD layer, So that the user can directly cancel the OSD layer-related operations when exiting the new interface, and maintain the original viewing experience.

In some embodiments, the display device 200 may also set display hierarchical relationships for various user interfaces. For example, the display hierarchical relationship of user interfaces may be: control home page-media resource details interface-playing interface. In this regard, in response to an exit operation input by the user, the display device 200 may also jump to an upper-level interface of the current interface. For example, no matter whether the user chooses to play movie A on the recommendation screen or displays movie A on the media asset details page, when the user presses the exit key on the control device 100, he will jump to the media asset details interface so that the user can continue. Operations such as watching or understanding detailed information meet the needs of some users.

In the above embodiments, the image to be recognized sent by the display device 200 to the server 400 may be a screenshot of the user interface or a specific picture file. When the image to be recognized is a screenshot of the user interface, the display device 200 may also perform an image recognition operation on the entire user interface or a part of the user interface according to different needs of the user. For example, for a display device 200 that supports touch interactive operations, after the display device 200 enters the screen capture operation state, the user can adjust the size of the screen capture area by sliding with multiple fingers, so as to realize the partial screenshot operation of the user interface.

In order to facilitate user operations, the display device 200 can also crop the screenshot image after taking a screenshot, so as to obtain the image content of a local area of the user interface, that is, as shown in FIG. In the step of the image recognition instruction input by the user, the user screen capture interaction action is detected, and a screen capture operation is performed on the user interface according to the screen capture interaction action to generate a screen capture picture; and then the screen capture picture is cut to obtain the to-be Identify images.

For example, after the user presses the screenshot button on the control device 100 for a long time, the display device 200 can display the screenshot result on the interface. During the image display, the display device 200 can receive the button action on the control device 100 in real time and respond to the The operation on the key moves the cropping range, so that after the user presses the OK key, the cropping of the screenshot image of the user interface is completed.

After the cropped image, the main image area in the image can be sent to the server 400 for image recognition operation, thereby reducing the impact on the image recognition result when there are too many content elements in the image to be recognized, and improving the accuracy of the image recognition operation Rate.

In some embodiments, in order to recognize the image, the display device 200 may also start a picture recognition application in the step of performing a screen capture operation on the user interface according to the screen capture interaction action, so as to perceive the image through the picture recognition application. A screenshot event in the screenshot interaction action. The image recognition application may be a system application or a third-party application installed in the operating system of the display device 200 .

As shown in FIG. 15 , after the image recognition application is started, a screenshot command may be broadcast in the service operating system of the display device 200, so that the service operating system may respond to the screenshot command to perform a screenshot operation to generate a screenshot image. The business operating system then sends the screenshot to the image recognition application, so that the image recognition application can continue to perform operations such as image recognition on the screenshot. Through the map recognition application, the processing capability of the display device 200 can be expanded, so that more map recognition functions can be realized through continuous update and maintenance of the map recognition application, and different user needs can be met.

In order to facilitate the user to perform interactive operations based on the recommended screen, in some embodiments, the display device 200 may parse the associated data from the associated data in the step of rendering or presenting the recommended screen in the user interface according to the associated data. Identify target information. Wherein, the recognition target information includes target introduction text and target detail link, which are used to indicate the recognition object and recognition result in the image. After parsing and obtaining the identified target information, the display device 200 may add target introduction text and target detail links to the recommendation screen, so that users can learn about the recommended content they are interested in in time, and then select specific content.

In some embodiments, the recommended content determined through image recognition can enable the user to obtain recommended content related to the image content without viewing specific media content. The recommended content may include recommended media asset items and other recommended resource items, such as games, applications, web pages, text, and the like. The recommended content can be obtained by matching in the resource database. For example, the recommended media asset items are obtained by matching in the media asset database based on the image recognition results. Therefore, when the media asset database has a large amount of resources and the image recognition results tend to be associated with popular resources, it can be found in the media asset database. A large number of media assets are matched. However, when there are a large number of media asset items, a larger content display window is required for display, which not only blocks the user interface, but also hinders users from selecting the media asset items they are interested in, reducing user experience.

In order to meet the user's demand for recommended resources and reduce the number of irrelevant recommended items, in some embodiments of the present application, a display device 200 is also provided. The controller 250 of the display device may also be configured to execute a method for displaying recommended windows based on business scenarios, as shown in FIG. 16 , which specifically includes the following steps:

Obtain the image recognition instruction input by the user. This process is the same as the above-mentioned acquisition of the image recognition instruction input by the user, and will not be repeated here.

After acquiring the image recognition instruction, the display device 200 may respond to the image recognition instruction to detect the business scene to which the current user interface belongs.

In some embodiments, the service scenarios may be different modes set in the control system or application according to resource types. According to different operating system forms, the display device 200 can provide different types and quantities of business scenarios. For example, the business scenarios may include: children's mode, education mode, film and television mode, entertainment mode and so on. In different modes, the display device 200 can recommend different resources for the user. For example, in the children’s mode, the display device 200 can only display sub-supplied media items, such as cartoons, children’s films, etc., in the media recommendation interface; in the education mode, the display device 200 can display the Science and education media projects, such as course videos, online live courses, popular science documentaries, etc.

In some embodiments, the display device 200 will update the currently recorded service identifier after switching the mode on the user interface. In subsequent use, the current service scenario can be determined by detecting the service identifier. The display device 200 may also send the service identifier to the server 400, so that the server 400 can determine the current service scenario by detecting the service identifier.

In some embodiments, the acquisition of the business scene is not marked when the mode is switched, but the text and/or the name of the person in the screenshot is recognized, and then according to the recognized text and/or the name of the person, and the preset The mapping relationship determines the business identifier that represents the current business scenario. The mapping relationship is based on the mapping relationship between the name of the media resource resource, poster text, characters, etc., and the business scene corresponding to the media resource, which is established and stored in advance. Characters can be real or virtual. In some embodiments, if text or graphic content such as "Grade 1", "Grade 2", "Basic Mathematics" and "High School Sprint" are identified in the screenshot, it will be determined according to the preset mapping relationship that it is currently an educational scene In the same way, if the characters and or animation images such as "Wang Wang team make great contributions" and "Cute Chicken Squad" are recognized, it is determined that it is currently a children's scene; if "movie name 1", "movie name 2", " movie name 3", "movie name 4" etc., then it is determined that the current mode is a movie. The process of identifying the text and/or the name of the person in the image to determine the business scene may be to upload the business identification representing the business scene to the server 400 after the display device 200 is determined, or not to perform the business scene on the display device 200. Instead, the server 400 performs character recognition and business scene judgment based on uploaded screenshots.

In some embodiments, the business scenario may be a channel theme corresponding to a resource type displayed on a face-to-face page. Exemplarily, the server 400 classifies the media resources according to their types and classifies them into different channel themes, and the user can select different channel themes to display different types of media resources. The channel theme can be "children", "education", "shopping", etc. as shown in Figure 5A or Figure 6, and the control displaying the channel theme can be called TAB bar. The user controls the content area to display the content corresponding to the channel theme by manipulating the movement of the focus on different channel themes in the TAB column, and after displaying, the user can move the focus down to operate the controls in the content area. After the channel theme control is switched, the display device 200 will update the currently recorded service identifier.

In each business scenario, the display device 200 may only provide the user with resource items suitable for the business scenario on a page presenting media resources. For example, according to the user orientation of the display device 200, the service scenarios that the display device 200 can provide may include a normal mode and an education mode. In the regular mode, the display device 200 can display comprehensive media items such as popular movies, TV dramas, and variety shows on the media item recommendation interface, and can also display recommended items according to the user's viewing history. In the education mode, the display device 200 may display related course items in the media resource recommendation interface according to the course resources subscribed by the user.

In addition to the above manner of classifying business scenarios according to media asset item types, in some embodiments, the display device 200 may also classify business scenarios according to functions used by users. For example, some display devices 200 may provide a game mode. In the game mode, the display device 200 may display recommended game applications in an application list for users to choose to install and run.

It should be noted that the business scenarios that the display device 200 can provide are not limited to the above-mentioned children's mode, education mode, film and television mode, entertainment mode, regular mode, and game mode, etc. As well as the status of the resource library supported by the display device 200, the display device 200 can also provide other types of business scenarios, and in this business scenario, display resource items that match the current business scenario.

Based on the foregoing service scenarios, the display device 200 may detect the service scenario to which the current user interface belongs in various ways. In some embodiments, the display device 200 can monitor the user's operation of entering or exiting various business scenarios in real time, that is, obtain the control instruction input by the user for entering or exiting the service scenario, and respond to the control instruction, display the current Business scenarios are written into the system property database.

For example, as shown in FIG. 17 , the user controls the focus cursor to move to the "mode switch" control on the control home page through the control device 100, and selects the "education mode" option in the pop-up mode selection window, then the display device 200 can be controlled to enter educational model. At this time, the display device 200 can automatically record the current business scenario in the system attribute database as "education mode" by modifying the recording parameters in the system attribute database.

The display device 200 can mark various business scenarios by identifying character strings, and store them in the system attribute database. For example, a data table dedicated to recording business scenarios may be provided in the system attribute database, and the data table includes a type entry "Mode name". item to modify its corresponding value from "standard" for the general mode to "education" for the education mode.

Therefore, when detecting the business scene to which the current user interface belongs, the business scene to which the current user interface belongs is queried from the system attribute database. That is, after the display device 200 acquires the image recognition instruction input by the user, it reads the current state value of the type entry in the system attribute database in response to the image recognition instruction, so that the state value of the current type entry is read as "education". , it is determined that the business scenario to which the current user interface belongs is "education mode".

Since the display device 200 can also play the media item provided in the multimedia application by running the multimedia application. Different multimedia asset applications can provide media asset items from different platforms and with different characteristics. Therefore, the service scenario can also be determined according to the running multimedia resource application. For example, when the display device 200 runs the "AA-Early Education for Children" application, the display device 200 can query the local database or the cloud server 400 according to the application name "AA-Early Education for Children" about the type of the application, that is, an educational application for children. Then, according to the application type obtained from the query, match specific business scenarios that conform to the scenario division method of the display device 200 . That is, the media resources provided by the "AA-Children's Early Education" application are of the same type as the children's mode, so it can be determined that the current service scenario of the display device 200 is the children's mode.

Since some applications support feeding back their business scenarios to the display device 200, such as system applications, applications developed by operators, and third-party applications certified by operators, etc., the current business scenarios can also be detected through the results reported by the business applications. That is, in some embodiments, in the step of detecting the business scene to which the current user interface belongs, the business application of the current user interface is invoked, and a scene report notification is sent to the business application, and then the business application returns the business scene for the scene report notification.

In some embodiments, when the current business scenario cannot be determined through the system attribute database and application detection, the display device 200 can also identify the business scenario it belongs to based on the content contained in the current user interface. That is, in the step of detecting the service scene to which the current user interface belongs, the display device 200 may first obtain the position of the focus cursor in the user interface, and then extract the name of the current focus channel according to the position of the focus cursor to determine the current service scene.

In order to mark each business scenario, in the operating system of the display device 200, a scenario label can be set for each business scenario, and the display device 200 can identify various business scenarios according to the scenario tags. For example, as shown in FIG. 18 , the control homepage of the display device 200 may include multiple tabs, and each tab is provided with a name for marking the channel, including TV series, movies, documentaries, children, education, etc. The display device 200 determines the name of the current focus channel by detecting the location of the focus mark, that is, the name corresponding to the tab where the focus mark is located is "kids", thereby determining that the current service scene is "kids mode".

In order to meet the display diversity of the UI interface, the display device 200 will modify the names of the tabs when displaying some tabs. For example, the display device 200 may modify the tab name from "children" to "happy summer vacation" during part of the summer vacation. Therefore, in order to accurately determine the service scene, the display device 200 may call the standard service library after extracting the focus channel name, so as to match the service scene corresponding to the channel name through the standard service library. For example, the business scenario corresponding to "happy summer vacation" can be matched as "children's mode" in the standard business library. Wherein, a standard identification code of each service scenario and a channel name used to represent the service scenario may be recorded in the standard service library. The standard business library can follow the UI update policy of the operating system to update records in real time.

After detecting the current business scene, the display device 200 can generate an image recognition request corresponding to the target image to be recognized according to the detected current business scene and the map recognition instruction, and send the image recognition request to the server 400, so that the server 400 can respond to the image recognition request Feedback associated data. In some embodiments, the associated data includes hot search text and recommended items. The recommended items are items that meet business scenarios and are obtained by querying the resource database according to the hot search text.

In some embodiments, generating an image recognition request includes an image to be recognized (also referred to as a target image, for example, a screenshot) and a scene identifier representing a business scene. Screenshots can be used to provide feedback on screenshot results, and scene identification is used to obtain hot search text. In some embodiments, generating the image recognition request includes a screenshot and does not include a scene identifier representing a business scene, where the screenshot is used for feedback of a result of the screenshot, and the current business scene is determined according to the result of the view.

As shown in FIG. 19 , after receiving the image recognition request sent by the display device 200 , the server 400 may first perform image recognition on the target image to obtain the hot search text. The server 400 may perform image recognition processing on the target image according to an image recognition algorithm, so as to identify a specific target from the target image. The image recognition process is the same as above, and will not be repeated here.

In order to obtain the hot search text, in some embodiments, the server 400 may input the target image into the image recognition model to obtain the image recognition result. Wherein, the image recognition result includes keywords. Then, by extracting the hot search thesaurus in the business scenario, and matching the hot search text associated with the keyword in the hot search thesaurus.

In some embodiments, different business scenarios correspond to different hot search word bases.

In some embodiments, it is also possible to classify the hot search words with different business attributes in a hot search word library, and filter the hot search words corresponding to business scenarios in the hot search word library.

The server 400 may maintain a resource item database, and the resource item database may store a hot search word base based on network or local search engine statistics, that is, a set of words with high search frequency by users. All media asset items in the current platform can also be stored in the resource item database. After the word "variety show" is recognized in the image through the image recognition operation, the server 400 can match the hot search text related to "variety show" in the hot search word database, such as "Come on ××", "Run ××" and so on.

After the server 400 performs recognition processing on the target image to obtain the hot search text, the server 400 may feed back the hot search text to the display device 200 . The display device 200 then renders or presents a recommendation window according to the fed back hot search text. In the recommendation window, the hot search text can be used as a quick search item, which is used to search for related resource items using the selected hot search text as a keyword after the user selects it.

In some embodiments, the server 400 also feeds back the results of image recognition and image recognition recommendations.

In some embodiments, the display device 200 receives image recognition results and image recognition recommendations fed back by the server 400, as well as hot search word results and hot search recommendations.

In some embodiments, the display in the video layer continues, that is, if there is a video to play, it will continue to play, if there is a carousel picture to continue to carousel, all the display of the picture will be carried out according to the original logic of the screenshot interface, and will not be displayed. interrupt. In the floating layer above the video layer, the results fed back by the server 400 are displayed.

In some embodiments, the picture recognition title control is generated according to the picture recognition result and the picture recognition recommendation, and the hot search word title control is generated according to the hot search word result and the hot search recommendation. These two controls can acquire focus, and the focus is on the picture recognition title. When the control is used, the image recognition results and image recognition recommendations are displayed in the floating layer, and when the focus is on the hot search word title control, the hot search word results and hot search recommendations are displayed. The picture recognition title control and the hot search word title control are stored in the floating layer as the title bar for a long time, the logical relationship between the title bar in the floating layer and the content display area that can be switched to be displayed in the floating layer, and the title bar in the home page is the home page Logical relationship between content areas.

In some embodiments, when the focus is on the picture recognition title control, a screenshot is displayed in the middle of the content area in the floating layer, the person recognized by the picture recognition is displayed on one side of the screenshot, and the recommendation is displayed on the other side (similar to Figure 6) . When the focus is on the title control of the hot search words, the search box and the hot search word controls are displayed in the middle position, and the hot search recommendations are displayed on both sides of the middle position.

In some embodiments, only the middle position is switched, and one side of the middle position displays the characters for image recognition feedback, and the other side displays recommendations. When the focus is on the image recognition title control, the middle position of the content area in the floating layer displays screenshots. The focus is on the hot search word title control, and the search box and hot search word control are displayed in the middle.

During the search process based on the recommendation window, the display device 200 may send a search request to the server 400 according to the hot search text selected by the user. After receiving the search request, the server 400 can parse the hot search text in the search request, and query the resource database for items matching the business scenario according to the hot search text, so as to obtain recommended items.

For example, as shown in FIG. 20A , when the image recognition instruction received by the display device 200 is to perform a map recognition operation on a target image with a car. After detecting that the service scene to which the current user interface belongs is a video service scene, an image recognition request is sent to the server 400 . After receiving the image recognition request, the server 400 can analyze the target image, recognize the "car" target in the image, and match the hot search texts "speed and ××" and "×× flying car" through the hot search lexicon, And feed back to the display device 200 .

The display device 200 then renders or presents a recommendation window according to the hot search texts "speed and XX" and "XX speed", and the recommendation window includes options of "speed and XX" and "XX speed". After the user controls to select the "speed and ××" option, the display device 200 sends a search request containing the selected hot search text to the server 400 . According to the search request, the server 400 searches the resource subset corresponding to the current video service scene for recommended items associated with "speed and ××", that is, movie resources such as "speed and ××8", "speed and ××10" .

It can be seen that in different business scenarios, the recommended items queried by the server 400 according to the hot search text are different. For example, for the same picture containing a car, in the video business scenario, the hot search text and recommended items are movie information related to cars, that is, the hot search text is "speed and ××", "×× speed", etc. The name of the movie, and the recommended items are movie resources such as "Speed and ××", "×× Speed". In the educational scenario, the hot search texts and recommended items are car-related cartoons or teaching videos. That is, as shown in FIG. 20B , the hot search terms are cartoon titles such as "Car Town" and "Cars", and the recommended items are cartoon resources such as "Car Town" and "Cars". In the application game scenario, the recommended hot search texts and recommended items are game resources related to cars, that is, as shown in Figure 20C, the hot search texts are game names such as "XX Kart" and "XX Speed". The content of the game is also a car-related game.

In the above embodiments, the recommendation window may be an independent window suspended and displayed on the upper layer of the user interface, or may be a new interface after the display device 200 jumps. Options generated based on popular search text and recommended item links may be included in the recommendation window. The user can perform interactive operations through the recommendation window, control the display device 200 to perform operations such as selecting and previewing based on the recommended items, and control the display device 200 to perform operations such as playing, browsing, and jumping based on the recommended items in the recommendation window.

In some embodiments, the recommendation window may include multiple areas for displaying different content. Operations on the recommendation window and the recommendation window are similar to the above, and will not be repeated here.

In some embodiments, in order to present the recommendation window, the display device 200 may create a display layer above the layer where the user interface is located, and acquire hot search text and recommended items. Then call the display template of the recommendation window to add hot search text and recommended items to the display template to form a recommendation screen. Finally, display the recommended screen in the display layer. For example, after performing a search operation, the display device 200 may display a search result interface on the OSD layer of the current user interface.

As shown in FIG. 20A , when performing a search, the display device 200 may perform a search within the current business scenario, so as to search for resource items associated with the selected hot search text from the resource subset corresponding to the current business scenario. For example, after the user selects the hot search text "speed and ××", the display device 200 can send the search word containing the text "speed and ××" to the server 400, so that the server 400 can match the search terms with "speed and ××" on the current media platform. ××” related media asset items, and then feed it back to the display device 200, so as to render a display window on the OSD layer of the display device 200 to display the matching recommended items, as shown in FIG. 21 .

In some embodiments, during the search process, the display device 200 may first search in the current business scene and display the search results. When the search results in the current business scenario are displayed for a certain period of time, or after the user enters a command to search the entire network, the selected hot search text is used as the search term to search the entire network, and the search results of the entire network are presented.

The display device 200 may display recommended items in the recommended item area. The displayed recommended items can be changed in real time according to the user's interactive operation. For example, after the user clicks on the hot search text option of "speed and ××", the display device 200 can obtain information related to " The recommended items associated with "Speed and ××", that is, "Speed and ××1", "Speed and ××8" and other media asset items, are displayed in the recommended item area.

The display device 200 may also display initial recommendation content in the recommended item area when rendering or presenting the recommendation window. Wherein, the initial recommended content may be a recommended item queried by the server 400 according to the matching hot search text when performing image recognition. That is, in some embodiments, after the server 400 performs image recognition to obtain the hot search text, according to the hot search text, it queries the resource database for recommended items that meet the business scenarios, and forms the associated data together with the recommended items and the hot search text. , to feed back to the display device 200. The display device 200 then renders a recommendation window in the user interface according to the associated data.

For example, after the server 400 recognizes the car object in the image through the image recognition operation, it matches the hot search texts corresponding to the car object in the children's mode as "Car Town" and "Car Story". The server 400 then matches the video content related to "Car Town" and "Car Story" in the resource item database corresponding to the children's mode according to the hot search text, that is, "Car Town" and "Car Story", and extracts the corresponding media address. Therefore, the server 400 can combine hot search texts such as "Car Town" and "Cars" and media resource addresses of "Car Town" and "Cars" into associated data, and feed it back to the display device 200 . Therefore, when the recommendation window is displayed, the media asset items of "Car Town" and "Cars" can be displayed in the recommended item area.

It can be seen that through the method for displaying the recommendation window provided in the above embodiment, the display device 200 can use the detected current business scene to filter the hot search text and recommended items when using the image recognition function, so that the display in the recommendation window is more in line with the user's current situation. The hot search items and recommended resources that are required, simplify the resource items in the recommendation window, and facilitate user operations.

In the above embodiments, the target image of the picture recognition function may be a specific picture file, or a specific picture in the current user interface. When the image recognition operation targets different target images, the display device 200 may adopt different program steps. That is, as shown in FIG. 22 , in some embodiments, the display device 200 may detect the target image specified in the image recognition instruction after acquiring the image recognition instruction input by the user. If the specific image file indicated by the target image is specified in the image recognition instruction, it is determined that the user is performing image recognition operation on a specific image file, so the image can be directly extracted to generate an image recognition request and sent to the server 400 for image recognition processing.

If no target image is specified in the image recognition instruction, it is determined that the user has not used the image recognition function for a specific image file. At this time, the display device 200 can use the image recognition function for the current display content by default, so it can generate a screenshot command and broadcast the screenshot command to the service operating system. After receiving the screenshot command, the service operating system may respond to the screenshot command and perform a screenshot operation on the current user interface to generate a target image.

For example, when the user sees the poster or cover of some movie on the current user interface, he may be interested in the movie or actor, or want to know the detailed information corresponding to the movie. To this end, the user can press the image recognition button on the control device 100, and since there is no specific image file, the display device 200 can generate a screenshot command, and take a screenshot of the current user interface according to the screenshot command to generate a target image. Then, the image recognition request is sent to the server 400 to perform recognition processing on the image through image recognition technology, so as to recognize related information such as movie name and actor name from the image.

In some embodiments, when the number of hot search texts fed back from the server 400 to the display device 200 is large, the display device 200 may filter the hot search texts after receiving the associated data. That is, when the display device 200 renders or presents a recommendation window in the user interface according to the associated data, it may traverse the hot search texts in the associated data to filter out text sets that meet the business scenario, and then generate hot search options according to the text sets. In some embodiments, each hot search option includes at least one hot search text that meets the business scenario, so as to add the hot search option in the recommendation window.

In some embodiments, the recommended items in the recommendation window can also be sorted according to usage popularity, so that the user can select popular item content. That is, when the recommendation window is rendered in the user interface according to the associated data, the display device 200 can traverse the recommended items in the associated data, and then query the usage popularity of each item according to the name of the recommended item, so that the usage popularity is in descending order to add the recommended item to the recommendation window.

In some embodiments of the present application, a server 400 is also provided, and the server 400 includes: a storage unit, a communication unit, and a processing unit. Wherein, the storage unit is configured to store the data of media asset items; the communication unit is configured to connect to the display device 200; as shown in FIG. 23 , the processing unit is configured to execute the following program steps:

Responding to the image recognition request from the display device, acquiring the image to be recognized sent by the display device;

performing image recognition on the image to be recognized, so as to extract associated data from the media asset item data according to the recognition result of the image recognition;

feeding back the associated data to the display device, so that the display device renders or presents a recommendation screen according to the associated data, where the recommendation screen includes options generated based at least on the hot search text and/or recommended link .

In an embodiment of the present application, the associated data includes hot search text and/or recommended link; the hot search text is the text associated with the recognition result; the recommended link is the media associated with the recognition result address and/or web page address. It can be seen that the server 400 provided in this embodiment can perform image recognition processing on the image to be recognized after acquiring the image to be recognized sent by the display device 200, and identify a specific target or text from the image to be recognized, thereby extracting the image to be recognized. The hot search text and recommended link matching the image content are fed back to the display device 200, so that a recommended screen is rendered or presented on the user interface of the display device 200 for the user to perform operations.

It can be seen that, in the above embodiment, the server 400 can perform recognition processing on the image to be recognized, so the data processing amount of the display device 200 can be reduced. Moreover, by using the server 400 to maintain various types of image recognition models, different recognition needs can be met to obtain more types of associated data.

In order to be able to feed back associated data to the display device 200, as shown in FIG. 24 , in some embodiments, in the step of performing image recognition on the image to be recognized, the server 400 may input the image to be recognized into the recognition model, and The target information output by the recognition model is acquired. Wherein, the target information includes the type code output by the feature target recognition model and the keyword output by the character recognition model.

The built-in recognition model in the server 400 may include a recognition model obtained based on machine learning algorithm training, that is, the artificial intelligence model obtained after training a large number of training sample images can include specific targets in the output image after the image to be recognized is input, such as people, scenery, etc. etc., the classification probabilities of . The recognition module can also include a character recognition model based on Optical Character Recognition (OCR) technology, after the image to be recognized is input, the recognition model can determine its shape by detecting dark and bright patterns in the image, and then recognize The text information contained in the image.

After receiving the image to be recognized sent by the display device 200, the server 400 may copy the image to be recognized according to the number of built-in recognition models, so as to obtain multiple images and input them into the recognition models respectively. Different recognition models can output the classification probability of a specific target based on the same image content, as well as recognize text in the image to obtain target information. After obtaining the target information output by the recognition model, the server 400 can also extract the same type of media asset address and/or web page address from the media asset item data according to the type code to obtain a recommended link; Match synonyms in to get associated text.

For example, after the image recognition application obtains the image information to be recognized, the image to be recognized can be transmitted to the server 400 to trigger the server 400 to perform character (person or object) recognition on the image. After the calculation of the recognition model, the server 400 can identify After the target information is obtained, the identified character information can be digitized according to the rules agreed upon with the display device 200, that is, the type data is formed according to the agreement, such as "TYPE: 1" representing a sports star, etc., and then the digitized information Return to the image recognition application, and at the same time, perform hot search word matching based on the content identified in the current picture, determine the hot search word corresponding to the identified person, and send it to the display device 200 below.

In some embodiments, in the step of feeding back the associated data to the display device 400 , the server 400 may also obtain layout information of the current user interface on the display device 200 . For example, the server 400 may send a detection request to the display device 200 to trigger the display device 200 to upload the current layout information to the server 400 .

The layout information may include the shape, size, resolution, etc. of the currently recommended screen. The server 400 may calculate the number of options in the recommendation screen according to the layout information uploaded by the display device 200 . Finally, the associated data corresponding to the number of options is fed back to the display device 200 according to the calculated number of options. For example, when the current recommendation screen is obtained as a strip-shaped area at the bottom of the user interface, the server 400 may determine according to the width and height of both sides of the strip-shaped area that six recommended media asset options can be displayed in the current recommendation screen, namely, the

left side

3, 3 media asset display positions on the right. Therefore, the associated data that the server 400 may feed back to the display device 200 according to the calculated number of options may include six recommended media resource links.

In some embodiments of the present application, a server 400 is also provided. As shown in Figure 25, the processing unit is configured to perform the following procedural steps:

Acquire the image recognition request sent by the display device, where the image recognition request includes the target image and the business scene to which the current user interface of the display device belongs;

Perform image recognition on the target image to obtain trending text;

According to the hot search text, search the resource database for items that meet the business scenario to obtain recommended items;

generating associated data, the associated data including hot search text and recommended items;

The associated data is fed back to the display device, so that the display device renders a recommendation window in the user interface according to the associated data.

It can be seen from the above technical solutions that the server 400 provided in the above embodiment can perform image recognition on the target image to obtain the hot search text after obtaining the image recognition request sent by the display device 200, and then search the resource database for matching services according to the hot search text. Items of the scene to obtain recommended items; thereby generating associated data and feeding back the associated data to the display device 200, so that the display device 200 renders or presents a recommendation window in the user interface according to the associated data. The server 400 can obtain corresponding hot search texts and recommended items according to different business scenarios on the basis of image recognition, so that the items in the recommendation window rendered or presented by the display device 200 can adapt to the current business scenario , reduce redundant information, and improve user experience in this business scenario.

In some embodiments of the present application, a method for recommending media asset content is also provided, including the following steps:

The display device 200 acquires the image recognition instruction input by the user, and sends an image recognition request to the server 400 in response to the image recognition instruction, and the request includes the image to be recognized;

The server 400 performs image recognition on the image to be recognized, so as to extract associated data from the media asset item data according to the recognition result of the image recognition, and the associated data includes hot search text and/or recommended links ; The hot search text is the text associated with the recognition result; the recommended link is the media asset address and/or web page address associated with the recognition result;

The display device 200 renders or presents a recommendation screen in the user interface according to the associated data fed back by the server 400 , and the recommendation screen includes options generated based on the hot search text and/or the recommended link.

For example, when the user is watching the display device 200, by pressing the screenshot key (or voice wake-up) on the control device 100, the image recognition scene of the image recognition application in the display device 200 is triggered. After the scene is triggered, the image recognition application will sense the screenshot event first, and then broadcast a screenshot command to the business operating system of the display device 200 . After receiving the screenshot command, the service operating system of the display device 200 will execute a screenshot operation to obtain a screenshot image. After the screenshot operation is completed, the display device 200 transmits the screenshot image to the image recognition application. After receiving the screenshot, the image recognition application transmits the screenshot to the server 400 for person identification. The server 400 extracts the associated data according to the recognition result, and then returns the associated data to the image recognition application, and recognizes the content according to the current image, and distributes the hot search words.

After receiving the data returned by the server 400, the map recognition application presents the data to the terminal user through different UI interfaces according to the returned type. At the same time, after receiving the current keyword identified by the cloud, go to the media resource database to request relevant hot search data and popular media resources. Then, the media resource library performs data query according to the parameters of the display device 200, and returns the result to the display device 200. After receiving the returned result, the display device 200 renders and displays the data.

As can be seen from the above embodiments, the media asset content recommendation method provided by the present application can send the image to be recognized to the server 400 after the user inputs an image recognition command, so that the server 400 can perform image recognition on the image to be recognized, and generate an association according to the recognition result. The data. The display device 200 then renders or presents a recommendation screen according to the associated data, so as to display a recommendation interface including hot search text and/or recommendation link options in the user interface. The method can feed back related text and/or recommended links based on the recognition result of the image to be recognized, and increase the type of options in the recommendation screen, so that the user can select different related items according to requirements, and improve user experience.

Claims

A display device comprising:

a display configured to display an image;

a user input interface configured to receive user instructions;

a controller coupled to the display, user input interface and configured to:

Obtaining a picture recognition instruction input by a user while the display is displaying an image;

In response to the image recognition instruction, send an image recognition request to the server, the request includes the image to be recognized; and

receiving data associated with the image fed back by the server, the associated data at least including hot search text, where the hot search text is text associated with a recognition result obtained by performing image recognition on the image to be recognized ;

A recommendation screen is displayed on the display according to the associated data, the recommendation screen including an option generated based at least on the trending text to request a search related to the image.
The display device according to claim 1, wherein the controller is further configured to:

Obtain a search instruction input by the user based on the hot search text option in the recommendation screen; and

In response to the search instruction, send a search request to the server, where the search request includes the hot search text selected in the search instruction.
The display device according to claim 1, wherein the associated data further includes a recommended link, the controller is further configured to:

Acquiring the selected instruction input by the user based on the recommended link option in the recommended screen;

In response to the selection instruction, detecting the link type of the recommended link specified by the selection instruction, the link type includes a media resource address and/or a webpage address;

If the link type is a media asset address, send a data acquisition request to the server to jump to the playback interface to play the selected media asset;

If the link type is a webpage address, a webpage access request is sent to the server to jump to a webpage browsing interface to display the accessed webpage.
The display device according to claim 1, wherein the controller is further configured to:

extracting a recognition result obtained through image recognition and the associated data, the recommendation screen including a first tab and a second tab;

creating graphical options based on said recognition results and said associated data;

adding a graphics option created based on the recognition result in the first tab; and

In the second tab, an option of creating a graph based on the associated data is added.
The display device according to claim 1, wherein the controller is further configured to:

Create a display layer on the upper layer of the layer where the user interface is located;

Acquire the recognition result obtained through the image recognition and/or the associated data, so as to call the display template of the recommendation screen;

adding the recognition result and/or the associated data to the display template to generate a recommendation screen; and

The recommendation screen is displayed in the display layer.
The display device according to claim 1, wherein the controller is further configured to:

parsing identifying targeting information from said associated data, said identifying targeting information including targeting introductory text and/or targeting detail links; and

Add the target introduction text and/or target detail link in the recommendation screen.
The display device according to claim 1, wherein the controller is further configured to:

In response to the map recognition instruction, detect the business scene to which the current user interface belongs;

Wherein, the image recognition request sent also includes the business scenario, and

Wherein, the associated data received from the server further includes a recommended item, and the recommended item is an item conforming to the business scenario obtained by querying a resource database according to the hot search text.
The display device according to claim 7, wherein the controller is further configured to:

In the case of detecting the business scenario to which the current user interface belongs, call the business application of the current user interface;

sending a scenario report notification to the business application; and

receiving the service scenario returned by the scenario report notification from the service application.
The display device according to claim 7, wherein the controller is further configured to:

In the case of detecting the business scene to which the current user interface belongs, acquiring the position of the focus cursor in the user interface;

Extracting the name of the current focus channel according to the position of the focus cursor;

calls to standard business libraries; and

Search the standard service library for service scenarios matching the name of the focus channel.
The display device according to claim 7, wherein the controller is further configured to:

Obtain the control instructions input by the user for entering or exiting the business scene;

In response to the control instruction, write the current business scenario into the system attribute database;

When detecting the business scene to which the current user interface belongs, query the business scene to which the current user interface belongs from the system attribute database.
The display device according to claim 7, wherein the controller is further configured to:

Detecting whether the image to be recognized is specified in the image recognition instruction;

If the image to be recognized is not specified in the image recognition command, a screenshot command is generated;

broadcast the screenshot command to the business operating system;

Controlling the service operating system to execute a screenshot operation on the current user interface in response to the screenshot command, so as to generate the image to be recognized.
The display device according to claim 7, wherein the controller is further configured to:

traversing the hot search texts in the associated data to filter out text collections matching the business scenarios;

Generate hot search options according to the text collection, each of the hot search options includes at least one hot search text that meets the business scenario;

The hot search option is added to the recommendation window.
The display device according to claim 7, wherein the controller is further configured to:

iterating over recommended items in said associated data;

Query the usage popularity of each project according to the recommended project name;

The recommended items are added to the recommended screen in descending order of the usage popularity.
A method for recommending media asset content for a display device, comprising:

Obtain the image recognition instruction input by the user through the display device,

And in response to the image recognition instruction, send an image recognition request to the server, the request includes the image to be recognized;

receiving data associated with the image obtained by performing image recognition on the server, the associated data at least including hot search text, the hot search text being the recognition result obtained by performing image recognition on the image to be recognized associated text;

A recommendation screen is displayed in a user interface based on the associated data, the recommendation screen including an option generated based at least on the trending text to request a search related to the image.