WO2023000950A1 - Dispositif d'affichage et procédé de recommandation de contenu multimédia - Google Patents

Dispositif d'affichage et procédé de recommandation de contenu multimédia Download PDF

Info

Publication number
WO2023000950A1
WO2023000950A1 PCT/CN2022/103154 CN2022103154W WO2023000950A1 WO 2023000950 A1 WO2023000950 A1 WO 2023000950A1 CN 2022103154 W CN2022103154 W CN 2022103154W WO 2023000950 A1 WO2023000950 A1 WO 2023000950A1
Authority
WO
WIPO (PCT)
Prior art keywords
display device
image
search
user
recognition
Prior art date
Application number
PCT/CN2022/103154
Other languages
English (en)
Chinese (zh)
Inventor
王光强
Original Assignee
聚好看科技股份有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Priority claimed from CN202110836063.0A external-priority patent/CN115695844A/zh
Priority claimed from CN202111120100.4A external-priority patent/CN115866313A/zh
Application filed by 聚好看科技股份有限公司 filed Critical 聚好看科技股份有限公司
Priority to CN202280049050.1A priority Critical patent/CN117643061A/zh
Publication of WO2023000950A1 publication Critical patent/WO2023000950A1/fr

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
    • H04N21/431—Generation of visual interfaces for content selection or interaction; Content or additional data rendering

Definitions

  • the present application relates to the technical field of display devices, in particular to a display device and a method for recommending media content.
  • a display device refers to a terminal device capable of outputting specific display images, which may be terminal devices such as smart TVs, mobile terminals, smart advertising screens, and projectors.
  • terminal devices such as smart TVs, mobile terminals, smart advertising screens, and projectors.
  • smart TV is based on Internet application technology, has an open operating system and chip, has an open application platform, can realize two-way human-computer interaction, and integrates multiple functions such as audio-visual, entertainment, and data. Products are used to meet the diverse and individual needs of users.
  • the display device can provide a user with various user interfaces, and the user can control the display device to perform different operations based on various types of interfaces to meet the needs of the user.
  • the user interface provided by the display device may include a user interface for displaying links to multimedia resource resources, such as a media resource recommendation interface, a media resource list interface, and the like.
  • multimedia resource resources such as a media resource recommendation interface, a media resource list interface, and the like.
  • the media resource links displayed in the user interface are usually uniformly issued by the server connected to the display device, and can be adjusted individually according to the actual operation rules of the user.
  • the display device can adjust the arrangement order of the links of the media assets in the media asset recommendation interface according to the viewing history of the user, so that the media assets matching the viewing type of the user are arranged in the front area.
  • this media asset recommendation method relies too much on the user's viewing habits, which is not conducive to the user's exposure to new media asset types, making the content on the media asset recommendation page too single, and reducing user experience.
  • the display device can also support the image recognition function, that is, the user can control the display device to perform image processing on the target image (hereinafter also referred to as "image to be recognized") to recognize the characters, features, text, etc. in the target image information. Then, according to the image recognition result, match the associated media asset content in the media asset database, and display the media asset items through a specific content display window, so that the user can select the media asset items that may be of interest to play.
  • image recognition function that is, the user can control the display device to perform image processing on the target image (hereinafter also referred to as "image to be recognized") to recognize the characters, features, text, etc. in the target image information. Then, according to the image recognition result, match the associated media asset content in the media asset database, and display the media asset items through a specific content display window, so that the user can select the media asset items that may be of interest to play.
  • the associated media content is obtained through matching in the media database based on the image recognition results, when the resources in the media database are large and the image recognition results tend to be associated with popular resources, you can A large number of media assets are matched. However, when there are a large number of media asset items, a larger content display window is required for display, which not only blocks the user interface, but also hinders users from selecting the media asset items they are interested in, reducing user experience.
  • the present application provides a display device and a method for recommending media asset content to solve the problems of single media asset content and too many matching results of traditional image recognition functions in the user interface of traditional display devices.
  • an embodiment of the present application provides a display device, including:
  • a display configured to display an image
  • a user input interface configured to receive user instructions
  • a controller coupled to the display, user input interface and configured to:
  • the server receiving data associated with the image fed back by the server, the associated data at least including hot search text, where the hot search text is text associated with a recognition result obtained by performing image recognition on the image to be recognized ;
  • a recommendation screen is displayed on the display according to the associated data, the recommendation screen including an option generated based at least on the trending text to request a search related to the image.
  • an embodiment of the present application provides a method for recommending media content on a display device, including:
  • the associated data includes at least hot search text, and the hot search text is the same as the recognition result obtained by performing image recognition on the image to be recognized associated text;
  • a recommendation screen is displayed in a user interface based on the associated data, the recommendation screen including an option generated based at least on the trending text to request a search related to the image.
  • FIG. 1 is a usage scenario of a display device in an embodiment of the present application
  • Fig. 2 is a block diagram of the hardware configuration of the control device in the embodiment of the present application.
  • FIG. 3 is a hardware configuration diagram of a display device in an embodiment of the present application.
  • FIG. 4 is a software configuration diagram of a display device in an embodiment of the present application.
  • Figure 5A is a schematic diagram of the home page in the embodiment of the present application.
  • FIG. 5B is a schematic diagram of media asset recommendation in the embodiment of the present application.
  • FIG. 6 is a schematic diagram of a recommended screen in an embodiment of the present application.
  • FIG. 7 is a schematic structural diagram of a display device in an embodiment of the present application.
  • FIG. 8 is a time series relationship diagram of a method for recommending media asset content in an embodiment of the present application.
  • FIG. 9A is a schematic diagram of hot search text options in the embodiment of the present application.
  • FIG. 9B is a schematic diagram of search results based on the current media asset platform in the embodiment of the present application.
  • FIG. 9C is a schematic diagram of search results in the entire network in the embodiment of the present application.
  • FIG. 9D is a schematic diagram of the search input interface in the embodiment of the present application.
  • Fig. 10 is a schematic diagram of the image recognition tab in the embodiment of the present application.
  • Fig. 11A is a schematic diagram of a search tab in the embodiment of the present application.
  • FIG. 11B is a schematic diagram of another search tab in the embodiment of the present application.
  • Figure 12 is a schematic diagram of the search operation process in the embodiment of the present application.
  • Figure 13 is a schematic diagram of the selection operation process in the embodiment of the present application.
  • FIG. 14 is a schematic flow diagram of cutting a screenshot in an embodiment of the present application.
  • FIG. 15 is a sequence diagram of the operation flow of the image recognition application in the embodiment of the present application.
  • FIG. 16 is a schematic flowchart of a method for displaying a recommendation window in an embodiment of the present application.
  • Fig. 17 is a schematic diagram of the mode switching window in the embodiment of the present application.
  • Figure 18 is a schematic diagram of the children's mode interface in the embodiment of the present application.
  • FIG. 19 is a schematic flow diagram of the server generating associated data in the embodiment of the present application.
  • FIG. 20A is a schematic diagram of a recommendation window in the video mode in the embodiment of the present application.
  • FIG. 20B is a schematic diagram of the recommendation window in the children's mode in the embodiment of the present application.
  • FIG. 20C is a schematic diagram of the recommendation window in the game mode in the embodiment of the present application.
  • Figure 21 is a schematic diagram of the search result interface in the embodiment of the present application.
  • Fig. 22 is a schematic diagram of the process of generating a target image in the embodiment of the present application.
  • FIG. 23 is a schematic diagram of the server structure in the embodiment of the present application.
  • FIG. 24 is a schematic diagram of the process of extracting associated data by the server in the embodiment of the present application.
  • Fig. 25 is a sequence diagram of the display process of the recommendation window in the embodiment of the present application.
  • Fig. 1 is a schematic diagram of a usage scenario of a display device according to an embodiment.
  • the display device 200 also performs data communication with the server 400 , and the user can operate the display device 200 through the smart device 300 or the control device 100 .
  • control device 100 may be a remote controller, and the communication between the remote controller and the display device includes at least one of infrared protocol communication, Bluetooth protocol communication, and other short-distance communication methods, and the display device is controlled wirelessly or wiredly.
  • Device 200 The user can control the display device 200 by inputting user instructions through at least one of buttons on the remote control, voice input, and control panel input.
  • the display device 200 also performs data communication with the server 400 .
  • the display device 200 may be allowed to communicate via a local area network (LAN), a wireless local area network (WLAN), and other networks.
  • the server 400 may provide various contents and interactions to the display device 200 .
  • FIG. 2 is a configuration block diagram of the control device 100 according to some embodiments.
  • the control device 100 includes a controller 110 , a communication interface 130 , a user input/output interface 140 , a memory, and a power supply.
  • the control device 100 can receive the user's input operation instruction, and convert the operation instruction into an instruction that the display device 200 can recognize and respond to, and play an intermediary role between the user and the display device 200 .
  • FIG. 3 is a block diagram of a hardware configuration of a display device 200 according to some embodiments.
  • the display device 200 includes a tuner and demodulator 210, a communicator 220, a detector 230, an external device interface 240, a controller 250, a display 260, an audio output interface 270, a memory, a power supply, and a user interface 280. at least one of .
  • the communicator 220 is a component for communicating with external devices or servers according to various communication protocol types.
  • the communicator may include at least one of a Wifi module, a Bluetooth module, a wired Ethernet module and other network communication protocol chips or near field communication protocol chips, and an infrared receiver.
  • the display device 200 can establish transmission and reception of control signals and data signals with the control device 100 or the server 400 through the communicator 220 .
  • the external device interface 240 may include but not limited to the following: high-definition multimedia interface (HDMI), analog or data high-definition component input interface (component), composite video input interface (CVBS), USB input interface ( Any one or more interfaces such as USB), RGB port, etc.
  • HDMI high-definition multimedia interface
  • component analog or data high-definition component input interface
  • CVBS composite video input interface
  • USB input interface Any one or more interfaces such as USB
  • RGB port etc.
  • the controller 250 and the tuner-demodulator 210 may be located in different split devices, that is, the tuner-demodulator 210 may also be located in an external device of the main device where the controller 250 is located, such as an external set-top box Wait.
  • the controller 250 controls the operation of the display device and responds to user operations through various software control programs stored in the memory.
  • the controller 250 controls the overall operations of the display device 200 . For example, in response to receiving a user command for selecting a UI object to be displayed on the display 260, the controller 250 may perform an operation related to the object selected by the user command.
  • the user can input user commands through a graphical user interface (GUI) displayed on the display 260, and the user input interface receives user input commands through the graphical user interface (GUI).
  • GUI graphical user interface
  • the user may input a user command by inputting a specific sound or gesture, and the user input interface recognizes the sound or gesture through a sensor to receive the user input command.
  • the system is divided into four layers, from top to bottom are respectively the application (Applications) layer (abbreviated as “application layer”), application framework (Application Framework) layer (abbreviated as “framework layer”) "), Android runtime (Android runtime) and system library layer (referred to as “system runtime layer”), and the kernel layer.
  • application layer application layer
  • application framework Application Framework
  • Android runtime Android runtime
  • system library layer system library layer
  • there is at least one application program running in the application program layer and these application programs can be window (Window) program, system setting program or clock program etc. that come with the operating system; they can also be developed by third-party developers. s application.
  • the application program packages in the application program layer are not limited to the above examples.
  • the framework layer provides an application programming interface (application programming interface, API) and a programming framework for applications in the application layer.
  • the application framework layer includes some predefined functions.
  • the application framework layer is equivalent to a processing center, which decides to make the applications in the application layer take actions.
  • the API interface Through the API interface, the application program can access the resources in the system and obtain the services of the system during execution.
  • the application framework layer includes managers (Managers), content providers (Content Provider) etc.
  • the manager includes at least one of the following modules: activity manager (Activity Manager) Interact with all activities running in the system; the Location Manager is used to provide system services or applications with access to the system location service; the Package Manager is used to retrieve the information currently installed on the device Various information related to the application package; Notification Manager (Notification Manager) is used to control the display and clearing of notification messages; Window Manager (Window Manager) is used to manage icons, windows, toolbars, wallpapers on the user interface and desktop widgets.
  • Activity Manager Activity Manager
  • the Location Manager is used to provide system services or applications with access to the system location service
  • the Package Manager is used to retrieve the information currently installed on the device Various information related to the application package
  • Notification Manager Notification Manager
  • Window Manager Window Manager
  • the kernel layer is a layer between hardware and software. As shown in Figure 4, the kernel layer at least includes at least one of the following drivers: audio driver, display driver, bluetooth driver, camera driver, WIFI driver, USB driver, HDMI driver, sensor driver (such as fingerprint sensor, temperature sensor, pressure sensors, etc.), and power drives, etc.
  • the kernel layer at least includes at least one of the following drivers: audio driver, display driver, bluetooth driver, camera driver, WIFI driver, USB driver, HDMI driver, sensor driver (such as fingerprint sensor, temperature sensor, pressure sensors, etc.), and power drives, etc.
  • the user can control the display device 200 to display various user interfaces during the process of using the display device 200 .
  • recommended resources may be displayed according to the viewing habits of the user.
  • the display device can display homepages of different channels.
  • different media resources can be displayed on the homepage column for media resource recommendation. Users can select the column position to expand or view the next page. Users can also search for media resources by entering the search page through the search page set on the display page. Users can also directly obtain feedback results through voice search.
  • the user can display the information related to movie A in the "guess you like" area according to the type, content, author and other information of movie A.
  • Movie B of the same type, movie C with similar content to movie A, and movie D with the same author as movie A are available for the user to choose.
  • the display device 200 may have a built-in control program for recording the user's viewing habits. As the user watches, the display device 200 can record the content watched by the user by starting the control program to form historical record information. When the recommended content needs to be displayed, the display device 200 can use the historical record information to perform matching in the resource library, so as to obtain the recommended content that matches the content watched by the user.
  • the method of determining recommended content based on historical records can only obtain part of the content associated with the media assets of the historical records, which is not conducive to the user's understanding of new content. For example, when a user sees a poster or a screenshot of a certain movie, he or she may be interested in the movie or actors, or want to learn more about the movie.
  • the display device 200 can perform image recognition processing on the image through image recognition technology, so as to recognize relevant information such as movie titles and actor names from the image, thereby providing a more convenient search experience for users.
  • the user may control the display device 200 to perform an image recognition operation, and in this operation, a screenshot operation may be performed to obtain a screenshot image.
  • An image recognition algorithm may be executed on the screenshot image, so as to identify the person target in the screenshot image according to the pixel point distribution law in the screenshot image.
  • the recommended media resource content suitable for the character type is determined. For example, for a character of the film and television star type, different film and television works of the same character can be determined; for a task of the sports star type, the sports-related video resources can be determined.
  • the display device 200 may display the corresponding recommended media asset item at a specific position on the user interface for the user to select.
  • the position of movie B and/or movie C in FIG. 6 can be used to display the person recognition result.
  • the remaining display slots may be used to display recommended media asset items.
  • the display content is displayed on the video layer
  • the screenshot recognition result is displayed on the OSD layer above the video layer
  • the person recognition result and recommended media asset items are all displayed through the display position control on the OSD layer.
  • the focus is set to any control position on the OSD layer, and at this time, the control of the video layer cannot obtain the focus, that is, the focus will not move to the video when the OSD layer is displayed.
  • the position of the control on the layer After the user chooses to cancel the display of the OSD layer, the control of the video layer starts to get the normal focus.
  • the recommended content determined through image recognition can enable the user to obtain recommended content related to the image content without viewing specific media content.
  • the method of recommending media asset items only based on image recognition results is far from meeting the needs of users for richer content. For example, some users use the image recognition method not to obtain other works of the characters in the image, but to understand the characters in the image.
  • a display device 200 is provided.
  • the display device 200 includes: a display 260 , a communicator 220 and a controller 250 .
  • the display 260 can be used to display images.
  • the communicator 220 may connect to the server 400 through remote communication such as a network connection, so as to realize data interaction with the server 400 .
  • the user input interface is configured to receive user instructions.
  • the controller 250 may be configured to execute an image recognition-based content recommendation method for displaying various types of recommended content in all or part of the image, and it may be configured to: acquire user An input map recognition instruction; in response to the map recognition instruction, send an image recognition request to the server, the request includes an image to be recognized; and receive data associated with the image fed back by the server, the associated data At least including hot search text, the hot search text is text associated with the recognition result obtained by performing image recognition on the image to be recognized; displaying a recommendation screen on the display according to the associated data, the recommendation A screen includes an option generated based at least on the trending text to request a search related to the image.
  • the display device 200 can monitor the user's interactive actions in real time. Different interactive actions can be used to control the display device 200 to implement different functions. Wherein, when the user inputs an interactive action for controlling the display device 200 to perform image recognition, it means that the user inputs an image recognition instruction. According to different specific interaction modes of the display device 200, the user may input a map recognition instruction through different interaction modes.
  • the user may perform an interactive action through the control device 100 provided with the display device 200 to input a map recognition instruction.
  • a screenshot key may be provided on the control device 100, and when the user presses the screenshot key, the display device 200 may be triggered to perform a screenshot operation to obtain a screenshot image.
  • the display device 200 can also set screenshot rules, that is, in different user interfaces, image processing-related operations are automatically executed after taking screenshots to obtain images. For example, if the user presses the screenshot key when the display device 200 is displaying the playback interface, after the screenshot is taken to obtain a specific image, the image obtained by the screenshot is saved. When the user presses the screenshot key when the display device 200 displays the media asset list interface, the image can be automatically identified after the specific image is captured by screenshot. That is, the image recognition command can be input by the user by pressing a button on the control device 100 when the display device 200 displays the media asset list interface.
  • screenshot rules that is, in different user interfaces, image processing-related operations are automatically executed after taking screenshots to obtain images. For example, if the user presses the screenshot key when the display device 200 is displaying the playback interface, after the screenshot is taken to obtain a specific image, the image obtained by the screenshot is saved.
  • the image can be automatically identified after the specific image is captured by screenshot. That is, the image recognition command can be input by
  • the display device 200 may also display controls for the image recognition operation in the user interface for the user to select.
  • the user may call out the status bar of the user interface through the control device 100, and the status bar may include a "screenshot/picture recognition" option.
  • the user controls the movement of the focus cursor and selects the "screenshot/picture recognition” option through the direction keys and the confirmation key on the control device 100, and triggers the display device 200 to perform screen capture and picture recognition operations.
  • the image recognition command can be input based on the option control in the user interface.
  • an intelligent voice system may be built in, and the user may use the intelligent voice system to input voice input image recognition instructions.
  • the user can input voice content such as "recognize the content in the picture” and “recognize the person in the current picture”, then the intelligent voice system can recognize the voice content and convert it into a specific control command to drive the display device 200 to perform the recognition.
  • Figure operations For part of the display device 200, an intelligent voice system may be built in, and the user may use the intelligent voice system to input voice input image recognition instructions.
  • the user can input voice content such as "recognize the content in the picture” and “recognize the person in the current picture”, then the intelligent voice system can recognize the voice content and convert it into a specific control command to drive the display device 200 to perform the recognition.
  • the user can not only input an image recognition instruction for a specific screen in the user interface, but also input an image recognition instruction for a specific image file.
  • the user can control the display device 200 to perform a picture recognition operation on the opened picture file by long pressing the screenshot button on the control device 100 .
  • the display device 200 may send the image to be recognized to the server in response to the image recognition instruction.
  • the image to be recognized may have different image forms according to different targets of the image recognition operation. For example, when the image recognition instruction received by the display device 200 is triggered by a screen capture operation, the image to be recognized is a screen capture image obtained by the screen capture operation.
  • the image recognition instruction received by the display device 200 is for a picture file that has been opened by the display device 200, the image to be recognized is the picture file.
  • the display device 200 may extract the image to be recognized indicated by the image recognition instruction, and transcode and compress the extracted image file to form a data packet. Then according to the network connection relationship between the display device 200 and the server 400, the data packet is sent to the server 400 with a specific transmission protocol to perform an image recognition request.
  • the server 400 after receiving the image recognition request, the server 400 obtains the data packet, and may decompress and decode the data packet, thereby parsing the image to be recognized in the data packet. Then the server 400 performs image recognition processing on the image to be recognized according to the image recognition algorithm, so as to identify a specific target from the image to be recognized.
  • the server 400 may have a built-in recognition model for recognizing human objects in images, and after the display device 200 sends the image to be recognized to the server 400, the server 400 may input the image to be recognized sent by the display device 200 into the recognition model.
  • the recognition model After the calculation of the recognition model, it is possible to output whether the image to be recognized contains a person target, and the person information conforming to the characteristics of the person target.
  • the content output by the recognition model may specifically be the classification probability that the person object belongs to a certain person, and finally the person information of the characteristics of the person object is determined as the label type with the highest classification probability.
  • Recognition of different targets in the image to be recognized can be realized by presetting multiple types of recognition models in the server 400 .
  • the server 400 can also preset a character recognition model for recognizing specific text content contained in an image, a scene recognition model for recognizing scene content contained in an image, and a scene recognition model for recognizing image
  • the specific input recognition model of the image to be recognized can be set through the display device 200 . That is, the more types of information that need to be recognized in the image, the more types of recognition models the server 400 will input the image to be recognized into. Obviously, when the server 400 inputs more types of recognition models for images to be recognized, the time required to obtain the recognition results will be longer. Therefore, in order to balance the types of recognition results and the recognition time, in some embodiments, the server 400 can A feature target recognition model and a character recognition model are respectively set up, which are respectively used to recognize and obtain graphic information in pictures.
  • associated data extracted based on the recognition result may be fed back to the display device 200 . That is, the display device 200 may receive associated data fed back by the server 400 .
  • the associated data includes hot search text and/or recommended links.
  • the hot search text may be the text associated with the recognition result obtained by performing image recognition on the image to be recognized;
  • the recommended link may be the media resource address and/or webpage address associated with the recognition result.
  • the server 400 when the server 400 performs character matching and recognition on the images received from the display device 200 , it also performs character recognition.
  • the server 400 will obtain the relative position of the recognition object in the image, and then, after obtaining the result of the person recognition and the character recognition result, can recognize the person whose positional relationship meets the preset condition
  • the results are associated with the text recognition results.
  • the positional relationship may be that the distance is less than a preset threshold, or that the object corresponding to the character recognition result is below the image corresponding to the person recognition result, or a combination of both.
  • the server 400 may maintain a media asset item database, and the media asset item database may store a hot search term base based on network or local search engine statistics, that is, a set of words with high search frequency by users. All media asset items in the current platform may also be stored in the media asset item database.
  • the server 400 can match the hot search text related to "variety show” in the hot search word database.
  • the server 400 can match the film and television content related to "Zhang San” in the media asset project database, and extract the corresponding media asset address. Therefore, the server 400 can combine the hot search text related to "Variety Show” and the media resource address related to "Zhang San” into associated data, and feed it back to the display device 200 .
  • the server 400 may recommend hot search texts only based on the recognized text results, and at the same time recommend media resources only based on the character recognition results. In some embodiments, it is also possible to recommend popular texts and/or media resources based on the recognized text results and character recognition results.
  • the server 400 can recommend hot search texts only based on the text recognition results that meet the preset positional relationship with the person recognition results, and ignore the texts that do not meet the positional relationship. Recognition results.
  • the server 400 may also recommend hot search texts according to the hot searches of the current search function.
  • the display device 200 may display a recommendation screen in the user interface according to the associated data.
  • the recommendation screen may be rendered for display.
  • the recommendation screen includes options generated based on hot search text and/or recommended links.
  • the recommendation screen may be a new interface that the display device 200 jumps to, or may be a floating window displayed in a specific area of the original interface.
  • the recommendation screen may be located at the bottom area of the current user interface.
  • the long strip window of the recommendation screen there may be a search area in the middle and recommended item areas on both sides.
  • the display device 200 may display a search box and hot search text options below the search box in the search area for the user to select. When the user selects any hot search text option, the display device 200 may automatically perform a search operation for the hot search text.
  • the display device 200 may display a search result interface on the OSD layer of the current user interface. As shown in FIG. 9B , when performing a search, the display device 200 may search within the current media asset platform, so as to search the current media asset platform for media asset items associated with the selected hot search text. For example, after the user selects the hot search text "Zhang San", the display device 200 can send the search term containing the text "Zhang San" to the server 400, so that the server 400 can match the media related to "Zhang San" on the current media platform. and then feed it back to the display device 200 to render or present a media resource display window on the OSD layer of the display device 200 to display the associated items obtained by matching.
  • the display device 200 when it performs a search, it may also perform a network-wide search through a network search engine. For example, after the user selects any hot search text, the display device 200 can search the entire network through the search website by visiting a specified search website and using the selected hot search text as a keyword, and display the search results on the display device 200.
  • the OSD layer displays the search result webpage, as shown in FIG. 9C.
  • the display device 200 may successively display the search results in the above two manners. That is, after the user selects any hot search text, two windows can be rendered or presented on the OSD layer, which are used to display the search results in the current media platform and the search results of the whole network respectively.
  • the display device 200 may first search in the current media asset platform and display the search results.
  • the search results in the current media asset platform are displayed for a certain period of time, or after the user enters an instruction to search the entire network, the selected hot search text is used as the search term to search the entire network, and the search results of the entire network are displayed.
  • the display device 200 can start the normal search function. That is, as shown in FIG. 9D , after the user selects the search box, input controls, such as a keyboard, a tablet, and a voice assistant, may be displayed in the OSD layer. Users can input the text content they want to search based on the displayed input control.
  • input controls such as a keyboard, a tablet, and a voice assistant
  • the display device 200 can also display recommended links in the recommended item area. When the user selects any recommended link, the display device 200 can be controlled to jump to the specified media asset details interface or a specific web page interface to realize playback or access operations and satisfy the user's needs. specific needs.
  • a part of the time will be consumed for data transmission. That is, when the display device 200 sends the image to be recognized to the server 400, it will consume a part of the time, and when the server 400 feeds back the associated data to the display device 200, it will also consume a part of the time. Combined with the time consumed by the image recognition process, it will lead to the entire recommendation process. for too long.
  • a recognition model with a high frequency of use can also be preset in the display device 200, so that when the user inputs an image recognition instruction, the recognition model built in the display device 200 can be used first.
  • Image recognition is performed first, so as to obtain some types of image recognition results, so that the display device 200 can first render or present a recommended picture for the user to choose.
  • the display device 200 then sends the image to be recognized to the server 400 and receives associated data fed back by the server 400 .
  • the efficiency of content recommendation can be improved, and the diversity of content recommendation types can be taken into account.
  • the image recognition operation can be completed by the display device 200 and the server 400 respectively, the data processing amount of the display device 200 or the server 400 can also be reduced, and excessive occupation of computing resources in the image recognition process can be avoided.
  • the recommendation screen rendered by the display device 200 may include two tabs (tab pages), one of which is used to display image recognition results; the other tab is used to display searched and recommended results.
  • the recommendation screen may include a " ⁇ Picture Recognition” tab and a "Search” tab. Recognize the thumbnail of the image, and display the recognized results from the image, such as people, text, products, etc., in the areas on both sides. Therefore, after the user inputs the image recognition command, the display device 200 can first input the image to be recognized into the product recognition model and the person recognition model respectively, so as to obtain product recommendation content and character recommendation content, and display them in the " ⁇ picture recognition" tab. Render or present as map recognition result options for users to view and select.
  • the user can also control the display device 200 to switch to this tab by selecting the "Search" option.
  • a search box can be displayed in the central area, and multiple hot search texts can be displayed below the search box.
  • the media resource links and/or web page links recommended according to the image recognition results are displayed in the areas on both sides. For this reason, when the user clicks on the "Search" tab, the display device 200 can send the image to be recognized to the server 400, and further perform image recognition operations on the image to be recognized through the recognition model in the server 400, so as to obtain information including hot search text, Data associated with media links and/or web links.
  • the recommendation screen can have a specific layout. For example, as shown in FIG. 10 , the product identification result may be displayed on the left area of the recommendation screen, and the recognized image and name of the person may be displayed on the right area. Different tabs can maintain the same layout, as shown in Figure 11A, or change the layout after switching tabs, as shown in Figure 11B, the layout can be rearranged in the left display area to display new content.
  • the recommendation screen presented by the display device 200 may be displayed through an OSD layer located above the video layer. That is, in some embodiments, after the user inputs the image recognition instruction, the display device 200 may respond to the image recognition instruction and call the OSD layer to display or render the interface.
  • the display device 200 may call the display template of the recommended screen, and obtain the image recognition result output by the image recognition model and the associated data fed back by the server 400 . Finally, a specific recommended picture is rendered or presented according to the display template and the image recognition result, so as to be displayed on the OSD layer.
  • different tabs in the recommendation screen may also be displayed through different OSD layers.
  • the recommendation screen includes two tabs of "XX Image Recognition” and "Search”
  • two OSD layers can be called to render or present "XX Image Recognition” respectively when rendering or presenting the recommendation screen Image recognition result screen in the tab and recommended items in the "Search" tab.
  • the user can display the content of each tab by controlling the display/hide state of the corresponding OSD layer.
  • the display device 200 can issue the content in advance through the server 400, that is, when the user inputs a picture recognition command, the server 400 directly completes the picture recognition operation and content recommendation, and combines the picture recognition result and the recommended item together. And send it to the display device 200, so as to perform rendering or presentation operations on different OSD layers.
  • the content displayed in each tab can also be obtained after the user selects the tab label control, that is, after the display device 200 detects that the user clicks the "search" option, it obtains the recommended item from the server 400 for rendering Or render the Search tab.
  • the display process of the recommended screen will not affect the display process of the original user interface of the display device 200, thereby maintaining the normal display of the video layer screen and improving user experience.
  • displaying the recommendation screen through the OSD layer also facilitates the user to return to the original user interface through simple interactive operations. For example, when the search operation does not need to be performed, the display device 200 can be controlled to cancel displaying the recommendation screen and continue to display the original user interface by pressing the "exit" button on the control device 100 .
  • the user can also perform further interactive operations based on the displayed recommended screen, as shown in FIG. Obtain the network resource content corresponding to the hot search text.
  • the display device 200 may first obtain a search instruction input by the user based on the hot search text option in the recommendation screen. For example, the user can use the direction keys on the control device 100 to control the focus cursor to move to the hot search text of interest.
  • the display device 200 can be controlled to search using the selected hot search text as a search term, that is, the display device 200 obtains the search command input by the user.
  • the display device 200 may send a search request to the server 400 in response to the search instruction.
  • the search request includes the hot search text selected in the search instruction.
  • the server 400 can perform a search operation after starting a network resource search engine or a local search engine, that is, the server 400 can search and select the media resource associated with the hot search text in the resource item data, and feed back the search result to the display device 200 .
  • the display device 200 After the display device 200 receives the associated media asset link fed back by the server 400 for the search request, it may update the options in the recommendation screen according to the associated media asset link. For example, when multiple characters can be identified in the picture to be recognized through the image recognition operation, the server 400 extracts the name of the person who has been searched more frequently among the multiple characters as the hot search text, and feeds back the name of the person who has been searched more frequently to The device 200 is displayed, and at the same time, a resource link of a representative work of each character is fed back. The display device 200 then renders or presents a recommendation screen according to the received character name and representative works. That is, the names of multiple characters are displayed in the central region of the recommendation screen, and the representative works of each character are displayed in the regions on both sides of the recommendation screen.
  • the display device 200 can send a search request with the hot search text "Li Si” to the server 400, so that the server 400 can further search for media related to "Li Si". Links and/or web links.
  • the server 400 feeds back the search results to the display device 200, and the display device 200 replaces the representative works in the areas on both sides according to the received search results, and replaces them with "Li Si" related media links or Web links.
  • the display device 200 can also only update the content of the OSD layer, and maintain the separate rendering process of the OSD layer. Therefore, the video layer interface, such as the playback interface, the control home page, etc., can be played or displayed normally to meet the user's viewing needs.
  • the display device 200 may further display a specific recommendation interface after receiving the associated media resource link fed back by the server 400 for the search request.
  • the recommendation interface may be displayed in the upper OSD layer of the OSD layer where the recommendation screen is located.
  • the associated media resource link can be displayed through the OSD layer, that is, while ensuring the normal playback or display of the video layer interface, the associated media asset link can be displayed separately through the OSD.
  • the recommended interface for media resource links reduces the interference of the search process on the user's viewing process.
  • the display device 200 may perform a page jump operation to access or play the selected recommendation content. That is, as shown in FIG. 13 , the display device 200 may obtain the selection instruction input by the user based on the recommended link option in the recommendation screen.
  • the selection command can also be input through the buttons of the control device 100, the touch screen, the intelligent voice assistant, and the like.
  • the display device 200 may respond to the selection instruction and detect the link type of the recommended link specified by the selection instruction.
  • the link type includes a media asset address and a web page address. If the link type is a media asset address, that is, when it is determined that the user controls to play the recommended media asset, the display device 200 may send a data acquisition request to the server 400, and jump to a playback interface to play the selected media asset. If the link type is a webpage address, that is, when it is determined that the user controls access to the recommended webpage, the display device 200 may send a webpage access request to the server 400 to jump to a webpage browsing interface to display the visited webpage.
  • the display device 200 may add the web page link of the profile of "Zhang San” identified from the image and the link of the character's related media assets and works in the recommendation screen. Therefore, the "Zhang San" profile option and the movie A, movie B, and movie C options that Zhang San starred in can be displayed on both sides of the recommendation screen.
  • the display device 200 can detect that the link type is a web page address, so it can send an access request to the server 400, thereby obtaining the content of the web page corresponding to the profile web page of "San Zhang", and jumping to The browser displays the profile page of "Zhang San”.
  • the display device 200 can detect that the selected link type is a media asset address, so it can send a data acquisition request to the server 400 to obtain the playback file of the movie A, and jump to the playback interface at the same time. Movie A plays.
  • the display device 200 may be controlled to exit the new interface through an interactive operation.
  • the user may press the return key on the control device 100 to control the display device 200 to exit the playback interface when the display device 200 displays the playback interface of the movie A.
  • the display device 200 can return to the user interface displaying the recommendation screen according to the recorded operation path, and after returning to the recommendation screen, the focus cursor can also be set on the OSD layer where the recommendation screen is located, so that the user continues to perform other interactive actions based on the recommendation screen, such as clicking on the option of movie B to jump to the playback interface of movie B.
  • the display device 200 may also return to the original user interface that does not include the recommendation screen. For example, when the user triggers the display of a recommendation screen on the control home page, and selects to play movie A in the recommendation screen, when the user inputs an exit operation, the display device 200 can return to the display control home page, that is, return to the video layer interface that does not include the OSD layer, So that the user can directly cancel the OSD layer-related operations when exiting the new interface, and maintain the original viewing experience.
  • the display device 200 may also set display hierarchical relationships for various user interfaces.
  • the display hierarchical relationship of user interfaces may be: control home page-media resource details interface-playing interface.
  • the display device 200 in response to an exit operation input by the user, may also jump to an upper-level interface of the current interface. For example, no matter whether the user chooses to play movie A on the recommendation screen or displays movie A on the media asset details page, when the user presses the exit key on the control device 100, he will jump to the media asset details interface so that the user can continue. Operations such as watching or understanding detailed information meet the needs of some users.
  • the image to be recognized sent by the display device 200 to the server 400 may be a screenshot of the user interface or a specific picture file.
  • the display device 200 may also perform an image recognition operation on the entire user interface or a part of the user interface according to different needs of the user. For example, for a display device 200 that supports touch interactive operations, after the display device 200 enters the screen capture operation state, the user can adjust the size of the screen capture area by sliding with multiple fingers, so as to realize the partial screenshot operation of the user interface.
  • the display device 200 can also crop the screenshot image after taking a screenshot, so as to obtain the image content of a local area of the user interface, that is, as shown in FIG.
  • the user screen capture interaction action is detected, and a screen capture operation is performed on the user interface according to the screen capture interaction action to generate a screen capture picture; and then the screen capture picture is cut to obtain the to-be Identify images.
  • the display device 200 can display the screenshot result on the interface.
  • the display device 200 can receive the button action on the control device 100 in real time and respond to the The operation on the key moves the cropping range, so that after the user presses the OK key, the cropping of the screenshot image of the user interface is completed.
  • the main image area in the image can be sent to the server 400 for image recognition operation, thereby reducing the impact on the image recognition result when there are too many content elements in the image to be recognized, and improving the accuracy of the image recognition operation Rate.
  • the display device 200 may also start a picture recognition application in the step of performing a screen capture operation on the user interface according to the screen capture interaction action, so as to perceive the image through the picture recognition application.
  • the image recognition application may be a system application or a third-party application installed in the operating system of the display device 200 .
  • a screenshot command may be broadcast in the service operating system of the display device 200, so that the service operating system may respond to the screenshot command to perform a screenshot operation to generate a screenshot image.
  • the business operating system then sends the screenshot to the image recognition application, so that the image recognition application can continue to perform operations such as image recognition on the screenshot.
  • the processing capability of the display device 200 can be expanded, so that more map recognition functions can be realized through continuous update and maintenance of the map recognition application, and different user needs can be met.
  • the display device 200 may parse the associated data from the associated data in the step of rendering or presenting the recommended screen in the user interface according to the associated data. Identify target information.
  • the recognition target information includes target introduction text and target detail link, which are used to indicate the recognition object and recognition result in the image.
  • the display device 200 may add target introduction text and target detail links to the recommendation screen, so that users can learn about the recommended content they are interested in in time, and then select specific content.
  • the recommended content determined through image recognition can enable the user to obtain recommended content related to the image content without viewing specific media content.
  • the recommended content may include recommended media asset items and other recommended resource items, such as games, applications, web pages, text, and the like.
  • the recommended content can be obtained by matching in the resource database.
  • the recommended media asset items are obtained by matching in the media asset database based on the image recognition results. Therefore, when the media asset database has a large amount of resources and the image recognition results tend to be associated with popular resources, it can be found in the media asset database. A large number of media assets are matched. However, when there are a large number of media asset items, a larger content display window is required for display, which not only blocks the user interface, but also hinders users from selecting the media asset items they are interested in, reducing user experience.
  • a display device 200 is also provided.
  • the controller 250 of the display device may also be configured to execute a method for displaying recommended windows based on business scenarios, as shown in FIG. 16 , which specifically includes the following steps:
  • the display device 200 may respond to the image recognition instruction to detect the business scene to which the current user interface belongs.
  • the service scenarios may be different modes set in the control system or application according to resource types.
  • the display device 200 can provide different types and quantities of business scenarios.
  • the business scenarios may include: children's mode, education mode, film and television mode, entertainment mode and so on.
  • the display device 200 can recommend different resources for the user.
  • the display device 200 can only display sub-supplied media items, such as cartoons, children’s films, etc., in the media recommendation interface; in the education mode, the display device 200 can display the Science and education media projects, such as course videos, online live courses, popular science documentaries, etc.
  • the display device 200 will update the currently recorded service identifier after switching the mode on the user interface.
  • the current service scenario can be determined by detecting the service identifier.
  • the display device 200 may also send the service identifier to the server 400, so that the server 400 can determine the current service scenario by detecting the service identifier.
  • the acquisition of the business scene is not marked when the mode is switched, but the text and/or the name of the person in the screenshot is recognized, and then according to the recognized text and/or the name of the person, and the preset
  • the mapping relationship determines the business identifier that represents the current business scenario.
  • the mapping relationship is based on the mapping relationship between the name of the media resource resource, poster text, characters, etc., and the business scene corresponding to the media resource, which is established and stored in advance. Characters can be real or virtual.
  • the process of identifying the text and/or the name of the person in the image to determine the business scene may be to upload the business identification representing the business scene to the server 400 after the display device 200 is determined, or not to perform the business scene on the display device 200. Instead, the server 400 performs character recognition and business scene judgment based on uploaded screenshots.
  • the business scenario may be a channel theme corresponding to a resource type displayed on a face-to-face page.
  • the server 400 classifies the media resources according to their types and classifies them into different channel themes, and the user can select different channel themes to display different types of media resources.
  • the channel theme can be "children", "education", “shopping”, etc. as shown in Figure 5A or Figure 6, and the control displaying the channel theme can be called TAB bar.
  • the user controls the content area to display the content corresponding to the channel theme by manipulating the movement of the focus on different channel themes in the TAB column, and after displaying, the user can move the focus down to operate the controls in the content area.
  • the display device 200 will update the currently recorded service identifier.
  • the display device 200 may only provide the user with resource items suitable for the business scenario on a page presenting media resources.
  • the service scenarios that the display device 200 can provide may include a normal mode and an education mode.
  • the display device 200 can display comprehensive media items such as popular movies, TV dramas, and variety shows on the media item recommendation interface, and can also display recommended items according to the user's viewing history.
  • the display device 200 may display related course items in the media resource recommendation interface according to the course resources subscribed by the user.
  • the display device 200 may also classify business scenarios according to functions used by users. For example, some display devices 200 may provide a game mode. In the game mode, the display device 200 may display recommended game applications in an application list for users to choose to install and run.
  • the business scenarios that the display device 200 can provide are not limited to the above-mentioned children's mode, education mode, film and television mode, entertainment mode, regular mode, and game mode, etc.
  • the display device 200 can also provide other types of business scenarios, and in this business scenario, display resource items that match the current business scenario.
  • the display device 200 may detect the service scenario to which the current user interface belongs in various ways.
  • the display device 200 can monitor the user's operation of entering or exiting various business scenarios in real time, that is, obtain the control instruction input by the user for entering or exiting the service scenario, and respond to the control instruction, display the current Business scenarios are written into the system property database.
  • the user controls the focus cursor to move to the "mode switch” control on the control home page through the control device 100, and selects the "education mode” option in the pop-up mode selection window, then the display device 200 can be controlled to enter educational model.
  • the display device 200 can automatically record the current business scenario in the system attribute database as "education mode” by modifying the recording parameters in the system attribute database.
  • the display device 200 can mark various business scenarios by identifying character strings, and store them in the system attribute database.
  • a data table dedicated to recording business scenarios may be provided in the system attribute database, and the data table includes a type entry "Mode name". item to modify its corresponding value from "standard” for the general mode to "education” for the education mode.
  • the business scene to which the current user interface belongs is queried from the system attribute database. That is, after the display device 200 acquires the image recognition instruction input by the user, it reads the current state value of the type entry in the system attribute database in response to the image recognition instruction, so that the state value of the current type entry is read as "education”. , it is determined that the business scenario to which the current user interface belongs is "education mode”.
  • the display device 200 can also play the media item provided in the multimedia application by running the multimedia application.
  • Different multimedia asset applications can provide media asset items from different platforms and with different characteristics. Therefore, the service scenario can also be determined according to the running multimedia resource application.
  • the display device 200 can query the local database or the cloud server 400 according to the application name "AA-Early Education for Children" about the type of the application, that is, an educational application for children. Then, according to the application type obtained from the query, match specific business scenarios that conform to the scenario division method of the display device 200 . That is, the media resources provided by the "AA-Children's Early Education" application are of the same type as the children's mode, so it can be determined that the current service scenario of the display device 200 is the children's mode.
  • the current business scenarios can also be detected through the results reported by the business applications. That is, in some embodiments, in the step of detecting the business scene to which the current user interface belongs, the business application of the current user interface is invoked, and a scene report notification is sent to the business application, and then the business application returns the business scene for the scene report notification.
  • the display device 200 can also identify the business scenario it belongs to based on the content contained in the current user interface. That is, in the step of detecting the service scene to which the current user interface belongs, the display device 200 may first obtain the position of the focus cursor in the user interface, and then extract the name of the current focus channel according to the position of the focus cursor to determine the current service scene.
  • a scenario label can be set for each business scenario, and the display device 200 can identify various business scenarios according to the scenario tags.
  • the control homepage of the display device 200 may include multiple tabs, and each tab is provided with a name for marking the channel, including TV series, movies, documentaries, children, education, etc.
  • the display device 200 determines the name of the current focus channel by detecting the location of the focus mark, that is, the name corresponding to the tab where the focus mark is located is "kids", thereby determining that the current service scene is "kids mode".
  • the display device 200 will modify the names of the tabs when displaying some tabs. For example, the display device 200 may modify the tab name from "children" to "happy summer vacation” during part of the summer vacation. Therefore, in order to accurately determine the service scene, the display device 200 may call the standard service library after extracting the focus channel name, so as to match the service scene corresponding to the channel name through the standard service library. For example, the business scenario corresponding to "happy summer vacation" can be matched as "children's mode" in the standard business library. Wherein, a standard identification code of each service scenario and a channel name used to represent the service scenario may be recorded in the standard service library. The standard business library can follow the UI update policy of the operating system to update records in real time.
  • the display device 200 can generate an image recognition request corresponding to the target image to be recognized according to the detected current business scene and the map recognition instruction, and send the image recognition request to the server 400, so that the server 400 can respond to the image recognition request Feedback associated data.
  • the associated data includes hot search text and recommended items.
  • the recommended items are items that meet business scenarios and are obtained by querying the resource database according to the hot search text.
  • generating an image recognition request includes an image to be recognized (also referred to as a target image, for example, a screenshot) and a scene identifier representing a business scene. Screenshots can be used to provide feedback on screenshot results, and scene identification is used to obtain hot search text.
  • generating the image recognition request includes a screenshot and does not include a scene identifier representing a business scene, where the screenshot is used for feedback of a result of the screenshot, and the current business scene is determined according to the result of the view.
  • the server 400 may first perform image recognition on the target image to obtain the hot search text.
  • the server 400 may perform image recognition processing on the target image according to an image recognition algorithm, so as to identify a specific target from the target image.
  • the image recognition process is the same as above, and will not be repeated here.
  • the server 400 may input the target image into the image recognition model to obtain the image recognition result.
  • the image recognition result includes keywords. Then, by extracting the hot search thesaurus in the business scenario, and matching the hot search text associated with the keyword in the hot search thesaurus.
  • different business scenarios correspond to different hot search word bases.
  • the server 400 may maintain a resource item database, and the resource item database may store a hot search word base based on network or local search engine statistics, that is, a set of words with high search frequency by users. All media asset items in the current platform can also be stored in the resource item database. After the word "variety show” is recognized in the image through the image recognition operation, the server 400 can match the hot search text related to "variety show" in the hot search word database, such as "Come on ⁇ ", "Run ⁇ " and so on.
  • the server 400 may feed back the hot search text to the display device 200 .
  • the display device 200 then renders or presents a recommendation window according to the fed back hot search text.
  • the hot search text can be used as a quick search item, which is used to search for related resource items using the selected hot search text as a keyword after the user selects it.
  • the server 400 also feeds back the results of image recognition and image recognition recommendations.
  • the display device 200 receives image recognition results and image recognition recommendations fed back by the server 400, as well as hot search word results and hot search recommendations.
  • the display in the video layer continues, that is, if there is a video to play, it will continue to play, if there is a carousel picture to continue to carousel, all the display of the picture will be carried out according to the original logic of the screenshot interface, and will not be displayed. interrupt.
  • the results fed back by the server 400 are displayed.
  • the picture recognition title control is generated according to the picture recognition result and the picture recognition recommendation
  • the hot search word title control is generated according to the hot search word result and the hot search recommendation.
  • a screenshot is displayed in the middle of the content area in the floating layer, the person recognized by the picture recognition is displayed on one side of the screenshot, and the recommendation is displayed on the other side (similar to Figure 6) .
  • the search box and the hot search word controls are displayed in the middle position, and the hot search recommendations are displayed on both sides of the middle position.
  • the middle position is switched, and one side of the middle position displays the characters for image recognition feedback, and the other side displays recommendations.
  • the middle position of the content area in the floating layer displays screenshots.
  • the focus is on the hot search word title control, and the search box and hot search word control are displayed in the middle.
  • the display device 200 may send a search request to the server 400 according to the hot search text selected by the user.
  • the server 400 can parse the hot search text in the search request, and query the resource database for items matching the business scenario according to the hot search text, so as to obtain recommended items.
  • the server 400 can analyze the target image, recognize the "car” target in the image, and match the hot search texts "speed and ⁇ " and " ⁇ flying car” through the hot search lexicon, And feed back to the display device 200 .
  • the display device 200 then renders or presents a recommendation window according to the hot search texts "speed and XX” and “XX speed", and the recommendation window includes options of "speed and XX” and "XX speed”.
  • the display device 200 sends a search request containing the selected hot search text to the server 400 .
  • the server 400 searches the resource subset corresponding to the current video service scene for recommended items associated with "speed and ⁇ ", that is, movie resources such as "speed and ⁇ 8", "speed and ⁇ 10" .
  • the recommended items queried by the server 400 according to the hot search text are different.
  • the hot search text and recommended items are movie information related to cars, that is, the hot search text is "speed and ⁇ ", " ⁇ speed”, etc.
  • the name of the movie, and the recommended items are movie resources such as "Speed and ⁇ ", " ⁇ Speed”.
  • the hot search texts and recommended items are car-related cartoons or teaching videos. That is, as shown in FIG. 20B , the hot search terms are cartoon titles such as "Car Town” and “Cars”, and the recommended items are cartoon resources such as “Car Town” and "Cars”.
  • the recommended hot search texts and recommended items are game resources related to cars, that is, as shown in Figure 20C, the hot search texts are game names such as "XX Kart" and "XX Speed".
  • the content of the game is also a car-related game.
  • the recommendation window may be an independent window suspended and displayed on the upper layer of the user interface, or may be a new interface after the display device 200 jumps. Options generated based on popular search text and recommended item links may be included in the recommendation window.
  • the user can perform interactive operations through the recommendation window, control the display device 200 to perform operations such as selecting and previewing based on the recommended items, and control the display device 200 to perform operations such as playing, browsing, and jumping based on the recommended items in the recommendation window.
  • the recommendation window may include multiple areas for displaying different content. Operations on the recommendation window and the recommendation window are similar to the above, and will not be repeated here.
  • the display device 200 may create a display layer above the layer where the user interface is located, and acquire hot search text and recommended items. Then call the display template of the recommendation window to add hot search text and recommended items to the display template to form a recommendation screen. Finally, display the recommended screen in the display layer. For example, after performing a search operation, the display device 200 may display a search result interface on the OSD layer of the current user interface.
  • the display device 200 may perform a search within the current business scenario, so as to search for resource items associated with the selected hot search text from the resource subset corresponding to the current business scenario. For example, after the user selects the hot search text "speed and ⁇ ", the display device 200 can send the search word containing the text "speed and ⁇ " to the server 400, so that the server 400 can match the search terms with "speed and ⁇ " on the current media platform. ⁇ ” related media asset items, and then feed it back to the display device 200, so as to render a display window on the OSD layer of the display device 200 to display the matching recommended items, as shown in FIG. 21 .
  • the display device 200 may first search in the current business scene and display the search results.
  • the search results in the current business scenario are displayed for a certain period of time, or after the user enters a command to search the entire network, the selected hot search text is used as the search term to search the entire network, and the search results of the entire network are presented.
  • the display device 200 may display recommended items in the recommended item area.
  • the displayed recommended items can be changed in real time according to the user's interactive operation. For example, after the user clicks on the hot search text option of "speed and ⁇ ", the display device 200 can obtain information related to " The recommended items associated with "Speed and ⁇ ", that is, “Speed and ⁇ 1", “Speed and ⁇ 8" and other media asset items, are displayed in the recommended item area.
  • the display device 200 may also display initial recommendation content in the recommended item area when rendering or presenting the recommendation window.
  • the initial recommended content may be a recommended item queried by the server 400 according to the matching hot search text when performing image recognition. That is, in some embodiments, after the server 400 performs image recognition to obtain the hot search text, according to the hot search text, it queries the resource database for recommended items that meet the business scenarios, and forms the associated data together with the recommended items and the hot search text. , to feed back to the display device 200. The display device 200 then renders a recommendation window in the user interface according to the associated data.
  • the server 400 After the server 400 recognizes the car object in the image through the image recognition operation, it matches the hot search texts corresponding to the car object in the children's mode as “Car Town” and "Car Story”. The server 400 then matches the video content related to "Car Town” and "Car Story” in the resource item database corresponding to the children's mode according to the hot search text, that is, "Car Town” and "Car Story", and extracts the corresponding media address. Therefore, the server 400 can combine hot search texts such as “Car Town” and “Cars” and media resource addresses of "Car Town” and “Cars” into associated data, and feed it back to the display device 200 . Therefore, when the recommendation window is displayed, the media asset items of "Car Town” and “Cars” can be displayed in the recommended item area.
  • the display device 200 can use the detected current business scene to filter the hot search text and recommended items when using the image recognition function, so that the display in the recommendation window is more in line with the user's current situation.
  • the hot search items and recommended resources that are required, simplify the resource items in the recommendation window, and facilitate user operations.
  • the target image of the picture recognition function may be a specific picture file, or a specific picture in the current user interface.
  • the display device 200 may adopt different program steps. That is, as shown in FIG. 22 , in some embodiments, the display device 200 may detect the target image specified in the image recognition instruction after acquiring the image recognition instruction input by the user. If the specific image file indicated by the target image is specified in the image recognition instruction, it is determined that the user is performing image recognition operation on a specific image file, so the image can be directly extracted to generate an image recognition request and sent to the server 400 for image recognition processing.
  • the display device 200 can use the image recognition function for the current display content by default, so it can generate a screenshot command and broadcast the screenshot command to the service operating system. After receiving the screenshot command, the service operating system may respond to the screenshot command and perform a screenshot operation on the current user interface to generate a target image.
  • the user when the user sees the poster or cover of some movie on the current user interface, he may be interested in the movie or actor, or want to know the detailed information corresponding to the movie.
  • the user can press the image recognition button on the control device 100, and since there is no specific image file, the display device 200 can generate a screenshot command, and take a screenshot of the current user interface according to the screenshot command to generate a target image.
  • the image recognition request is sent to the server 400 to perform recognition processing on the image through image recognition technology, so as to recognize related information such as movie name and actor name from the image.
  • the display device 200 may filter the hot search texts after receiving the associated data. That is, when the display device 200 renders or presents a recommendation window in the user interface according to the associated data, it may traverse the hot search texts in the associated data to filter out text sets that meet the business scenario, and then generate hot search options according to the text sets.
  • each hot search option includes at least one hot search text that meets the business scenario, so as to add the hot search option in the recommendation window.
  • the recommended items in the recommendation window can also be sorted according to usage popularity, so that the user can select popular item content. That is, when the recommendation window is rendered in the user interface according to the associated data, the display device 200 can traverse the recommended items in the associated data, and then query the usage popularity of each item according to the name of the recommended item, so that the usage popularity is in descending order to add the recommended item to the recommendation window.
  • a server 400 is also provided, and the server 400 includes: a storage unit, a communication unit, and a processing unit.
  • the storage unit is configured to store the data of media asset items;
  • the communication unit is configured to connect to the display device 200; as shown in FIG. 23 , the processing unit is configured to execute the following program steps:
  • the display device feed back the associated data to the display device, so that the display device renders or presents a recommendation screen according to the associated data, where the recommendation screen includes options generated based at least on the hot search text and/or recommended link .
  • the associated data includes hot search text and/or recommended link;
  • the hot search text is the text associated with the recognition result;
  • the recommended link is the media associated with the recognition result address and/or web page address.
  • the server 400 provided in this embodiment can perform image recognition processing on the image to be recognized after acquiring the image to be recognized sent by the display device 200, and identify a specific target or text from the image to be recognized, thereby extracting the image to be recognized.
  • the hot search text and recommended link matching the image content are fed back to the display device 200, so that a recommended screen is rendered or presented on the user interface of the display device 200 for the user to perform operations.
  • the server 400 can perform recognition processing on the image to be recognized, so the data processing amount of the display device 200 can be reduced. Moreover, by using the server 400 to maintain various types of image recognition models, different recognition needs can be met to obtain more types of associated data.
  • the server 400 may input the image to be recognized into the recognition model, and The target information output by the recognition model is acquired.
  • the target information includes the type code output by the feature target recognition model and the keyword output by the character recognition model.
  • the built-in recognition model in the server 400 may include a recognition model obtained based on machine learning algorithm training, that is, the artificial intelligence model obtained after training a large number of training sample images can include specific targets in the output image after the image to be recognized is input, such as people, scenery, etc. etc., the classification probabilities of .
  • the recognition module can also include a character recognition model based on Optical Character Recognition (OCR) technology, after the image to be recognized is input, the recognition model can determine its shape by detecting dark and bright patterns in the image, and then recognize The text information contained in the image.
  • OCR Optical Character Recognition
  • the server 400 may copy the image to be recognized according to the number of built-in recognition models, so as to obtain multiple images and input them into the recognition models respectively.
  • Different recognition models can output the classification probability of a specific target based on the same image content, as well as recognize text in the image to obtain target information.
  • the server 400 can also extract the same type of media asset address and/or web page address from the media asset item data according to the type code to obtain a recommended link; Match synonyms in to get associated text.
  • the image to be recognized can be transmitted to the server 400 to trigger the server 400 to perform character (person or object) recognition on the image.
  • the server 400 can identify
  • the target information is obtained
  • the identified character information can be digitized according to the rules agreed upon with the display device 200, that is, the type data is formed according to the agreement, such as "TYPE: 1" representing a sports star, etc., and then the digitized information
  • the image recognition application performs hot search word matching based on the content identified in the current picture, determine the hot search word corresponding to the identified person, and send it to the display device 200 below.
  • the server 400 may also obtain layout information of the current user interface on the display device 200 .
  • the server 400 may send a detection request to the display device 200 to trigger the display device 200 to upload the current layout information to the server 400 .
  • the layout information may include the shape, size, resolution, etc. of the currently recommended screen.
  • the server 400 may calculate the number of options in the recommendation screen according to the layout information uploaded by the display device 200 . Finally, the associated data corresponding to the number of options is fed back to the display device 200 according to the calculated number of options. For example, when the current recommendation screen is obtained as a strip-shaped area at the bottom of the user interface, the server 400 may determine according to the width and height of both sides of the strip-shaped area that six recommended media asset options can be displayed in the current recommendation screen, namely, the left side 3, 3 media asset display positions on the right. Therefore, the associated data that the server 400 may feed back to the display device 200 according to the calculated number of options may include six recommended media resource links.
  • a server 400 is also provided. As shown in Figure 25, the processing unit is configured to perform the following procedural steps:
  • the associated data is fed back to the display device, so that the display device renders a recommendation window in the user interface according to the associated data.
  • the server 400 can perform image recognition on the target image to obtain the hot search text after obtaining the image recognition request sent by the display device 200, and then search the resource database for matching services according to the hot search text. Items of the scene to obtain recommended items; thereby generating associated data and feeding back the associated data to the display device 200, so that the display device 200 renders or presents a recommendation window in the user interface according to the associated data.
  • the server 400 can obtain corresponding hot search texts and recommended items according to different business scenarios on the basis of image recognition, so that the items in the recommendation window rendered or presented by the display device 200 can adapt to the current business scenario , reduce redundant information, and improve user experience in this business scenario.
  • a method for recommending media asset content including the following steps:
  • the display device 200 acquires the image recognition instruction input by the user, and sends an image recognition request to the server 400 in response to the image recognition instruction, and the request includes the image to be recognized;
  • the server 400 performs image recognition on the image to be recognized, so as to extract associated data from the media asset item data according to the recognition result of the image recognition, and the associated data includes hot search text and/or recommended links ;
  • the hot search text is the text associated with the recognition result;
  • the recommended link is the media asset address and/or web page address associated with the recognition result;
  • the display device 200 renders or presents a recommendation screen in the user interface according to the associated data fed back by the server 400 , and the recommendation screen includes options generated based on the hot search text and/or the recommended link.
  • the image recognition scene of the image recognition application in the display device 200 is triggered.
  • the image recognition application will sense the screenshot event first, and then broadcast a screenshot command to the business operating system of the display device 200 .
  • the service operating system of the display device 200 will execute a screenshot operation to obtain a screenshot image.
  • the display device 200 transmits the screenshot image to the image recognition application.
  • the image recognition application transmits the screenshot to the server 400 for person identification.
  • the server 400 extracts the associated data according to the recognition result, and then returns the associated data to the image recognition application, and recognizes the content according to the current image, and distributes the hot search words.
  • the map recognition application After receiving the data returned by the server 400, the map recognition application presents the data to the terminal user through different UI interfaces according to the returned type. At the same time, after receiving the current keyword identified by the cloud, go to the media resource database to request relevant hot search data and popular media resources. Then, the media resource library performs data query according to the parameters of the display device 200, and returns the result to the display device 200. After receiving the returned result, the display device 200 renders and displays the data.
  • the media asset content recommendation method provided by the present application can send the image to be recognized to the server 400 after the user inputs an image recognition command, so that the server 400 can perform image recognition on the image to be recognized, and generate an association according to the recognition result.
  • the data The display device 200 then renders or presents a recommendation screen according to the associated data, so as to display a recommendation interface including hot search text and/or recommendation link options in the user interface.
  • the method can feed back related text and/or recommended links based on the recognition result of the image to be recognized, and increase the type of options in the recommendation screen, so that the user can select different related items according to requirements, and improve user experience.

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • User Interface Of Digital Computer (AREA)

Abstract

L'invention concerne un dispositif d'affichage (200), un serveur (400) et un procédé de recommandation de contenu multimédia. Le dispositif d'affichage comprend un écran (260), qui est conçu pour afficher une interface utilisateur ; et un dispositif de commande (250), qui est conçu pour : acquérir une instruction de reconnaissance d'image entrée par un utilisateur ; en réponse à l'instruction de reconnaissance d'image, envoyer une requête de reconnaissance d'image à un serveur, la requête comprenant une image à reconnaître ; recevoir des données qui sont associées à l'image et renvoyées par le serveur, les données associées comprenant au moins un texte d'activation de recherche, et le texte d'activation de recherche étant un texte associé à un résultat de reconnaissance obtenu au moyen de la réalisation d'une reconnaissance d'image sur l'image à reconnaître ; et afficher un écran de recommandation dans l'interface utilisateur en fonction des données associées, l'écran de recommandation comprenant une option générée au moins sur la base du texte d'activation de recherche afin de demander une recherche associée à l'image.
PCT/CN2022/103154 2021-07-23 2022-06-30 Dispositif d'affichage et procédé de recommandation de contenu multimédia WO2023000950A1 (fr)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202280049050.1A CN117643061A (zh) 2021-07-23 2022-06-30 显示设备及媒资内容推荐方法

Applications Claiming Priority (4)

Application Number Priority Date Filing Date Title
CN202110836063.0 2021-07-23
CN202110836063.0A CN115695844A (zh) 2021-07-23 2021-07-23 一种显示设备、服务器及媒资内容推荐方法
CN202111120100.4A CN115866313A (zh) 2021-09-24 2021-09-24 显示设备、服务器及推荐窗口显示方法
CN202111120100.4 2021-09-24

Publications (1)

Publication Number Publication Date
WO2023000950A1 true WO2023000950A1 (fr) 2023-01-26

Family

ID=84978970

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2022/103154 WO2023000950A1 (fr) 2021-07-23 2022-06-30 Dispositif d'affichage et procédé de recommandation de contenu multimédia

Country Status (2)

Country Link
CN (1) CN117643061A (fr)
WO (1) WO2023000950A1 (fr)

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103064863A (zh) * 2011-10-24 2013-04-24 北京百度网讯科技有限公司 一种提供推荐信息的方法与设备
US20150220543A1 (en) * 2009-08-24 2015-08-06 Google Inc. Relevance-based image selection
CN108259973A (zh) * 2017-12-20 2018-07-06 青岛海信电器股份有限公司 智能电视及电视画面截图的图形用户界面的显示方法
CN113094521A (zh) * 2021-03-12 2021-07-09 北京达佳互联信息技术有限公司 一种多媒体资源搜索方法、装置、系统、设备及存储介质
CN113111286A (zh) * 2021-05-12 2021-07-13 北京字节跳动网络技术有限公司 一种信息展示的方法、装置以及计算机存储介质

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20150220543A1 (en) * 2009-08-24 2015-08-06 Google Inc. Relevance-based image selection
CN103064863A (zh) * 2011-10-24 2013-04-24 北京百度网讯科技有限公司 一种提供推荐信息的方法与设备
CN108259973A (zh) * 2017-12-20 2018-07-06 青岛海信电器股份有限公司 智能电视及电视画面截图的图形用户界面的显示方法
CN113094521A (zh) * 2021-03-12 2021-07-09 北京达佳互联信息技术有限公司 一种多媒体资源搜索方法、装置、系统、设备及存储介质
CN113111286A (zh) * 2021-05-12 2021-07-13 北京字节跳动网络技术有限公司 一种信息展示的方法、装置以及计算机存储介质

Also Published As

Publication number Publication date
CN117643061A (zh) 2024-03-01

Similar Documents

Publication Publication Date Title
US20180152767A1 (en) Providing related objects during playback of video data
US11972099B2 (en) Machine learning in video classification with playback highlighting
WO2020000973A1 (fr) Procédé d'accès à des informations, client, appareil d'accès à des informations, terminal, serveur, et support de stockage
US20150293928A1 (en) Systems and Methods for Generating Personalized Video Playlists
US20210392403A1 (en) Smart Television And Server
US10158920B2 (en) Interaction system and interaction method thereof
US11924513B2 (en) Display apparatus and method for display user interface
US20170235828A1 (en) Text Digest Generation For Searching Multiple Video Streams
CN112000820A (zh) 一种媒资推荐方法及显示设备
WO2022012271A1 (fr) Dispositif d'affichage et serveur
CN116261857A (zh) 一种显示设备及应用程序界面显示方法
CN111625716A (zh) 媒资推荐方法、服务器及显示设备
WO2022078172A1 (fr) Dispositif d'affichage et procédé d'affichage de contenu
WO2023000950A1 (fr) Dispositif d'affichage et procédé de recommandation de contenu multimédia
WO2022083554A1 (fr) Agencement et procédé d'interaction d'interface utilisateur, et dispositif d'affichage tridimensionnel
US20220329908A1 (en) Display Apparatus and Method for Displaying Image Recognition Result
CN115695844A (zh) 一种显示设备、服务器及媒资内容推荐方法
CN113722542A (zh) 视频推荐方法及显示设备
WO2022012299A1 (fr) Dispositif d'affichage et procédé de reconnaissance et de présentation de personnes
US11997341B2 (en) Display apparatus and method for person recognition and presentation
CN115866313A (zh) 显示设备、服务器及推荐窗口显示方法
CN117812377A (zh) 一种显示设备及智能剪辑方法
CN114168765A (zh) 服务器和媒资标签获取方法
CN117806747A (zh) 显示设备及显示设备的屏保跳转方法
CN114302242A (zh) 一种媒资推荐方法、显示设备及服务器

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 22845115

Country of ref document: EP

Kind code of ref document: A1

WWE Wipo information: entry into national phase

Ref document number: 202280049050.1

Country of ref document: CN

NENP Non-entry into the national phase

Ref country code: DE