WO2016125166A1

WO2016125166A1 - Systems and methods for analyzing video and making recommendations

Info

Publication number: WO2016125166A1
Application number: PCT/IL2016/050131
Authority: WO
Inventors: Dan RON
Original assignee: Ankan Consulting Ltd.
Priority date: 2015-02-05
Filing date: 2016-02-04
Publication date: 2016-08-11

Abstract

Video segments are tagged with one or more tags. After a user finishes to watch a video segment a recommendation is issued to watch a follow-up video segment. The recommendation is based on the tags of the first video segment and optionally available user profile data.

Description

SYSTEMS AND METHODS FOR ANALYZING VIDEO AND MAKING RECOMMENDATIONS

BACKGROUND

There are many advertising approaches currently used by the entities that provide services or products for the purpose of promoting their contents. Among the most accepted techniques is an advertising via newspapers, magazines, television and radio. But in the modern times promoting content on the Internet is probably the most popular technique. The promotional messages that are being delivered via the Internet use email, social media, sponsored content, mobile advertising etc.

An advertiser provides the content or service, which the publisher puts on its website for advertisement. The main goal of every advertiser is to find a user that may become a consumer of the proposed content. By collecting relevant information about a user, from the browsing history and searches that were conducted by the user to provided user profile, the advertiser can better predict the user's taste and needs and offer said user a content that might be of interest to him. These recommendation mechanisms are extensively used in recent years and have a lot of applications such as books, music, movies, financial or insurance services, dating or food industry. Currently the recommendation mechanisms are adjusted to text based analysis and further recommending a relevant content to a user.

In spite of the very extensive use of the recommending mechanisms, such

mechanisms currently apply mainly to text based content. There are no similar mechanisms for video-based contents that are successful today. The conversion rates, that refers to a percentage of users that choose to take the desired action offered to them by an advertiser such as subscribing, downloading content or purchasing product are still very low for video-based contents and there is a constant need for improving the conversion rates of video-based content.

For video-based content, the used methods today, for example, in YouTube™ include displaying small videos of similar content next to the video being watch hoping the user will choose one of them or automatically starting a new video segment after a first video segment ends. Both of these methods yield a very small conversion rate (percentage of users who watch a second video after the first one), typically below 1%.

SUMMARY

It's an object of the present invention to provide content recommendation for a user watching a video.

It's another object of the present invention to provide content recommendations based on analysis of a video being watched by a user.

The present invention relates to a computing system comprising at least one processor; and at least one memory communicatively coupled to the at least one processor comprising computer-readable instructions that when executed by the at least one processor cause the computing system to implement a method of

recommending video content, the method comprising: displaying a first video segment; and recommending a second video segment for viewing based on one or more tags associated with said first video.

In some embodiments, the method comprises an initial step of tagging a video segment with one or more tags associated with the content of the video segment.

In some embodiments, recommending the second video segment is done via a video segment, graphics, text, voice, audio or any combination thereof.

In some embodiments, the tagging of a video segment is done by the video segment creator, by a computerized process, by a designated person, by one or more viewers or any combination thereof.

In some embodiments, a recommendation for viewing a second video segment is created automatically after the identification of one or more tags in said first video segment.

In some embodiments, the method further comprises a step of preparing in advance a recommendation for viewing a second video segment based on one or more tags of a first video segment.

In some embodiments, recommending the second video segment is based also on user profile information of the first video viewer. In some embodiments, the user profile information comprises: a user's Internet navigation history, user's preferences, history of watched video segments, life style, hobbies, socio-economic data about the user, purchasing history, user's social media profile or any combination thereof.

In some embodiments, the one or more tags relate to: a person, an object, a

geographic location, an artistic work, a sports entity, a hobby, a genre, age, sex, mood, emotion, physical trait or any combination thereof.

In some embodiments, the second video segment is an advertisement or a promoted content.

In some embodiments, recommending a second video segment is done by displaying a recommendation about said second video segment while said first video segment is being watched.

In some embodiments, the displayed recommendation is a pop-up, a separate display aside the first video segment, a virtual layer integrated with the first video segment, or an integral part of said first video segment.

In some embodiments, the recommendation about said second video segment comprises video, text, graphics, an image, a link or any combination thereof.

In some embodiments, after recommending said second video segment, said second video segment is displayed automatically.

In another aspect, the present invention further relates to a computer system

comprising a processor; and a memory communicatively coupled to the processor comprising computer-readable instructions that when executed by the processor cause the computer system to: display a first video segment; and recommend a second video segment for viewing based on one or more tags associated with said first video.

In some embodiments, all users or authorized users can enter reviews about a video segment. The review can include rating and opinions regarding the quality of the recommendation, video image quality, abusive content etc. BRIEF DESCRIPTION OF THE DRAWINGS

Fig. 1 shows a flowchart illustrating an embodiment of a method for recommending video content to a user.

Figs. 2-7 are screenshots of Google Analytics ^tm illustrating a test case of actual videos promoted via recommendations of the invention. The interactions described are between a video and the following recommendation video, and then between a recommendation video and the suggested video, and onward in the same manner. All the numbers represent sessions (views).

DETAILED DESCRIPTION

The present invention relates to methods and systems for analyzing a video segment and recommending a relevant following video segment that may be of interest to the user.

The present invention relates to a computing system comprising at least one processor; and at least one memory communicatively coupled to the at least one processor comprising computer-readable instructions that when executed by the at least one processor cause the computing system to implement a method of recommending video content, the method comprising:

displaying a first video segment; and recommending a second video segment for viewing based on one or more tags associated with said first video. For the sake of clarity, the description uses the terms "first video" and "second video", but it should be understood that the system will keep recommending a next video to watch after each video watched (or during the watched video), so a user may thus watch any number of recommended videos sequentially, one after the other, without any limit. The recommended video segment is typically of a content identified to be of interest to the viewer, though it may also be an advertisement or a promoted content in order to generate a revenue stream for the owner of the site where the video is displayed.

Typically, an initial step is performed of tagging each video segment with appropriate tags. The tag or tags for each video segment can be assigned by different sources such as the video segment creator, by a computerized process, by a designated person, by one or more viewers or any combination thereof. Tags can relate to an infinite amount of subjects. In some embodiments, a tag can relate to a person, an object, a geographic location, an artistic work, a sports entity, a hobby, a genre, age, sex, mood, emotion, physical trait or any combination thereof.

The computerized process for tagging a video segment can use several methods of video analysis comprising: identifying logos, products, objects, moving objects, face recognition, speech recognition, text recognition, use of tags or meta information provided by the video creator, human analysis by an expert or viewers or a designated person, music recognition, information about the creator of the video segment, information about the participants of the video segments, location identification, broadcasting system\source, date published or any combination thereof.

In some embodiments, tags can be assigned weights and can be ranked, such that a video segment may be assigned several tags, but some tags are marked as more important or relevant. For example, tags can be ranked from 1 to 5 where 5 is very relevant, 3 is relevant and 1 is somehow relevant. Alternatively, tags can ranked as percentages where the same tag can be 90% relevant for one video segment but only 60% relevant for another video segment.

There are different ways to identify objects in a video segment. Any number of frames (for example, every 10^th, 20^th, 30^th frame) can be analyzed using image recognition techniques to identify the image contents. Another way to identify objects of importance in the video segment is to analyze a frame sequence (a predetermined time length, for example, 10, 20 or 30 seconds) by an algorithm and identify objects that move between the frames. There are several algorithms, each may be useful for a different kind of objects. Face recognition can be performed by an algorithm that analyzes the features of the face and searches the image that will match the database. The images are also being searched and matched to the ones in the image database.

In another embodiment, another way of analyzing the video file is by extracting features. Among features the analyzing method can identify colors, texture properties such as contrast, regularity, roughness and two dimensional shapes.

In another embodiment, other media components such as text or audio also carry valuable information. Text identification mechanisms are already extensively used in current recommendation mechanisms. The text identified in the video segment may appear as either a scene text or as superimposed text and it may appear in different backgrounds and font sizes. Audio files are also easily fragmented and analyzed according to well established methods.

In yet another embodiment, the video content analysis may involve one or more of the above mentioned approaches or any combination of them.

The term "video segment", as used herein, refers to any video content which may be watched or broadcasted on the internet, television, cinema, mobile phone or any kind of computer or screen such as laptop, tablet , notebook, home or public screen, smart\cellphone, smart television, ipad and ipod.

In some embodiments, tags can be ranked according to the order of relevance to a specific viewer. In this case ranking can be first determined according to the analysis performed on the video segment independently. Next, the tag ranking can be adjusted according to a specific viewer user profile. User profile information can include any relevant information about the user, provided by the user and/or gathered from different sources. User profile information can include a user's Internet navigation history, user's preferences, history of watched video segments, life style, hobbies, socio-economic data about the user, purchasing history, user's social media profile or any combination thereof. Tags can then be selected and ranked according to their relevance to the specific viewer of a video segment. The recommended video segment with the most relevant tags to a specific viewer will have a high probability of matching the viewer's preferences or interests. If the system has access to the viewer's watched video segments history, the system makes sure not to recommend a video segment that the viewer has already watched in the past.

For example, if a viewer was interested in the past in purchasing an item and that item was recognized in the video segment watched by the viewer and being analyzed by the method of the present application, the recommendation mechanism of the present invention can recommend the viewer to watch a second video segment that is relevant to the identified item, with good chances of matching the user's interests with that recommendation . In some embodiments, the recommendation mechanism may offer a video

recommendation to a single person or to a group of people. In group recommendation, the recommended second video segment to watch reflects preferences of the group.

The recommendation of a follow-up, second video to watch can be right after the user has finished watching a video segment (post-roll) and / or during the watched video (mid-roll). In the second option, mid-roll, while a user is watching a video, the system can recommend a follow-up video to watch a via pop-up, a separate display aside the first video segment, a virtual layer integrated with the first video segment.

Alternatively, the recommendation to watch a follow-up video can be an integral part of the watched video segment, for example, as part of the end sequence of the video segment. The pop-up or separate display for the recommendation can also happen for a post-roll recommendation. In some embodiments, the watched video is reduced in size on the screen at the end of the video segment, and a second window appears with a recommendation.

The recommendation to watch a follow-up video can be via video, text, graphics, an image, a link or any combination thereof. In some embodiments, after watching a recommendation, the recommended video is played automatically. In some embodiments, the viewer has to actively select to play the recommended video.

The recommended follow-up video may either be prepared in advanced and extracted from a video database, or alternatively can be created on-the-fly.

The system can produce a "recommendation segment\promotion". For example, if a person was identified in the watched content then the system can extract (edit and cut) the part of the video where the person appeared, and add to it voiceover and/or a text of the recommendation.

In another example, if a location was identified then graphics presenting the specific location with the recommendation text and voiceover can be produced.

In some embodiments, the recommended follow-up video to watch may be an advertisement related to the video segment being watched optionally in combination with user profile information. For example, if a viewer watches a video about surfing, the recommended video may be a promotional message from a surf board

manufacturer. As a result, more targeted recommendation mechanism results in increasing the number of user's that might be interested in the products or services or content offered to them by an advertiser or publisher. An advertiser\publisher may also track the conversion rate, which refers to the percentage of users that choose to take the desired action offered to them by an advertiser or publisher such as subscribing, downloading content or purchasing product. Due to the method described in the present invention the conversion rates shall improve due to the fact that said method will allow to match relevant contents in a much more accurate way to the potential consumers.

The term "viewer", as used herein, refers to any individual or collection of individuals or a business entity that watches the video segment that is further analyzed by the method of the present application.

The recommendation system may be very useful to any user either an individual or a business entity that would like to watch relevant contents or services; said system may also offer recommendations for new videos recently published while the user was offline or turned off its device. Users may also benefit from the system of the invention and improve their watching satisfaction since the system shall provide recommendations for content based on the identified interests (tags) which relate to the recently watched video.

The recommendation system of the invention may be very useful to an advertiser that may be an individual or a business entity that would like to advertise its contents or services; such system may also interest a publisher that would like to employ its website and to sell an advertising space. Advertising agencies may also benefit from the system to improve their marketing and remarketing strategies since they provide a service of creating and planning advertising strategy for their clients.

The suitable advertising platforms may be content or service sites, social networks, blogs, e-mails, websites and mobile sites and applications, SMS, search engines. The recommended video content may appear as a pop-up or a pop under, floating ad, display advertising. Such content may appear while the user is watching the video or right after the user finished watching the video.

Fig. 1 is a flowchart illustrating the sequence of steps of the method of recommending video segments. In the first step (10) a video segment is a tagged with one or more tags that are relevant to its content. Next in step (20) the first video is displayed to a user (viewer). At the end of the first video (post-roll) or during the first video (mid- roll), in step (30) a recommendation to watch a second video is displayed. In step (40) the second video is displayed. In step (50) a follow-up (now third) video segment is recommended to the user. The method will keep recommending a video to watch after each video watched for as long as the user engages with the video system.

Fig. 2 describes a method of elements classification which may be based on any combination of the 3 inputs. One involves elements classification based on the analysis of the video segment independently (100). A second input is based on the analysis of the data obtained from the user's profile (110). A third approach relates to analyzing the user history (content watched, sites visited, items purchased etc.) (120). In step (130) the elements are classified according to any combination of the previous steps 100, 110 and/or 120.

Examples:

1. A user watches a video file with an image of Honda car. The system retrieves a new video file recommending watching a follow-up video with an advertisement of a Honda.

2. A user watches a video file with a famous actor. The system retrieves a new video file with the same actor recommending the user to watch a second video with the actor in it.

3. A user watches a video file of a sports game. The system automatically

retrieves from the watched video file a short segment and adds to it a voice over and/or a text recommending an advertisement related to the watched video.

4. A user watches a video file with a funny movement from an identified creator.

The system automatically retrieves a new video file recommending additional video content posted by the same creator.

5. A user watches a video file presenting an exotic vacation location. The system recommends watching a video with another exotic vacation location.

Test Case Results

Six video segments from YouTube ^tm were selected. Each video is identified by a number: Video 1951 (Ultimate crashes 2015); 1903 (Top 10 Bruce Lee moments); 2035 (Very funny videos 2015 HD); 20 (Lionel Messi); 2061 (Only the best videos of youtube 2014. All in one). The videos were available in the website "http://star- vdo.com/haixi/player/2061" (2061 should be replaced with 1951, 1903, 2035, 20 and 2061 for the other videos). The viewing numbers relate to a period of approximately 45 days between 15.12.15 and 31.1.16. The initial video segments were promoted via Google Adwords to attract initial traffic and see how much of this initial traffic will continue to watch videos based on a presenter recommending to watch the follow-up video.

After each video was seen, a short video followed with a presenter recommending watching a follow-up video. The following tables demonstrates the for each video, how many people watched and how many followed the recommendation to the following and following videos, up to the 8^th video!

Table 1 - Conversion Rates from 1^st to 2ⁿ Video

Table 1 shows the conversion rates for the 6 videos. For example, for video number 1951, 1,310 people watched it (sessions) and 75% of them (through %) continued to watch recommendation (Rec. #) number 1222, a video segment displayed when video 1951 finished playing. It is to be noted that since the recommendations were independent video segments, in this case, some viewers arrived directly to the recommendation video without watching the previous video. In this case, the number of sessions for a recommendation 1222 was actually 1,330 composed of 982 people who watch video 1951 (75% of total 1310 sessions) and an additional 348 people who arrived directly to watch the recommendation without watching video 1951 before.

29% (second Through %) of the people watching recommendation 1222 continued to watch the next video 1225. 29% is a fantastic conversion rate! Way above industry standards for conversion (automatic play of next video or showing on the side suggested videos) which is in the very low single digit conversion rates (sometimes below 1%). Video # Sessions Through Rec. # Sessions Through Next

% % Video #

1225 147 78% 1222 150 58% 64

1299 139 96% 1223 143 34% 1225

1951 1 18 68% 136/90 90 51 % 1313

97 97 88% 82 82 62% 29

1903 92 52% 79 79 92% 2035

Table 2 - Conversion Rates from 2" to 3^r Video

Table 2 shows the conversion rates from the 2ⁿ video to the 3^r video.

Similarly, Tables 3-6 below show how many people watched each video, progressed to following recommendation and continued to the following video, up to the 6^th video!

Table 3 - Conversion Rates from 3^r to 4 Video

Table 4 - Conversion Rates from 4 to 5 Video

Table 6 - Conversion Rates of 5 Video

Figs. 2-7 are screenshots of Google Analytics ^tm illustrating the same data of the above tables. The interactions described are between a video and the following recommendation video, and then between a recommendation video and the suggested video, and onward in the same manner. All the numbers represent sessions (views). Fig. 2 illustrates the conversion rate between the 1^st video and its recommendation. Fig. 3 illustrates the conversion rate between the 1^st recommendation and the 2^nd video. Fig. 4 illustrates the conversion rate between the 2^nd video and its

recommendation. Fig. 5 illustrates the conversion rate between the 2^nd

recommendation and the 3 video. Fig. 6 illustrates the conversion rate between the 3^rd recommendation and the 4^th video. Fig. 7 illustrates the conversion rate between the 4^th video and its recommendation. Fig. 8 illustrates the conversion rate between the 4^th recommendation and the 5^th video.

It will be readily apparent that the various methods and algorithms described herein may be implemented by, e.g., appropriately programmed general purpose computers and computing devices. Typically a processor (e.g., one or more microprocessors) will receive instructions from a memory or like device, and execute those instructions, thereby performing one or more processes defined by those instructions. Further, programs that implement such methods and algorithms may be stored and transmitted using a variety of media in a number of manners. In some embodiments, hard-wired circuitry or custom hardware may be used in place of, or in combination with, software instructions for implementation of the processes of various embodiments. Thus, embodiments are not limited to any specific combination of hardware and software

A "processor" means any one or more microprocessors, central processing units (CPUs), computing devices, microcontrollers, digital signal processors, or like devices.

The term "computer-readable medium" refers to any medium that participates in providing data (e.g., instructions) which may be read by a computer, a processor or a like device. Such a medium may take many forms, including but not limited to, nonvolatile media, volatile media, and transmission media. Non-volatile media include, for example, optical or magnetic disks and other persistent memory. Volatile media include dynamic random access memory (DRAM), which typically constitutes the main memory. Transmission media include coaxial cables, copper wire and fiber optics, including the wires that comprise a system bus coupled to the processor.

Transmission media may include or convey acoustic waves, light waves and electromagnetic emissions, such as those generated during radio frequency (RF) and infrared (IR) data communications. Common forms of computer-readable media include, for example, a floppy disk, a flexible disk, hard disk, magnetic tape, any other magnetic medium, a CD-ROM, DVD, any other optical medium, punch cards, paper tape, any other physical medium with patterns of holes, a RAM, a PROM, an EPROM, a FLASH-EEPROM, any other memory chip or cartridge, a carrier wave as described hereinafter, or any other medium from which a computer can read.

Various forms of computer readable media may be involved in carrying sequences of instructions to a processor. For example, sequences of instruction (i) may be delivered from RAM to a processor, (ii) may be carried over a wireless transmission medium, and/or (iii) may be formatted according to numerous formats, standards or protocols, such as Bluetooth, TDMA, CDMA, 3G.

Where databases are described, it will be understood by one of ordinary skill in the art that (i) alternative database structures to those described may be readily employed, and (ii) other memory structures besides databases may be readily employed. Any illustrations or descriptions of any sample databases presented herein are illustrative arrangements for stored representations of information. Any number of other arrangements may be employed besides those suggested by, e.g., tables illustrated in drawings or elsewhere. Similarly, any illustrated entries of the databases represent exemplary information only; one of ordinary skill in the art will understand that the number and content of the entries can be different from those described herein.

Further, despite any depiction of the databases as tables, other formats (including relational databases, object-based models and/or distributed databases) could be used to store and manipulate the data types described herein. Likewise, object methods or behaviors of a database can be used to implement various processes, such as the described herein. In addition, the databases may, in a known manner, be stored locally or remotely from a device which accesses data in such a database.

The present invention can be configured to work in a network environment including a computer that is in communication, via a communications network, with one or more devices. The computer may communicate with the devices directly or indirectly, via a wired or wireless medium such as the Internet, LAN, WAN or Ethernet, Token Ring, or via any appropriate communications means or combination of

communications means. Each of the devices may comprise computers, such as those based on the Intel. RTM. Pentium.RTM. or Centrino.TM. processor, that are adapted to communicate with the computer. Any number and type of machines may be in communication with the computer.

Claims

1. A computing system comprising:

at least one processor; and

at least one memory communicatively coupled to the at least one processor comprising computer-readable instructions that when executed by the at least one processor cause the computing system to implement a method of recommending video content, the method comprising:

a) displaying a first video segment; and

b) recommending a second video segment for viewing based on one or more tags associated with said first video.

2. The system according to claim 1, further comprising an initial step of tagging a video segment with one or more tags associated with the content of said video segment.

3. The system according to claim 1, wherein recommending said second video segment is done via a video segment, graphics, text, voice, audio or any combination thereof.

4. The system according to claim 1, wherein the tagging of a video segment is done by the video segment creator, by a computerized process, by a designated person, by one or more viewers or any combination thereof.

5. The system according to claim 1, wherein a recommendation for viewing a second video segment is created automatically after the identification of one or more tags in said first video segment.

6. The system according to claim 1, further comprising a step of preparing in advance a recommendation for viewing a second video segment based on one or more tags of a first video segment.

7. The system according to claim 1, wherein recommending said second video segment is based also on user profile information of the first video viewer.

8. The system according to claim 7, wherein said user profile information

comprises: a user's Internet navigation history, user's preferences, history of watched video segments, life style, hobbies, socio-economic data about the user, purchasing history, user's social media profile or any combination thereof.

9. The system according to claims 1, wherein said one or more tags relate to: a person, an object, a geographic location, an artistic work, a sports entity, a hobby, a genre, age, sex, mood, emotion, physical trait or any combination thereof.

10. The system according to claim 1, wherein said second video segment is an advertisement or a promoted content.

11. The system according to claim 1, wherein recommending a second video

segment is done by displaying a recommendation about said second video segment while said first video segment is being watched.

12. The system according to claim 11, wherein said displayed recommendation is a pop-up, a separate display aside the first video segment, a virtual layer integrated with the first video segment, or an integral part of said first video segment.

13. The system according to claims 11 or 12, wherein said recommendation about said second video segment comprises video, text, graphics, an image, a link or any combination thereof.

14. The system according to claim 1, wherein after recommending said second video segment, said second video segment is displayed automatically.

15. A computer system comprising:

a processor; and

a memory communicatively coupled to the processor comprising computer- readable instructions that when executed by the processor cause the computer system to:

a) display a first video segment; and

b) recommend a second video segment for viewing based on one or more tags associated with said first video.

16. The system according to claim 15, wherein initially, a video segment is tagged with one or more tags associated with the content of said video segment.

17. The system according to claim 15, wherein recommending said second video segment is done via a video segment, graphics, text, voice, audio or any combination thereof.

18. The system according to claim 15, wherein the tagging of a video segment is done by the video segment creator, by a computerized process, by a designated person, by one or more viewers or any combination thereof.

19. The system according to claim 15, wherein a recommendation for viewing a second video segment is created automatically after the identification of one or more tags in said first video segment.

20. The system according to claim 15, wherein a recommendation for viewing a second video segment is prepared in advance based on one or more tags of a first video segment.

21. The system according to claim 15, wherein recommending said second video segment is based also on user profile information of the first video viewer.

22. The system according to claim 21, wherein said user profile information

23. The system according to claims 15, wherein said one or more tags relate to: a person, an object, a geographic location, an artistic work, a sports entity, a hobby, a genre, age, sex, mood, emotion, physical trait or any combination thereof.

24. The system according to claim 15, wherein said second video segment is an advertisement or a promoted content.

25. The system according to claim 15, wherein recommending a second video segment is done by displaying a recommendation about said second video segment while said first video segment is being watched.

26. The system according to claim 25, wherein said displayed recommendation is a pop-up, a separate display aside the first video segment, a virtual layer integrated with the first video segment, or an integral part of said first video segment.

27. The system according to claims 25 or 26, wherein said recommendation about said second video segment comprises video, text, graphics, an image, a link or any combination thereof.

28. The system according to claim 15, wherein after recommending said second video segment, said second video segment is displayed automatically.