NL1039228C2

NL1039228C2 - Method and system for capturing, generating and sharing activity data.

Info

Publication number: NL1039228C2
Application number: NL1039228A
Authority: NL
Inventors: Suzanne Meyer
Original assignee: Thinkaheads B V
Priority date: 2011-12-09
Filing date: 2011-12-09
Publication date: 2013-06-11

Description

Title: Method and system for capturing, generating and sharing activity data

DESCRIPTION

5

The invention refers to a method or system respectively for capturing, generating and sharing among users data related to any activity of common interest to said users.

10 US patent application US2009044237 discloses a hosting system and method are provided for sharing, analysis, and review of videos, to include sport videos and data concerning athletes and teams. The system can be used for many purposes to include athlete education, recruiting of players, scouting of 15 teams, and coaching analysis. The system is web based where a central server or computer links to a website that is accessible to authorized users over a communications network such as the internet. Data is uploaded to the central server from the various remote users, and the data is selectively 20 edited and shared among the authorized users.

One aspect of the present invention is to provide a social, educational and/or professional network aimed at exchanging, editing, commenting, rating, analyzing etc. sets of activity 25 data (or activity data sets) made of real and/or live activities like e.g. sports, school/college plays, making music, as well as operations of rescue workers (fire brigade, ambulance etc.), police, military etc.

30 In this patent application a set of activity data is considered to include data connected (or linked) to images (including accompanying sound tracks), e.g. video fragments, originating from and/or made from said activity, i.e. a set of data which surrounds a particular video image (e.g.

1039228 2 fragment) that determines its meaning and/or describes the content and/or describes the structure. The set of activity data may also include (a subset of) non-image activity data describing any activity of one or more users. Such a set of 5 activity data preferably includes the relevant content data and/or context data and/or meta data of the video images, i.e. data related to the meaning and/or the context and/or the content and/or the (data) structure of those images.

10 Images may include all kind of images, e.g. originating from video cameras, photo cameras, etc., images originating from localization/positioning systems, e.g. based on RFID, radar tags, public positioning systems, images originating from infrared sensors, environmental or weather sensors, etc. etc. 15

Contrary to US2009044237(Al) in the present method and system according to the invention no file system is used but a streaming system (technical streaming description). According to the invention a platform is provided for generating and 20 sharing e.g. sport (or any other) activity data as well as e.g. including information on top of sport sources like video or other sport capturing sensors. (XML semantic data). Users are not reviewing but start communication and learning (technical = asynchronous data, status workflow). Users are 25 not doing it alone but together (technical = network, same time access, multiuser. Besides, according to the method and system according to the present invention video files are not provided (created) by the users, and files are not uploaded via a website. Besides, the users do not access a file.

30

The present invention provides a method for sharing among users data related to (including originated by or made from) any activity, which method comprises providing sets of activity data related to said activity, enabling storage of 3 said sets of activity data and retrievable access to them by said users, and enabling said users to display said sets of activity data, to process them and to share the resulting processing results. Preferably, the method, moreover, 5 comprises enabling said users to make selections from said sets of activity data, enabling said users to rate and/or to comment and/or to analyze said sets of activity data and/or said selections, and enabling said users to share said selections and/or rating results and/or comments and/or 10 analyzing results. More preferably , the method comprises enabling said users to edit said sets of activity data and/or said selections, enabling said users to share said edited said sets of activity data and/or said selections.

15 The method according to the invention thus enables a social, educational and/or professional network aimed at exchanging, editing, commenting, rating, analyzing etc. sets of activity data made of real and/or live activities like e.g. sports, school/college plays, making music, as well as operations of 20 rescue workers (fire brigade, ambulance etc.), police, military etc.

The sets of activity data, related to, originated by or made from the relevant activity may include video data made by one 25 or more video cameras of said activity, e.g. a football match, a fire brigade or military exercise. According to the invention, such video images may be stored, displayed and processed by either or not professionally interested users, who may e.g. cut some interesting selections from such video, 30 rate, comment, analyze, edit the video and/or the selections made, and share all this with other users. For enabling all this the portal server and the local facilities (e.g. computers) of the users may be used in cloud computing mode.

4

Besides video data, said sets of activity data may also include location data of humans and/or non-human objects participating in said activity, hereinafter called participants, wherein use can be made of any (public) global 5 positioning system like e.g. GPS or any other (public) localization or positioning systems. For instance the participants may be provided with location receivers and (small) transmitters, streaming the participants' positions to a server which processes that location localization data 10 into sets of activity data (including sets of non-image activity data) , i.e. into data which are fit to build a (generally moving) image, which is to be displayed to the users and can be used to generate further activity data. Location data may also include data made by an local 15 positioning system at or near said activity's location, e.g. a local positioning system based on the use of radar and radar tags.

Localization data may be used to control video cameras at the 20 site of the activity, which is known per se from EP1928178 (TNO) and US7046273 (FUJI).

Besides the (more or less) dynamic activity data, originated by video and (e.g. GPS or radar based) localization sources, 25 (more or less) static data, e.g. related to the environment of the participants of activities (sports game, musical performance, military operation etc.) may be used to supplement those dynamic data in order to complete the whole set of activity data. All those subsets of activity data 30 (dynamic video, dynamic localization data, static environmental data) may be processed in augmented reality and/or virtual reality mode, i.e. in a way which enables the users to see virtual and/or real images (see e.g. http://en.wikipedia.org/wiki/Augmented_reality) 5

Hereinafter, the invention will be elucidated with reference to some figures.

5 Figure 1 shows a system which is known as such, but which is capable to be adapted to be part of a technical architecture arranged for performing the method according to the present invention; 10 Figure 2 shows an exemplary system architecture arranged to perform the method according to the invention.

The system shown in figure 1 is arranged for monitoring (tracing, tracking) , by means of a number of video cameras 1, 15 a game object 2a and game players 2b and 2c. In this exemplary embodiment, the system comprises one game object, a football, and 22 game players, two of which are drawn in figure 1. Both, the game object and the game players may be considered as participants, viz. human and non-human 20 respectively. The football and (some of) the players are provided with radar responsive means. Monitoring and/or tracing players can be used to evaluate the players' performance, either individual or (also) as a group.

25 The radar responsive means may be formed by a state-of-the-art small and light radar transponder. A transponder 3a is mounted inside the football 2. The players 2b and 2c may be provided with clothing, e.g. shirts, which include radar transponders 3b and 3c. In the practice of a football game 30 e.g. all 22 players may be followed individual by means of (22) individual radar transponders. All transponders are arranged to reflect an individual, unique identification code, which enables the radar means to calculate the position of each individual transponder and thus the position of the 6 ball and the individual players. The radar means may comprise a number of FMCW (Frequency-Modulated Continuous Wave) radar transceivers 4 which are arranged to monitor the radar transponders 3a, 3b and 3c and to calculate dynamically (or 5 to trace or track) the positions of the football 2a and players 2b and 2c to be monitored (traced). Output signals from the radar transceivers 4 may be fed to control means 5 which are arranged to control the various video cameras 1 to follow and record the game object 2a and game players 2b and 10 2c. To that end control signals are fed to the cameras 1 - each of them having a camera controller 6 - to direct and to focus the camera to the football 2a and players 2b and 2c in each position they may have during the football match. It will be clear that to trace the football and the (e.g. all) 15 players a great number of cameras 1 may be needed.

Figure 2 shows an exemplary system architecture arranged to perform the method according to the invention, comprising means, formed by the video cameras 1, for providing sets of 20 activity data related to the activity, i.e. a football game, means, formed by a database 7, arranged for storage of the sets of activity data and for retrievable access to them by the users, viz. via the Internet 8, under control of a server 9, as well as means, formed by software modules installed in 25 the server 9 and/or in terminals 10 of users 11, arranged for enabling the users 11 to display the sets of activity data, i.e. the video images made from the football game by the video cameras 1, to process them and to share the resulting processing results by means of software modules installed in 30 the server 9 and/or in the terminals 10. The hardware and software of server 9 are arranged to operate as a host and a portal which can be used by the users 11 which may either or not be members of a (e.g. paying or otherwise restricted) users group. The hardware and software means, incorporated in 7 the server 9 and (partly) in the terminals 10, arranged for processing the video images enable the users to make selections from the sets of activity data and to rate, to comment and to analyze them, as well as means arranged for 5 enabling the users to share, via the Internet 8, the selections made by them, their ratings, comments and analyses. The hardware and software means, incorporated in the server 9 and (partly) in the terminals 10, are further arranged to enable the users 11 to edit the videos and 10 selections therefrom and to share those edited videos.

The system shown in figure 2 thus comprises video cameras 1, control means 5 for controlling the video cameras 1, as well as processing means, mainly incorporated in the portal server 15 9, arranged for processing the video data under control of the users 11, using their terminals 10 (desktops, laptops, mobile devices etc.). The activity data are sent from the video cameras 1 to the control means 5, in which -in the configuration of figure 2- means may be incorporated for 20 transmitting the video data to the portal server 9 via the Internet 8.

As indicated in the discussion of figure 1, a location system is provided, arranged for localizing humans and/or non-human 25 objects participating in the activity, called participants. The location system may be based on the use of a (public) global positioning system (GPS) which is indicated in figure 2 by a GPS satellite 12. In that case each transponder 3a, 3b and 3c -working as individual identifier modules for the 30 participants 2a, 2b and 2c- need to comprise a GPS receiver arranged to compute its individual location from a number of satellite signals and to transmit its identifier and its location to a local receiver 13, which is connected to the control means 5. In this way the sets of activity data, 8 comprising video data, can be supplemented by activity data originated by the (stream of) localization data from each game participant, which can be assembled by the hardware and software of the portal server 9 to another kind of activity 5 data and/or stored in the database 7 which is retrievable by the users 11 via their terminals 10.

As revealed in the prior art system of figure 1, discussed above, the location system may include a (fully) local 10 positioning system at or near the activity's location, e.g. comprising a radar system, comprising one or more radar transceivers 4, preferably FMCW radar transceivers, at or near the activity's location, as well as radar tags 3a, 3b and 3c connected to one the participants 2a, 2b and 2c. The 15 localization data of all game participants are collected in the control means 5 (a local computer, acting more or less as a local server), which transmits the localization data of the individual participants to the portal server 9 and its database 7. Also here the activity data is supplemented by 20 sets of activity data originated by the localization means.

The locally originated activity data, originated by the video cameras 1 and public and/or (private) radar based localization sources, as well as global activity data, 25 including any other sensors such as user heartbeat, temperature and weather and wind sensors etc., related to the environment of the participants 2a,b,c and originated by e.g. public sources like Google Maps and Google Earth, may be used to supplement the sets of activity data originated by at the 30 local activity site (the football field in de example of figures 1 and 2) in order to complete the sets of activity data with activity data which may be relevant and/or interesting for the users 11, and which complete set of activity data may be used in their individual and collective 9 (shared) selection, editing, rating, and/or analysis process. All those subsets of activity data (video, localization data, global data) may be processed in the portal server 9 using augmented reality and/or virtual reality hardware and/or 5 software, thus enabling the users to see and process all activity data in augmented reality and/or virtual reality mode. For enabling all this without needing to use heavy duty computer facilities at the portal server's side, in the portal server and the local computers of the users are 10 operated in cloud computing mode.

It is noted that the activity data may be captured using automatic position tracking technology using audience head and/or eyes positions to locate where the interesting 15 activity data is. The activity data may also captured using manual controlled position tracking to capture interesting activity data, the tracking being controlled by various channels, e.g. a mobile application, a web application, remote hardware wirelessly linked to camera system. Besides, 20 the activity data may be captured e.g. under control of a coach when sport game is being captured, and/or audience with access to control system on real time on field and/or a third party service provider to offer tracking service.

25 The activity data may be analyzed for facial emotions of the audience, viewers and/or players to generate activity data about nature of activity data viz., boring, funny, exciting etc. The activity data may be used for automatically analyzing the players' body/physical movements to generate 30 meaningful data to be used to coach movements for better performance and/or to determine possible physical health problems for the players. Besides, data based on player body movements may be used to design improved sports goods by sports goods manufacturers.

10

The activity data may also be used for gamification, i.e. the use of game design techniques and/or mechanics to solve problems and engage audiences. The activity data may be 5 distributed by narrow casting, i.e. personalized sport video selection broadcast based on context. The activity data may, besides, be used for (home) game console integration (e.g. Wii®, Playstation®), enabling physical replay, using modern gaming device techniques, selected actions at home. Net TV 10 integration and/or image recognition may be provided, as well as the enablement of real time advice for the player and/or coach based on context using collected activity data. The simulation of a match between two teams may be based on collected activity data, as well as means for comparing 15 individual player with best performer from collected activity data. Players me be replaced with avatars in playback.

Automatic transforming the video in a schematic overview may be provided for schematic playback. Another option is to 20 register the hart beat of players e.g. in order to monitor their physical condition. Voice recognition may be used during a game, for the registration of activity data.

Finally, it is noted that smart phones 14 and other mobile 25 devices such as tablets 15 may be used as (wireless) user terminals. Some persons 16 (e.g. users) may be enabled to capture actions using mobile devices 17. Such mobile device 17 may also be enabled to control, e.g. via the control means 5, the cameras 1 and/or other capturing or sensing means.

30 Augmented reality may be performed by means of virtual reality devices 18.

1039228

Claims

A method for recording, generating and sharing among users data related to an activity, comprising: providing sets of activity data related to said activity; enabling storage of said sets of activity data and retrievable access to these by said users; make it possible to display said sets of activity data to said users, to process them and to share the resulting processing results. 15

2. Method as claimed in claim 1, wherein said sets of activity data comprise the content data and / or context data and / or metadata of images, related to the meaning and / or the context and / or the content and / or the structure of those images.

3. Method as claimed in any of the foregoing claims, comprising of making it possible for said users to make selections from said activity data; making it possible for said users to rate and / or comment on and / or analyze said activity data and / or said selections; making it possible for said users to share said selections and / or valuation results and / or comments and / or analysis results; make it possible for said users to add other activity data such as action category data or activity description data; 1039228 the generation by said users of the meaning of the activity data by recording an indication of the use of the activity data. 5

A method according to any one of the preceding claims, comprising enabling said users to process said activity data and / or said selections; 10. make it possible for said users to share said edited activity data and / or said selections; - making it possible for said users to add said activity data and / or said selections;

5. Method as claimed in any of the foregoing claims, wherein said activity data comprises video data made of one or more video cameras of said activity.

6. Method as claimed in any of the foregoing claims, wherein said activity data comprises location data of human or non-human objects participating in said activity, hereinafter referred to as participants.

The method of claim 6, wherein said location data comprises data made by a random positioning system. 30

The method of claim 6 or 7, wherein said location data comprises data created by any local positioning system at or near the location of said activity.

The method of claim 8, wherein said local positioning system is based on the use of radar and / or radar tags. 5

10. Method as claimed in any of the foregoing claims, wherein said activity relates to the practice of sports, music, theater, public events, entertainment, school games and / or school or university activities.

11. Method as claimed in any of the foregoing claims, wherein said activity relates to rescue actions or operations or to military actions or operations.

12. Method as claimed in any of the foregoing claims, wherein said processing of activity data comprises the processing thereof in augmented reality and / or virtual reality mode.

A method according to any of the preceding claims, wherein said activity data is recorded using automatic position tracking technology using main and eye positions of the audience to locate where the activity of interest is located.

Method according to any of the preceding claims, wherein said activity data is recorded using manually controlled position tracking to record activity data of interest, wherein the tracking is controlled via different channels, e.g. a mobile application, a web application, wireless hardware linked to a camera system.

Method according to claim 14, wherein said activity data is recorded using manually controlled position tracking system controlled by various means, e.g. under the control of a coach when a sporting game is recorded, and / or audience at the real-time playing field to the operating system and / or a third-party service provider for providing tracking service.

16. Method according to any of the preceding claims, wherein said activity data is analyzed for facial emotions of the audience, spectators and / or players to generate activity data about the nature of the activity data such as bored, funny, exciting etc.

17. Method as claimed in any of the foregoing claims, wherein said activity data comprises images of people, hereinafter referred to as players, whose physical / physical movements are automatically analyzed to generate activity data for use to coach movements for better performance and / or to identify potential physical health problems in players.

18. A method according to any one of the preceding claims, wherein the generated meaningful data based on movements of the bodies of players is used to develop improved sports articles by manufacturers of sports articles.

19. A method according to any one of the preceding claims, wherein the activity data is used for gamification, i.e. the use of game design techniques and / or mechanics to solve problems and engage the public.

20. A method according to any one of the preceding claims, wherein the activity data is distributed by means of narrow casting, i.e. personalized broadcast of sports videos based on context.

21. Method according to any of the preceding claims, wherein the activity data is used for gaming equipment, game console integration, enabling physical replay, using gaming equipment techniques and / or selected actions at home.

22. Method as claimed in any of the foregoing claims, comprising net TV integration and / or image recognition.

23. Method as claimed in any of the foregoing claims, comprising real-time advice for the player and / or coach based on context using collected activity data.

A method according to any one of the preceding claims, comprising simulating a match between two teams based on collected activity data. 30

A method according to any one of the preceding claims, comprising comparing an individual player with the best performer from collected activity data.

A method according to any one of the preceding claims, comprising replacing players with avatars in playback.

A method according to any one of the preceding claims, comprising automatically transforming the activity data into a schematic overview for schematic playback.

A method according to any one of the preceding claims, comprising registering players' heartbeats.

29. Method as claimed in any of the foregoing claims, comprising using speech recognition during a game for registering activity data.

A system adapted to perform the method according to any of the preceding claims.

31. System as claimed in claim 30, comprising means (1,3,4,5,17) arranged for providing sets of activity data related to said activity, means (9) arranged for storing said sets of activity data and retrievable access to them by 25 said users (11) and means (8, 9, 10) adapted to enable said users to display said sets of activity data, to process them and to share the resulting processing results. 30

A system according to any one of the preceding system claims, comprising one or more video cameras (1.17) and control means (5.17) and / or processing means for respectively controlling and / or processing said one or more video cameras and / or said activity data .

A system according to any of the preceding system claims, wherein said processing means comprises cloud computing means. 1039228