GB2518602A - Systems and methods for virtual participation in a real, live event - Google Patents

Abstract

A system for allowing players in a virtual environment or game SC to virtually participate in a real-life sporting event comprises an infra-red tracking sensor S provided above a real-life sports event for tracking the locations and movements of participants in the sporting event, and means T for transmitting tracking data for incorporation in the virtual environment. The infra-red sensor S may be attached to or suspended from a fixed structure (not shown), or may be mounted on an airborne vehicle V such as a powered or tethered balloon or blimp, a helicopter or such-like. The infra-red sensor tracks participants without requiring the participants to be provided with tags or similar active emitters or tracking devices. The sports event is carried out on an enactment area, and the enactment area may include at least three infra-red beacons B1-B4 providing fixed ground reference points for assisting in calculating the positions of the participants, and/or assisting in stabilising the airborne vehicle. The participants may be sports players, such as football or basketball player, or they may be mechanical entities such as racing cars, motorcycles or power boats.

Description

Systems and methods for virtual participation in a real, live event The present invention is concerned with improvements in or relating to systems for enabling participants to experience taking part in a real live event within a virtual environment.

The present invention provides systems and methods, which together enable completely passive detection and high accuracy tracking of a multiplicity of mechanical and/or biological participants within a delimited field of play. The systems and their methods of operation create data streams, which are suitable in content, accuracy, performance and timeliness to provide real-virtual entcrtainment, training and compctition in sports and othcr activities.

There are many prior disclosures by way of patents and pending patent applications for systems that allow a multiplicity of players to be virtual competitors in real sporting events, for example motor racing; see: W02009127927, W00200318, EP1217567 and W00l05476 wherein the activity on a racing circuit is monitored and encoded for inclusion on computer devices that are fed with data streams from live events as they are enacted in real time and are interactive, or in events that are recorded and stored in such systems.

The majority of the prior art systems and their methods of operation require the participants in an event to cariy some form of apparatus to enable their position(s) to be measured, transmitted and converted to the required data streams in real time. These have undesirable characteristics such as adding unwanted mass or complexity to the participants and/or significant commercial and/or security disadvantages in that the participant's emitted data can be detected and damaged or plagiarised by third parties.

Some of the prior art disclosures suggest or speci' passive detection (which we define here as detection without any modification or addition whatsoever to the artifacts or participants to be detected); however, the disclosures give no descriptions, designs or details of any systems, methods or apparatus that can achieve such detection in order to ensure the accuracy and timeliness levels necessaiy to provide a realistic interaction between real and virtual participants in a live event. Using a representative example of motor racing these requirements can be summarised as latitude/longitude measurements to better than 50cm with a refresh rate better than 25Hz, indoor, outdoor, all weather, day or night. Some aspects of these requirements may be different for different sports or events as dictated by the over-arching requirement of smooth, real-virtual interactive game play'.

Over recent years commercially available technology both for RADAR (RAdio Detection And Ranging providing accurate range and speed determination in applications such as traffic control) and JR detection (Infra Red imaging which provides accurate angular determination in applications such as motion capture) could possibly be combined to give accurate 3D dctcrmination of moving artifacts and/or participants in sporting evcnts, however the complexity of any system based on such a sensor thsion approach is huge and no practical approach has emerged. The cuffent invention is inspired by the fact that almost all sports and competitions take place in a defined, ground based enactment area with a known teffain map (flat as in field games or undulating as in many motor racing sports) and that the need for RADAR ranging can be removed in favour of solely JR tracking if the JR detector can be located at a fixed and known height vertically above the enactment area. With such a system architecture the accurate angular detection of moving targets provided by the JR sensor can be converted by practical, real time trigonometric and terrain mapping calculation into 3D position, velocity and acceleration rates to the accuracies and latencies required for interactive gaming and/or training.

It is therefore the object of the present invention to provide an improved method and system for interactive real-virtual entertainment, training and competition in sport and other activities.

Thus, the present invention conveniently provides systems and methods of operation for monitoring a real live event and combining the real live event within a virtual environment, whereby, when the system is in use, an unlimited number of participants in the virtual environment can experience the real live event as any selected player within the real live event in real time, or not, according to choice.

The system is characterised as comprising solely a downward facing infra-red tracking sensor and general purpose computing equipment positioned vertically above a real live eyent for encoding the parameters of an enactment area within which the real live event takes place; and for passively monitoring, analysing in 3D and encoding the positions and movcmcnts of a multiplicity of artcfacts andlor participants taking part within thc enactment area of thc real live event in indoor, outdoor, day, night and all-weather conditions, by sensing and recording only the infra-red radiation naturally emitted by the artefacts and/or participants during said event and calculating their positions with an accuracy, timeliness and data rate required for the effective performance of the virtual environment; and means to transmit any data obtained by said infra-red tracking sensor and computing equipment for use within the virtual cnvironment.

The infra red sensor, computing and communications equipment located vertically above an enactment area where a real live event is taking place is either in a fixed position determined to ensure that the field of view of the sensor encompasses the whole enactment arena and the detection resolution of the infra red sensor enables the positions of the artifacts and/or participants to be determined to the required accuracies; or in a variabic position controlled within position and velocity limits to cnsure that thc ficid of view of the sensor encompasscs continuously the whole enactment arcna including at lcast thrcc ground bascd infra rcd reference bcacons such that thc detcction rcsolution of the infra red sensor enablcs the positions of thc artifacts and/or participants to be dctcrrnincd to thc required accuracies The virtual enviroment may contain a multiplicity of devices such as gaming consoles, personal computers, simulators, arcade gaming devices, satellite receiying/processing equipment, smart phones, tablets, etc. Ilie method of operation of the system is characterized by the following steps in that it; a) detects the infra-red radiation emitted naturally by the artifacts and/or participants in the event by distinguishing it from the background radiation in the enactment arena; and b) calculates in real time the 2D position and movements of the artifacts and/or participants taking part within the enactment area throughout the duration of the event at the required refresh n rate of at least 25Hz to the positional accuracy required for smooth real-virtual interactive game play and, c) where necessary combines the calculated 2D position and movements with a terrain map of the enactment area to further calculate 3D position and movements of the artifacts and/or participants; and, d) transmits the position and movement data for use in the global virtual environment in real time.

In one convenient system provided by the present invention the infra red sensor, computing and communications equipment is mounted in a fixed position vertically above the enactment arena, for example on a fixed structure, or suspended in a fixed position by wires connected to fixed structure around the enactment arena, or mounted on a mobile crane or gantry located close to the enactment arena, or mounted on a lighter-than-air balloon fixed in position by ground tethers.

In another convenient system provided by the present invention the infra red sensor, computing and communications equipment is mounted in a variable position vertically above the enactment arena on an aerostat comprising preferably a lighter-than-air aircraft such as a blimp or balloon, or a helicopter, supplemented by at least three infra red beacons located on the ground at convenient positions around the enactment arena. The lighter-than-air aircraft or helicopter is provided with a flight control system to maintain the variable position within required limits over the enactment arena such that the infrared beacons are maintained within the field of view of the infra red sensor.

Conveniently, the artifacts may comprise transportation vehicles and the like for use in motor spoils, powerboat racing, motorcycle racing and other vehicle related sports or competitions.

In a further convenient arrangement the participants in a real live event may be sports players, for example footballer players, basketball players, baseball players, field sports players and players in any competitive sports or events.

There now follows, by way of example of the system and method provided by the present invention, a detailed description that is to be read with reference to the accompanying drawings in which: * Figure 1 shows a diagrammatic representation of a system layout of a monitoring sub-system provided by the present invention for a typical motor racing event * Figure 2 shows a diagrammatic representation of the monitoring and tracking features of the monitoring sub-system of Figure 1 for a typical NASCAR event The event is enclosed within a field of play, viz. the enactment arena, which is typically 1Km diameter for US NASCAR (and typically 2Km for International Formula I Racing) and subject to the prevailing range of atmospheric, weather and lighting conditions depending upon the time of year and the location in which the event is taking place.

Whilst multiple trackside passive detectors, for example cameras, are routinely used at motor racing events to time cars, they can only provide longitudinal position data in terms of distance covered about a track and/or between distance markers; that is, the trackside passive detectors cannot provide lateral track positions of participants in a field of play and hence the respective latitude/longitude of each participant would not be to the required accuracy for use in computer gaming.

The system illustrated in Figure 1, on the other hand, comprises a lighter-than-air (L.TA) aircraft V that is a balloon or blimp, or other suitable alternative, which LTA is provided with an infra-red tracking sensor S, general purpose computing equipment for data calculations, control stabilisation equipment B and data transmission equipment 1.

The control stabilisation equipment E measures the pitch, yaw, roll and velocity vector of the LTA V. which measurements are used: i. by the LTA Vs navigation and flight control systems, not shown, in order to maintain the LTAs position approximately above the centre of the enactment area, viz, the circuit where a live event is taking place; and, ii. by the infra-red tracking sensor S in order to maintain a fine control of a sensor bore-sight of tile sensor S such that it is directed accurately enough to maintain beacons Bi, B2, B3 and B4, set at the perimeter of the enactment area, within the range of the infra-red sensor S array on a focal plane.

In an alternative method of operation of the monitoring sub-system, the sensor S transmits the positions of the beacons BI to B4 to the LTAs navigation and flight systems, not shown, which then controls the LTA V in order to maintain its position above the beacons BI to 34.

The intl-a-red tracking sensor S resolves all of a multiplicity of participants in an event, which, in the case of motor racing would mean of the order of 24 vehicles in an International Fl event and 43 vehicles in NASCAR. In addition the sensor simultaneously monitors the position of the beacons BI to B4 in order to provide Lat-Long 2D referencing and ensure the viability, accuracy and timeliness of the monitoring system. The nature of the passive sensing that is used relies on the infra red radiation emitted by the participants in a live event. This is possible because of the advances made in infra red detector technology over the last ten years or so.

Human players typically have a resolvable temperature difference of 15 -40 degrees C above the background, which temperature signature is detectable using Long Wave (LW) 8 -l4um infra-red sensors. Powered machines, on the other hand, such as motor cars, power boats, motor cycles, and the like, have higher temperature signatures and are detectable by commercial Medium Wave (MIAT) 3-Sum infra-red sensors as well as LW sensors. Hence a LW sensor provides the broadest application, but either or both LW and MW sensors can be installed in the LTA V. The MW and LW thfra-red radiation detected by the sensor S has strong characteristics through normal atmosphere and weather at the distances proposed with the system and methods of the present invention. With the LTA V being arranged at a height of above 200m (Fig. 2), the overall design requires a field of view (FOV) of less than 140 degrees, a focal plane array CCD detector of at least 4 megapixels, i.e. 2k x 2k pixels, to give a bearing accuracy of approximately 0.1 degrees and a detection refresh rate of at least 180Hz in order to track vehicles at speeds exceeding 200mph. These parameters are at or approaching those that can be achieved by the latest technology (which is improving year on year).

A bearing accuracy of approximately 0.1 degrees requires 13 bits of digital data; hence, for 43 (as in NASCAR) participants and a Lat-Long of 26 bits, equates to 1118 bits. At 180Hz, this will generate a 200Kbit/sec data-strcam, which is passed to the transmission apparatus T. For short distance transmission to a ground station, this is practical and an encryption device, not shown, can be added for extra security. It is envisaged that the system may be developed further to link several of the systems together in order to extend the coverage to larger events such as long distance car rallies.

This example has provided a quite demanding scenario. By way of a further example the infra red sensor and gencral purpose computing equipment would be conveniently mounted on a fixed structure approximately lSm above a standard NBA basketball court, giving a requircd ficld of view of the sensor of approximately 9Odeg and a less demanding angular accuracy requirement.

Player identification may be manual input at the start of the event or start of play, supported by clash detection algorithms to maintain identification throughout periods of continuous play.

In use, the system provides a utile arrangement for enabling a player to experience the thrills of being a virtual participant in a real live event. By way of explanation of how the current invention enables a very wide range of such player experiences in a global virtual environment there follows brief descriptions of a small number of options that would apply to the sport of motor racing.

Firstly, see Figure 1: a signal from the current invention, the monitoring sub-system, is transmitted to the receiver R in the virtual environment, which receiver R is connected to the computer C, which also monitors inceptors 1, i.e. steering wheel, gear change, accelerator pedal and brake pedal, in the motor racing simulator. Computer C contains software and data that stores the graphical and dynamic models for the motor racing circuit and the competing cars so that it generates via a projector P onto a screen SC, or via a suitable alternative visual media, a real time dynamic representation of the environment of the simulator car as it progresses around the virtual circuit.

The displayed environment includes the visual images of all the real competing cars that are in view from the perspective of the simulator car's virtual position. In this way the driver in the virtual environment will experience that they are actually participating and competing in the race that is happening in real time on a motor racing circuit, albeit with the real competitiors not responding to their actions, but as the virtual car can be substituted for any one of the real cars a thrilling, realistic and novel experience will be created.

The computer C will also store pre-run races such that the simulation can be re-run at any time after the actual event, but each re-run will be different from the perspective of the simulator driver, as their performance will differ from time to time. Operating the system in this mode is useful as a training aid or for practicing in relation to a particular environment.

In modified arrangcmcnts of the player experience which the present invention cnablcs by supplying its real-time data, the Tmmersion sub-system, viz. R, SC, P, C and 1, could have a number of configurations, including at least the three versions described hereinafter.

a) A number of specialist, purpose built, full-immersion sub-systems with realistic cockpit environment, large screen surround 3D (with polarised glasses or alternative technology) display, say for example 3m x lUm, and high powered computing to host the software. A number of these would be located on the site of a sporting event for VTP or pay-per-play participation.

b) A number of public environments such as a game arcade, bowling alley or theme park, modified by the inclusion of the virtual environment software and the integration of receiver R. In this case a number of virtual players can compete simultaneously with the real competitors with all of the real and virtual participants being displayed on the screens being viewed by the virtual players.

c) An unlimited number of domestic systems around the globe comprising either a device with a broadband data service or enhanced satellite T\ receiver box where the immersion software is present, or downloaded on a pay-per-play, or on a multiple access basis, or with a commercial games console, for example, a PlayStation, Xbox or the like, connected to a broadband data service, for example Xbox Live, or satellite receiver box, where the software can be located in the console or on a CD, DVD or other medium, which is loaded into or stored within a games console.

Further modifications are envisaged to be within the scope of the appended Claims, for example, the simulator and inceptors T would, in the case of other sports, be modified to represent motor cycles, power boats, running machine, cycling machine and the like, or motion capture devices such as Kinect, and the data supplied by the present invention could be used for purposes other than interactive game play, for example the generation of real-time statistics about the event, or the generation of live video pictures from a virtual roving camera positioned anywhere within the enactment area and controlled by the viewer in the virtual environment, or the generation of live video pictures from the perspective of one of the real live participants in the event -i.e. through the "player's eye".