WO2013054142A1

WO2013054142A1 - An immersive interactive video conferencing, social interaction or gaming system

Info

Publication number: WO2013054142A1
Application number: PCT/GB2012/052553
Authority: WO
Inventors: James Peter Rowan MORREY
Original assignee: University Of Ulster
Priority date: 2011-10-14
Filing date: 2012-10-15
Publication date: 2013-04-18
Also published as: GB201117799D0

Abstract

An immersive interactive video conferencing, social interaction or gaming system is disclosed comprising a device having one or more video cameras. The video cameras are arranged to record a plurality of simultaneous first video images of a user and the user's environment. The device also comprises one or more video projectors which are arranged to project a plurality of second video images relating to video content from other users.

Description

AN IMMERSIVE INTERACTIVE VIDEO CONFERENCING, SOCIAL INTERACTION OR

GAMING SYSTEM

The present invention relates to an immersive interactive video conferencing, social interaction or gaming system and a method of immersive interactive video conferencing, social interaction or gaming.

The preferred embodiment relates to a device that simultaneously transmits and projects video across the internet to create a live real-time social network that can be navigated using hand and body gestures.

CROSS-REFERENCE TO RELATED APPLICATION This application claims priority from United Kingdom Patent Application No.

1 1 17799.5 filed on 14 October 201 1 . The entire content of this application is incorporated herein by reference.

BACKGROUND TO THE PRESENT INVENTION

Social networking has, in recent years, become a significant social phenomenon. There are, for example, more than 500 million active users of social media and 50% of those active users log on to social networking sites such as Facebook (RTM) on any given day. The average user of such social networking sites has approximately 130 friends or contacts and users spend, in total, over 700 billion minutes per month on Facebook (RTM).

There are over 900 million objects that people can interact with (e.g. pages, groups, events and community pages) and the average user s connected to approximately 80 community pages, groups and events. An average user creates approximately 90 pieces of content each month and there are more than 30 billion pieces of content (e.g. web links, news stories, blog posts, notes, photo albums etc.) shared each month.

There are more than 70 translations available on the Facebook (RTM) site and about 70% of users are from outside the United States.

Entrepreneurs and developers from more than 190 countries build with Facebook Platform (RTM) and users of Facebook (RTM) install more than 20 million applications every day. Every month more than 250 million people engage with Facebook (RTM) on external websites. Social plug-ins were launched in April 2010 and an average of 10,000 new websites integrate with Facebook (RTM) every day. The social networking market taking into account Facebook (RTM) alone is worth £83.78bn and has recently (October 2012) hit over a billion users.

YouTube (RTM) receives over 2 billion views per day and as of February 2010 there were approximately 48,2 million registered users on YouTube (RTM) worldwide, 24 hours of video are uploaded every minute. There is also an online gaming market worth £9.56bn and which continues to grow. Social networking continues to develop and there is a desire amongst users for the ability to enjoy improved interactions with other users.

US-7643064 relates to a predictive video device system. A user sits in a room showing a remote meeting location with a robot device being located at the remote location. Video and audio are transmitted between the display and the robot device so that the user has the feeling of being present at the remote location by seeing it in a surround view. The participants in the meeting at the remote location have the feeling that the user is present by panels showing images of the head of the user i.e. the feeling of telepresence. It will be apparent that the arrangement disclosed in US-7843064 requires a dedicated room lined with foam to prevent lighting anomalies. The room requires cameras and projectors and the remote location requires a robot surrogate which is controlled by the user. The system uses directional sound and line-of-sight mapping so that the surrogate robot can

(theoretically at least) communicate the nuances of human interaction which may otherwise be lost through video conferencing. The disclosed arrangement has a complex control structure and contingencies built in for navigation and network errors.

US 2004/01 17087 discloses a mutually immersive mobile telepresence with gaze and eye contact preservation. The gaze of a user looking at an object in a display cube will appear, to the participant looking at the image of the user, to be a gaze directed to the actual object in the meeting room. Similarly, the gaze of a user looking at the eyes of the participant displayed in the display cube will appear to the participant gazing at the image of the eyes of the user on the surrogate's displays to be a gaze directed at the participant's eyes. This is useful to let the participant know that the user is paying attention and can be used to arbitrate between taking of turns in a conversation.

US-7324664 relates to a method of and system for determining angular orientation of an object such as a person's head. In particular, US-7324664 is concerned with determining the angle at which a camera engages the line of sight of a viewer in order to maintain the line of sight.

US 2005/0192721 relates to a mobile device control system. The control system moves a surrogate under wireless control and autonomously moves the surrogate to regain wireless control when the wireless control is lost.

The four references referred to above all relate to the field of mobile telepresence, in contrast to developments in the field of mobile telepresence, it is desired to provide an improved video conferencing, social interaction or gaming system which does not require the provision of a dedicated room or a surrogate robot.

SUMMARY OF THE PRESENT INVENTION

According to the present invention there is provided an immersive interactive video conferencing, social interaction or gaming system comprising:

a device comprising: (i) one or more video cameras, the video cameras being arranged and adapted to record a plurality of simultaneous first video images of a user and the user's environment; and (ii) one or more video projectors arranged and adapted to project a plurality of second video images.

According to an embodiment the device has a substantially cubic form. However, it should be understood that other embodiments are contemplated wherein the device may have a substantially different or more complex shape.

According to the preferred embodiment the one or more video cameras are located within the device.

According to an embodiment the one or more video projectors are located within the device.

The one or more video projectors are preferably located on substantially flat surfaces of the device. However, other embodiments are contemplated wherein the one or more video projectors are located at different positions within the device.

According to the preferred embodiment one or more of the plurality of first video images are transmitted remotely to one or more other users.

The plurality of second video images preferably comprise images of other users or the other users' environment or images from a game.

The device preferably further comprises a gesture recognition system.

The user can preferably control by gestures either: (i) the first video images which are recorded and/or transmitted to other users; and/or (ii) the second video images which are projected by the one or more video projectors; and/or (iii) a menu, setting or parameter related to the device.

The second video images are preferably arranged and adapted to be projected onto walls of a room or other surrounding.

According to another aspect of the present invention there is provided a method of immersive interactive video conferencing, social interaction or gaming comprising:

simultaneously recording one or more first video images of a user or the user's environment from different angles or perspectives; and

simultaneously projecting one or more second video images.

The method preferably further comprises automatically sensing gestures.

The method preferably further comprises controlling by gestures either: (i) the first video images which are recorded and/or transmitted to other users; and/or (ii) the second video images which are projected; and/or (iii) a menu, setting or parameter of a device which is arranged and adapted to record the first video images and to project the second video images.

The method preferably further comprises projecting the second video images onto walls of a room or other surrounding.

According to another aspect of the present invention there is provided an interactive video conferencing, social interaction or gaming system comprising:

a device;

one or more video cameras integrated into the device, the one or more video cameras being arranged and adapted to record one or more different first video images of a user or the user's environment; one or more video projectors integrated into the device, the one or more video projectors being arranged and adapted to project one or more second video images onto walls of a room or other surrounding; and

a gesture recognition system for recognising gestures made by the user, wherein the user can control either: (i) the first video images which are recorded and/or transmitted to other users; and/or (ii) the second video images which are projected by the one or more video projectors; and/or (iii) a menu, setting or parameter of the device.

According to another aspect of the present invention there is provided a method of interactive video conferencing, social interaction or gaming comprising:

providing a device, wherein one or more video cameras are integrated into the device, the one or more video cameras being arranged and adapted to record a plurality of different first video images of a user or the user's environment and wherein one or more video projectors are integrated into the device, the one or more video projectors being arranged and adapted to project one or more second video images onto walls of a room or other surrounding; and

recognising gestures made by the user, wherein the user can control either: (i) the first video images which are recorded and/or transmitted to other users; and/or (ii) the second video images which are projected by the one or more video projectors; and/or (iii) a menu, setting or parameter of the device.

The preferred embodiment relates to a device specifically for an immersive interactive social networking. However, it could also be used in place of expensive video conferencing installations and it has applications for future gaming as means of interacting, for example, in games like Second Life (RTM).

The preferred embodiment comprises a device that records video from one or more built-in cameras and transmits it via Wi-Fi or LAN connection across the internet whilst simultaneously projecting and playing streamed video from other users, onto the room in which the user is standing via one or more projector(s).

While the preferred embodiment is recording a user, the user is able to browse the network of other users in real-time using articulated gestures (e.g. swipe to scroll etc.) and navigate this stream of video sources of potential interest whilst interacting with multiple feeds as they go.

Despite significant advances in camera and web technology and also internet infrastructure, present day online social interactions are still limited to the framework of a website which is accessed using an internet browser, interactions on social networking sites may have become quicker and more media-rich, however, they are still essentially a static non-dynamic experience. Mainstream social networking is not a live user experience. A different way to interact online is proposed which changes the nature of online

relationships and social interactions.

Facebook (RTM) and other social networking sites still utilise a bulletin board type format of posting and waiting for replies. The preferred embodiment allows for a more personal, more interactive and more immersive user experience.

Programs like Skype (RTM) and other web cam based software and web solutions such as Chat Roulette (RTM) allow users to communicate via video and sound across a network of users. Using their web cams, users of this software/sites can record and view each other in real-time. However, they require a PC to use and are not immersive or interactive. The preferred embodiment has a richer, more interactive experience making social networking more social and more transparent. Also, since a user cannot hide behind a fake profile as they are being filmed then social interaction is more natural and real.

The preferred embodiment may be used in gaming as a controller device. The preferred device may also be used in events management to link events in different countries. The preferred device can also be used commercially in the home as a tool for entertainment.

BRIEF DESCRIPTION OF THE DRAWINGS

Various embodiments of the present invention will now be described, by way of example only, and with reference to the accompanying drawing in which:

Fig. 1 shows a preferred embodiment of the present invention comprising a device that simultaneously transmits and projects video across the internet to create a live realtime social network that can be navigated using hand and body gestures;

Fig. 2 shows an internal arrangement of video cameras and video projectors according to an embodiment of the present invention; and

Fig. 3 shows a 3D view of a preferred device showing the location of video projectors and video cameras.

DETAILED DESCRIPTION OF PREFERRED EMBODIMENT

Fig. 1 shows a preferred embodiment of the present invention and comprises a device that simultaneously transmits and projects video across the internet to create a live real-time social network that can be navigated using hand and body gestures.

The preferred embodiment comprises a social media device or other device which enables a user to interact with other internet users through a video link. Users are able to use the device to enjoy an online relationship and an immersive interactive experience with one or more other users without requiring the use of a computer.

The preferred device preferably records video from multiple built-in digital video cameras and preferably transmits digital video signals via a Wi-Fi or LAN connection across the internet. According to a particularly preferred embodiment, the same device also preferably simultaneously projects and plays streamed digital video from one or more other users onto e.g. the walls of the room in which the user is standing via one or more digital video projectors which are preferably mounted on the flat surface of the device.

According to an embodiment the device as shown in Fig. 1 may record and transmit images of a user whilst at the same time the user is able to browse a network of other users in real-time preferably using articulated gestures (e.g. swipe to scroll etc.). The user preferably has considerable freedom in how to navigate between and interact with multiple feeds and multiple other users.

Conventional online social interactions are still limited to the framework of a website which is viewed using an internet browser. Although user interactions on such internet sites have become substantially quicker and more media-rich, nonetheless the user interactions are still essentially static and social networking is not a rapidly changing live experience. At best, users may interact via online chat which is a text based interaction.

The preferred embodiment enables users to interact online in a manner which enables the nature of online relationships to be dramatically changed and improved. The preferred embodiment enables a live video stream to be projected from within the homes of users across the web to provide a large live network feed whilst simultaneously

broadcasting a video feed of the user.

A particularly preferred aspect of the present invention is that the user does not need to use a conventional keyboard in order to interact with other end users.

Advantageously, hand and body gestures may be used to navigate through the live stream and to enable a user to interact with one or more other users without requiring either a computer screen or an internet browser.

The preferred embodiment preferably utilises conventional camera technology as found, for example, in the Xbox Kinect (RTM) to record video and interpret gestures (for navigation) which are transmitted live via a LAN or Wi-Fi connection to a streaming network while video projectors on the device preferably display the feeds of other users onto the room around the user for an immersive interactive experience. According to an

embodiment the cameras are preferably situated in bevelled comers of the device while the projectors are preferably situated on the flat surfaces. Fig. 2 shows the internal arrangement of video cameras recessed within bevelled corners regions and video projectors recessed within flat surface or faces of the preferred device.

According to an embodiment the video cameras preferably comprise colour CMOS cameras and feature 380 TV lines, Lux of 1.5 at F1.2 and shutter: 1/50-1/63000. Power preferably comprises 7-12V DC, 30 mA.

According to an embodiment the video cameras preferably have dimensions of 8 mm x 8 mm x 10 mm and are light e.g. weighing approximately 1 g without cable. The cameras preferably have a 3.5 mm lens.

A CPU or processor is preferably used to store and operate the user interface and preferably connects the preferred device to the Internet. According to an embodiment the preferred device may incorporate a small microcomputer. A known microcomputer which may be utilised is a Raspberry Pi (RTM) and measures just 3.370 x 2.125 inches (85.60 mm x 53.98 mm). The Raspberry Pi (RTM) comprises a micro PC having a 700 MHz processor and 256 MB of RAM that supports 14 HDMI resolutions from 640x350 to

1920x1200 plus various PAL and NTSC standards. External interfaces include a Flash memory card slot, USB port, Ethernet and a serial port. It is powered by Linux (RTM). According to an embodiment the projectors ) which are used may comprise a micro projector available from, for example, 3M which are particular suitable for use specifically in mobile devices. The micro projectors are designed for integration into virtually any mobile platform. 3M's LED-illuminated projection engine is capable of throwing a 40-inch or larger VGA image from an ultra-compact module that is < 0.5" thick. The miniature projector boasts a high-quality image with no-speckle and a high-fill using an advanced Liquid Crystal on Silicon ("LCOS") electronic imager in conjunction with proprietary 3M optics technology.

The speakers preferably have an output power of 2.2 W and are preferably 36 mm x 1 mm with a SNR≥ 89 dB. The speakers may utilise a lithium-ion battery input of 3.7V and may have a charge voltage of 4.2 V. The battery capacity is preferably 300 mAh and the speakers preferably have a net weight of 50g.

The device according to the preferred embodiment is intended to have particular design appeal to male and female users aged 16-30 who are existing users of social networks.

According to an embodiment the device may be utilised as a device for promotional and/or events management purposes. For example, events managers may run events simultaneously and set up the preferred device to connect these events together.

A large industrial show model may be utilised which is capable of recording and projecting very high quality video.

Software programs such as Skype (RTM) and other web cam based software and web solutions such as Chat Roulette (RTM) allow users to communicate via video and sound across a network of users. Using their web cams, users of this software/sites can record and view each other in real-time. However, the experience is not an immersive one as the user is still interacting in front of a back-lit screen (usually seated).

it will be apparent, therefore, that there is fundamental difference between the present invention and current web cam programs which provide an essentially static web cam experience.

According to the preferred embodiment the preferred device projects onto the room around an user making the user feel immersed in the experience of talking to and interacting with their friends. According to the preferred embodiment a user is able to enjoy a full view of the environment of another friend or user which is preferably projected onto the wails of the user's home. The user can be standing or seated and able to face up to and talk with their friends as if they were present. Advantageously, a user does not require a computer to participate as the preferred device is independent of platform.

The preferred device is preferably run by a micro Unix based computer system or equivalent (to make the user interface work) and according to an embodiment there may be no wires visible. Furthermore, the preferred device does not require additional towers or peripherals so the user is free to interact with other users in a simple and uncluttered manner. The simple connectivity of the preferred embodiment removes the problem of their being any connectivity problems or software updates required. The preferred device is therefore very straight forward to set up and enables a user to spend a greater amount of their time interacting with friends. Users can connect to other users without the need for expensive equipment and this also means that a user's experience is unaffected if they have a low specification computer or low bandwidth internet connection.

Although the present invention has been described with reference to preferred embodiments, it will be understood by those skilled in the art that various changes in form and detail may be made without departing from the scope of the invention as set forth in the accompanying claims.

Claims

1 1054903v12

Claims 1 . An immersive interactive video conferencing, social interaction or gaming system comprising:

a device comprising: (i) one or more video cameras, said video cameras being arranged and adapted to record a plurality of simultaneous first video images of a user and said user's environment; and (ii) one or more video projectors arranged and adapted to project a plurality of second video images.

2. An immersive interactive video conferencing, social interaction or gaming system as claimed in claim 1 , wherein said device has a substantially cubic form.

3. An immersive interactive video conferencing, social interaction or gaming system as claimed in claim 1 or 2, wherein said one or more video cameras are located within said device.

4. An immersive interactive video conferencing, social interaction or gaming system as claimed in claim 3, wherein said one or more video cameras are located within bevelled corners of said device.

5. An immersive interactive video conferencing, social interaction or gaming system as claimed in any preceding claim, wherein said one or more video projectors are located within said device.

6. An immersive interactive video conferencing, social interaction or gaming system as claimed in claim 5, wherein said one or more video projectors are located on substantially flat surfaces of said device.

7. An immersive interactive video conferencing, social interaction or gaming system as claimed in any preceding claim, wherein one or more of said plurality of first video images are transmitted remotely to one or more other users.

8. An immersive interactive video conferencing, social interaction or gaming system as claimed in any preceding claim, wherein said plurality of second video images comprise images of other users or said other users' environment or images from a game.

9. An immersive interactive video conferencing, social interaction or gaming system as claimed in any preceding claim, wherein said device further comprises a gesture recognition system.

10. An immersive interactive video conferencing, social interaction or gaming system as claimed in claim 9, wherein said user can control by gestures either: (i) said first video images which are recorded and/or transmitted to other users; and/or (ii) said second video images which are projected by said one or more video projectors; and/or (iii) a menu, setting or parameter related to said device.

1 1 . An immersive interactive video conferencing, social interaction or gaming system as claimed in any preceding claim, wherein said second video images are arranged and adapted to be projected onto walls of a room or other surrounding.

12. A method of immersive interactive video conferencing, social interaction or gaming comprising:

simultaneously recording one or more first video images of a user or said user's environment from different angles or perspectives; and

simultaneously projecting one or more second video images.

13. A method as claimed in claim 12, further comprising automatically sensing gestures.

14. A method as claimed in claim 12 or 13, further comprising controlling by gestures either: (i) said first video images which are recorded and/or transmitted to other users;

and/or (ii) said second video images which are projected; and/or (iii) a menu, setting or parameter of a device which is arranged and adapted to record said first video images and to project said second video images.

15. A method as claimed in claim 12, 13 or 14, further comprising projecting said second video images onto walls of a room or other surrounding.

16. An interactive video conferencing, social interaction or gaming system comprising: a device;

one or more video cameras integrated into said device, said one or more video cameras being arranged and adapted to record one or more different first video images of a user or said user's environment;

one or more video projectors integrated into said device, said one or more video projectors being arranged and adapted to project one or more second video images onto walls of a room or other surrounding; and

a gesture recognition system for recognising gestures made by said user, wherein said user can control either: (i) said first video images which are recorded and/or transmitted to other users; and/or (ii) said second video images which are projected by said one or more video projectors; and/or (iii) a menu, setting or parameter of said device.

17. A method of interactive video conferencing, social interaction or gaming comprising: providing a device, wherein one or more video cameras are integrated into said device, said one or more video cameras being arranged and adapted to record a plurality of different first video images of a user or said user's environment and wherein one or more video projectors are integrated into said device, said one or more video projectors being arranged and adapted to project one or more second video images onto walls of a room or other surrounding; and

recognising gestures made by said user, wherein said user can control either: (i) said first video images which are recorded and/or transmitted to other users; and/or (ii) said second video images which are projected by said one or more video projectors; and/or (iii) a menu, setting or parameter of said device.