JP2015053051A - System and method for switching screen display between multiple views using gesture - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a system and a method for switching screen display between a video browsing view and a data browsing view.SOLUTION: The system includes a controller, a view presentation module, a screen detection module, and a view switching module. The controller receives data showing the fact that a participant joins a multiuser communication session. The view presentation module visually presents a video stream on a mobile apparatus associated with the participant. The screen detection module determines whether a detection trigger event occurs. The controller receives a video frame image in response to the occurrence of the detection trigger event. The screen detection module detects a data screen included in the video frame image. The view switching module switches a view being displayed on the mobile apparatus from a video browsing view to a data browsing view in response to a gesture performed by the participant.

Description

<Cross-reference of related applications>
This application is entitled “Method of Switching between Views in Mobile Videoconferencing Using Gestures” filed on May 20, 2013, and uses a gesture to switch screen display between multiple views during a mobile video conference. US patent application Ser. No. 61 / 825,482, which claims priority under 35 USC §119 (e), which is incorporated herein by reference in its entirety.

  This document relates to systems and methods for switching between a video browsing view and a data browsing view. In particular, the specification relates to a system and method for switching screen displays between a video viewing view and a data viewing view using gestures during a mobile video conference.

  Existing video conferencing systems collect and transmit data streams along with video and audio streams during video conferencing sessions. This is because, in most business meetings, users expect not only to see each other but also to exchange data information such as documents, presentation slides, handwritten comments, and the like. These data streams are typically captured directly from the computer screen, encoded separately by special coding tools, and screened alongside the video stream at the remote site.

  The explosive growth of mobile devices increasingly drives video conferencing service providers to develop mobile applications such as smartphone and tablet terminal applications. These mobile applications allow users to more easily access video conferencing services from anywhere using mobile devices.

  However, when both the video browsing view and the data browsing view are simultaneously displayed on the mobile device, the following problems may occur. Since the mobile device has a limited screen size, it is impossible to display the video browsing view and the data browsing view side by side at the maximum resolution. Currently, commonly used methods use a user interface that displays only one thumbnail view for the other viewing view while displaying one viewing view on a maximum screen scale. This type of user interface simultaneously displays a screen in a combination of a video browsing view and a data browsing view. Such a user interface cannot provide a centralized user experience by separating the user interface into multiple view display modes, and confuses the user when there are two or more data streams. There is a possibility of coming.

  The present disclosure includes systems and methods for switching screen displays between a video browsing view and a data browsing view on a mobile device. In one embodiment according to the present invention, the system includes a controller, a view presentation module, a screen detection module, and a view switching module. The controller receives data indicating that the participant is participating in a multi-user communication session. The view presentation module visually presents the video stream on the multi-user communication session on the mobile device associated with the participant. The screen detection module determines whether or not a detection trigger event has occurred. The controller receives a video frame image from the video stream in response to the occurrence of the detection trigger event. The screen detection module detects a first data screen in the video frame image. The controller receives data representing a first gesture (gesture gesture) performed toward the mobile device. In response to the first gesture, the view switching module switches a view being displayed on the screen from the video browsing view to the data browsing view in the mobile device. A view presentation module visually presents a first data stream associated with the first data screen on the mobile device.

  Yet another embodiment according to the present invention is implemented as a computer-implemented method, and a method having the following steps is executed as such a method. The method according to the present embodiment receives data indicating that a first participant, a second participant, and a third participant have joined a multi-user communication session. The method visually presents a video stream of a multi-user communication session on a mobile device associated with a third participant. The method includes a video frame image from a first video stream that includes as a source a first device associated with a first participant and a second device associated with a second participant. Receive. The method detects a first data screen from a first device and a second data screen from a second device included in a video frame image. The method receives data representing a first data screen selection operation performed on the mobile device. In response to the selection operation of the data screen, the method switches the view being displayed on the mobile device from the video browsing view to the first data browsing view corresponding to the first data screen. The method visually presents a first data stream on the mobile device.

  Other embodiments for practicing the invention include corresponding methods, systems, apparatuses and computer program products relating to the embodiments described above and other innovative embodiments.

  In the drawings appended hereto, the same reference numerals shown in the figures are used to refer to similar components, which are merely for the purpose of illustrating embodiments of the present invention. It is not intended to limit the technical scope of the invention.

1 is a high-level block diagram illustrating one embodiment of a system for switching between a video browsing view and a data browsing view. FIG. 6 is a high-level block diagram illustrating yet another embodiment of a system for switching between a video browsing view and a data browsing view. The block diagram which shows one Embodiment of a participating application. The figure which shows one Embodiment of the process for performing a data screen detection by a graphical representation. FIG. 4 is a graphical representation of one embodiment for switching between a video browsing view and a data browsing view on a mobile device using a gesture. FIG. 4 is a graphical representation of one embodiment of a graphical user interface showing a video view mode on a mobile device. FIG. 4 is a graphical representation of one embodiment of a graphical user interface showing a data view mode on a mobile device. FIG. 4 is a graphical representation of one embodiment of a graphical user interface showing an embedded data view mode on a mobile device. 6 is a flow diagram illustrating one embodiment of a method for switching between a video browsing view and a data browsing view using gestures during a multi-user communication session. 6 is a flow diagram illustrating yet another embodiment of a method for switching between a video browsing view and a data browsing view using gestures during a multi-user communication session. 6 is a flow diagram illustrating yet another embodiment of a method for switching between a video browsing view and a data browsing view using gestures during a multi-user communication session. 6 is a flow diagram illustrating yet another embodiment of a method for switching between a video browsing view and a data browsing view using gestures during a multi-user communication session. 6 is a flow diagram illustrating one embodiment of a method for switching a display between one of two different data viewing views and a video viewing view using a selection operation performed within a multi-user communication session.

  The system described in this disclosure is particularly advantageous in numerous ways. First, the system allows participants to switch between video browsing and data browsing views using gestures such as pinch gestures, thereby enabling mobile video conferencing sessions. Can provide a consistent and seamless user experience during a multi-user communication session.

  Second, the system can automatically detect the data screen in the video frame image. In this system, before the data screen is switched to the data viewing view that displays the data screen at full resolution, the participant can view the data screen in the video viewing view, so the data stream and video displayed on the data screen are displayed. The data screen allows the participant to understand the relationship between the video content displayed in the browsing view. As a result, this embodiment can avoid a situation in which a participant is confused when there are two or more data streams. For example, when a presenter in a conference room frequently moves between two or more data screens such as a projector screen and a whiteboard screen, the system according to the present embodiment allows a viewer at a distant location to Eliminate confusion caused by gazes moving between the video browsing view and the data browsing view in the projection screen, or frequently between the video browsing view and the data browsing view displayed on the screen on the whiteboard When switching, it is possible to eliminate the confusion caused by the viewer's line of sight at a distant position going back and forth between the screen of the projector and the screen on the whiteboard.

  Third, the system can support an embedded data stream and provide the embedded data stream to the user. For example, the system may present a current meeting data stream visually in a data view mode, but the current meeting data stream is a video clip for displaying the state of the meeting. This video clip embeds the slide and whiteboard stroke information presented during the previous meeting. The system switches from data view mode to embedded data view mode and presents the slide and whiteboard stroke information embedded as described above to the current meeting participant at maximum resolution. can do. The system can have numerous other technical advantages besides this.

  A system and method for switching the screen display between a video browsing view and a data browsing view is described below. In the following description, for the purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the present invention. However, it will be apparent to those skilled in the art that embodiments according to the invention may be practiced without these specific details. In other instances, structures and devices are shown in block diagram form in order to avoid obscuring the description of the invention. For example, the present invention is described in one embodiment below in connection with mobile devices such as smartphones and specific software and hardware. However, this description is equally applicable to any type of computing device that can receive data and commands and any peripheral device that provides services.

  References herein to "one embodiment" or "one embodiment" indicate that a particular feature, structure, or characteristic described in connection with that embodiment is included in at least one embodiment. means. The appearances of the phrase “in one embodiment” in various places in the specification are not necessarily all referring to the same embodiment.

  Some portions of the following detailed description of the present invention are presented in terms of algorithms and symbolic representations of operations performed on data bits within a computer memory. These algorithmic descriptions and representations are used by those skilled in the data processing arts to most effectively convey the substance of their research and development to others skilled in the art. Means. In this specification and in a general sense, an algorithm is considered to be a self-consistent sequential sequence of a series of processing operation steps to derive a desired result. The processing operation steps described above are steps for requesting physical manipulation of physical quantities. Usually, though not necessarily, these physical quantities are in the form of electrical or magnetic signals that can be stored, transferred, combined, compared, and otherwise manipulated. Take. It has proven convenient at times, principally for reasons of common usage of terms, to refer to these signals as bits, values, elements, symbols, characters, terms, numbers, or the like.

  However, it should be noted that all of these terms and similar terms should be associated with the appropriate physical quantities, and all of these terms and similar terms apply to refer to these physical quantities. It should be noted that this is just a convenient indication made. As will be apparent from the discussion below, unless otherwise stated, throughout the description herein, “processing” or “computing” or “calculating” or Discussions made using terms such as “determining” or “displaying” etc. refer to the operation and processing of a computer system (or similar electronic computing device). Understandably, such computer systems operate on data expressed as physical (electronic) quantities in the computer system's registers and in memory, and similarly in the computer system's memory or registers. Converting to other data expressed as physical quantities, or in other similar information storage devices, Others into other data represented as physical quantities as well in the device for transmission / display.

  The present invention also relates to an apparatus for performing the series of processing operations described herein. This device can be specially constructed for the required purpose, or it can be a general purpose computer that is selectively activated or reconfigured by a computer program stored in the computer. It is possible to provide. Such computer programs include, but are not limited to, any type of disk, including but not limited to floppy disk, optical disk, CD-ROM, and magnetic disk, read only memory (ROM). , Random access memory (RAM), EPROM, EEPROM, magnetic or optical card, flash memory including USB key with non-volatile memory, or any type of storage suitable for storing electronic instruction codes Each of the disk devices and storage devices described above can be coupled to a computer system bus.

  Some embodiments of the present invention may be implemented in a variety of forms, such as embodiments that are implemented entirely in hardware, embodiments that are implemented entirely in software, or embodiments that include both hardware and software elements. Can be taken. Preferred embodiments are implemented in software, which includes but is not limited to firmware, resident software, microcode, etc.

  Furthermore, some embodiments according to the invention may take the form of a computer program product accessible from a computer usable or computer readable medium, the computer program product comprising: Alternatively, program code executed by any instruction execution system or program code for use in connection with those devices is provided. For convenience in describing the present invention, a computer usable or computer readable medium is a program executed by or used in connection with an instruction execution system, apparatus, or device. It can be any device that can contain, store, communicate, propagate, or transmit.

  A data processing system suitable for storing and / or executing program code will include at least one processor coupled directly or indirectly to memory elements through a system bus. The memory elements can include local memory, mass storage, and cache memory that are used during actual execution of program code, and the cache memory can be used during program execution. Temporary storage memory that provides at least temporary storage of the program code to reduce the number of times that the program code must be retrieved from the mass storage device. Input / output devices or I / O devices (including but not limited to keyboards, displays, pointing devices, etc.) can be directly coupled to the system or via an intervening I / O controller to the system. It can be bound by any method of being bound.

  A network adapter is further coupled to the data processing system so that the data processing system can be connected to other data processing systems or remote printers or storage devices via an intervening private or public network. It may be possible to be combined. Modems, cable modems, and Ethernet cards are just a few of the currently available types of network adapters.

  Finally, the algorithms and displays presented herein are not inherently related to any particular computer or other device. Various general purpose systems can be used with the programs according to the teachings herein, or it may be convenient to construct more specialized devices to perform the required method steps. It will be understandable to those skilled in the art. The required structure for a variety of these systems will appear from the description below. In addition, the specification is not described with reference to any particular programming language. It will be appreciated that a variety of programming languages may be used to implement the teachings of the various embodiments described herein.

<System overview>
FIG. 1A shows a block diagram of a system 100 for switching a screen display between a video browsing view and a data browsing view according to one embodiment of the invention. The illustrated system 100 includes a hosting device 101 accessible by a host 135, a registration server 130, a camera 103, display devices 107a-107n, and mobile devices 115a-115n accessible by participants 125a-125n. including. In FIG. 1A and the remaining figures, a letter following a reference number, eg, “a” at the end of “115a”, represents a reference to a particular instance among the elements having that reference number. A reference number in the text without a following character, for example “115”, represents a general reference to one or more unspecified instances in a plurality of elements having that reference number. In the illustrated embodiment, these entities of system 100 are communicatively coupled via network 105.

  The network 105 can be a conventional type of wired or wireless network, and its network topology can have many different configurations that can include a star configuration, a token ring configuration, or other configurations. Can have. In addition, network 105 may include a local area network (LAN), a wide area network (WAN) (eg, the Internet), and / or other data transmission paths interconnected so that multiple devices can communicate with each other. May be included. In some embodiments according to the present invention, the network 105 may be a peer-to-peer network. Network 105 may also be coupled to or include, as part, a telecommunications network that transmits data via a variety of different communication protocols. In some embodiments according to the present invention, the network 105 includes a Bluetooth or cellular communication network for transmitting and receiving data, the cellular communication network comprising a short messaging service (SMS). ), Multimedia messaging service (MMS), hypertext transfer protocol (HTTP), direct data connection, WAP, email, etc. Although FIG. 1A shows one network 105 coupled to mobile device 115, hosting device 101, and registration server 130, in practice one or more networks 105 are connected to these entities. It is possible to do so.

  The hosting environment 137 can be an environment for hosting a multi-user communication session. Specific examples of multi-user communication sessions include communication sessions for video conferencing meetings. In some implementations, the hosting environment 137 is a room where all devices in the dashed box in FIG. 1A are visible to the user. For example, the hosting environment 137 may be a conference room environment that includes one or more display devices 107 and one or more cameras 103 present in the conference room. Specific examples of display device 107 include, but are not limited to, projectors, electronic whiteboards, liquid crystal displays, and any other conventional display device. In one embodiment according to the invention, the camera 103 is a high performance video conference camera. Specific examples of the camera 103 include a high-definition (HD) video camera that captures high-resolution video, a pan / tilt / zoom (PTZ) camera that can be mechanically controlled, or a multi-view or panoramic view within the hosting environment 137. This includes but is not limited to a group of cameras that provide a view. Although two display devices 107 and one camera 103 are shown in FIG. 1A, the hosting environment 137 may also include one or more display devices 107 and one or more cameras 103.

  The hosting device 101, the display devices 107a to 107n, and the camera 103 are installed in the hosting environment 137. The hosting device 101 is communicatively coupled to the display device 107 a via the signal line 116, is communicatively coupled to the display device 107 n via the signal line 118, and is communicable to the camera 103 via the signal line 114. Combined. Display device 107a is optionally coupled to registration server 130 via signal line 102, display device 107n is optionally coupled to registration server 130 via signal line 104, and camera 103 is optional. In addition, it is coupled to the registration server 130 via the signal line 112.

  Hosting device 101 may be a computing device that includes a processor and memory, and is coupled to network 105 via signal line 131. For example, the hosting device 101 is a hardware server. In yet another embodiment, hosting device 101 is a laptop computer or a desktop computer. Hosting device 101 is accessed by host 135 (eg, a user who manages the meeting). The hosting device 101 includes a hosting application 109 and a storage device for storing presentations generated by the hosting application 109.

  Hosting application 109 includes software for hosting a multi-user communication session. For example, the hosting application 109 hosts a video conference meeting managed by the host 135 and attended by one or more participants 125 using one or more mobile devices 115. In yet another implementation, the hosting application 109 generates a slide for giving a presentation.

  In one embodiment of the present invention, the hosting application 109 is shared with other participants 125 in one or more data screens on one or more display devices 107 residing in the hosting environment 137. The data to be displayed. Data shared with participants 125 includes text-based documents, web page content, presentation slides, video clips, handwritten comments based on strokes, and / or other user annotations, etc. Including, but not limited to. One or more data screens in the hosting environment 137 are visible via the camera 103. For example, a presentation slide shared with other participants 125 at a distant location is projected onto a wall, and the projection of the presentation slide (or at least a predetermined portion of the projection) is within the field of view of the camera 103. is there. In this case, the camera 103 can capture the projection of the slides of the presentation into one or more video frame images of the video stream.

  In yet another embodiment, when a user in the meeting room writes a comment on the electronic whiteboard, the hosting application 109 controls the camera 103 so that the electronic whiteboard is visible from the camera 103. The position can be moved. In this case, the camera 103 can capture the comments shown on the electronic whiteboard in one or more video frame images in the video stream. The camera 103 captures a video stream that includes a video frame image that depicts the hosting environment 137, which includes a data screen of the display device 107 and / or a video of the data screen of the hosting device 101.

  In an embodiment according to the present invention, the camera 103 transmits a video stream to the hosting device 101, and the hosting device 101 is triggered by one or more of the registration server 130 and the mobile device 115. For this, the video stream is transferred. In yet another embodiment of the present invention, the camera 103 transmits the video stream directly to the registration server 130 and / or transmits the video stream to the mobile device 115 via the network 105. To do. In yet another embodiment, the camera 103 transmits the latest video frame image captured by the camera 103 to the registration server 130 in response to the occurrence of the detection trigger event. The detection trigger event is described in more detail below with reference to FIG.

  In one embodiment according to the present invention, the hosting application 109 or the display device 107 captures a high quality version of the data stream displayed on the data screen of the display device 107. This high quality version of the data stream displayed on the data screen is referred to as the data stream associated with the data screen, and this high quality version displays the content displayed on the data screen of the display device 107 over time. Contains a series of data screen images (eg, screen shot images) to depict. The screen shot image of the data screen as described above depicts the content displayed on the data screen at a specific moment in time. Different screen shot images of the data screen are captured at different instantaneous times, and the plurality of screen shot images at a series of different points in the data screen form a data stream associated with the data screen. In some implementations, each of the screen shot images of the data screen may also be referred to as a data frame included in the data stream.

  In some implementations, the hosting application 109 captures a series of screen shot images representing the slide presentation in high resolution directly from the computing device for presentation. In some additional embodiments, the electronic whiteboard captures the original stroke information displayed on the screen of the electronic whiteboard, and then the electronic whiteboard captures the original stroke information. A screen shot image to be rendered is sent to the hosting application 109.

  In one embodiment according to the invention, the hosting application 109 transmits a data stream associated with the data screen to one or more of the mobile device 115 and the registration server 130. In yet another embodiment, display device 107 directly transmits a data stream including one or more data screen images to one or more of mobile device 115 and registration server 130. For example, the display device 107 periodically transmits the video of the newest data screen to the registration server 130.

  According to an embodiment of the present invention, the participating application 123a can be operated on the registration server 130. Registration server 130 includes a processor and memory and is coupled to network 105 via signal line 106. The registration server 130 includes a database for storing registered images. The registration server 130 registers the display device 107 and receives a feed of video image information related to the meeting from the camera 103. The video video information feed includes one or more video frame images. The registration server 130 executes an image matching algorithm, whereby the correspondence between the latest video frame and the latest screen shot image of the data screen associated with the display device 107 or hosting device 101. Find out. If a match is found, the matched image region is highlighted in the video frame image and displayed on the screen on the mobile device 115. Registration server 130 is described in more detail below with reference to FIGS. 2 and 3A.

  According to yet another embodiment of the present invention, a participating application 123b can be stored in the mobile device 115a, and the participating application 123b is connected to the network 105 via the signal line. The mobile devices 115a and 115b are devices including a memory and a processor, and have a limited screen display space (for example, a laptop computer, a tablet computer, a mobile phone, a smartphone, a personal digital assistant (PDA)). ), A mobile e-mail device, or other electronic device accessible to the network 105). Mobile device 115 includes a touch screen for displaying data and receiving gestures from participant 125. Specific examples of gestures include, but are not limited to, tap operations, double tap operations, long press operations, scroll operations, pan operations, flick operations, two-finger tap operations, pinch / open operations, pinch / close operations, etc. Not.

  In the illustrated embodiment, participant 125a interacts with mobile device 115a. The mobile device 115n is communicatively coupled to the network 105 via the signal line 110. Participant 125n interacts with mobile device 115n. Participant 125 may be a remote user participating in a multi-user communication session, such as a video conference session hosted by hosting device 101. The mobile devices 115a and 115n in FIG. 1A are used as specific examples. Although FIG. 1A shows two mobile devices 115a and 115n, the present disclosure also applies to a system architecture having any number of mobile devices 115 of one or more.

  In one embodiment according to the present invention, the participating application 123 includes a part of the participating application 123 that is partially stored in the mobile devices 115a and 115n, and another part of the participating application 123 that is partially stored in the registration server 130. Stored in a distributed fashion so that they can be stored automatically. For example, while the registration server 130 performs the screen detection step, the participating application 123b in the mobile device 115 serves as a thin client application that displays a video stream or a data stream. The participating application 123b in the mobile device 115a instructs the display device to visually present the video stream or data stream, for example, by rendering the image in a browser. The participating application 123b receives a user input operation (for example, a gesture) from the participant 125, and interprets the meaning of the user input operation. For example, assume that participating application 123b is currently displaying a video stream. Since the participation application 123b receives the user input operation from the participant 125a and greatly enlarges the screen, when the user input operation exceeds the threshold, the participation application 123b displays the stream to be displayed on the screen of the hosting device 101. Determine that it should be switched to the data stream. The participating application 123b transmits a command indicating switching from the video stream to the data stream to the participating application 123a on the registration server 130.

  Participating application 123 may be code and routines for participating in a multi-user communication session. In one embodiment according to the present invention, the participating application 123 may be implemented using hardware including a field programmable gate array (FPGA) or application specific integrated circuit (ASIC). In yet another embodiment, participating application 123 may be implemented using a combination of hardware and software. In various embodiments according to the present invention, the participating application 123 may be stored in a device and server combination or may be stored on either the device or the server.

  FIG. 1B is yet another embodiment of a system for switching the screen display between a video browsing view and a data browsing view. In this embodiment, there is no hosting device 101. Instead, the mobile device 115 may include the camera 103 and the participating application 123b. Mobile device 115 is coupled to display devices 107a and 107n via signal lines 136 and 138, respectively.

  The participant 125 can activate the camera 103 in the mobile device 115 and point the camera 103 toward the display devices 107a and 107n to capture the contents of the display devices 107a and 107n. The mobile device 115 can transmit the image directly to the registration server 130 via the signal line 154. For example, such an image may serve as a query from the mobile device 115. The participating application 123 detects the screen from the video browsing view using the captured image, and switches the screen display to the data browsing view in response to receiving an input operation by the gesture from the participant 125.

<Examples of participating applications>
Referring now to FIG. 2, an example of a participating application 123 is shown in more detail. FIG. 2 is a block diagram of a computing device 200 according to some examples, including a participating application 123, a processor 235, a memory 237, an input / output device 241, a communication unit 239, and a storage device 243. The components of computing device 200 are communicatively coupled by bus 220. Input / output device 241 is communicatively coupled to bus 200 via signal line 242. In some embodiments according to the invention, the computing device 200 can be either the mobile device 115 or the registration server 130. For example, in one embodiment, registration server 130 can include participating application 123 along with some of the components described below, and mobile device 115 includes some of the other components described below. Is possible.

  The processor 235 includes an arithmetic logic unit, microprocessor, general purpose controller, or some other processor array for performing operations and providing electronic display signals to a display device. Processor 235 is coupled to bus 220 via signal line 222 for communication with other components. The processor 235 may process a data signal and include a variety of computing architectures including a complex instruction set computer (CISC) architecture, a reduced instruction set computer (RISC) architecture, or an architecture that implements a combination of instruction sets. . Although FIG. 2 includes a single processor 235, multiple processors 235 may be included. In one embodiment, other processors, operating systems, sensor devices, display devices, and other physical configurations other than those described above may also be employed.

  Memory 237 stores instructions and / or data that may be executed by processor 235. Memory 237 is coupled to bus 220 via signal line 224 for communication with other components. The instructions and / or data may include program code for performing the techniques described herein. The memory 237 may be a dynamic random access memory (DRAM) device, a static random access memory (SRAM) device, flash memory, or some other memory device. In some embodiments, the memory 237 further includes a hard disk drive, a floppy disk drive, a CD-ROM device, a DVD-ROM device, a DVD-RAM device, a DVD-RW device, a flash memory device, Or non-volatile memory or similar permanent storage devices and media including some other mass storage device for storing information in a more permanent form.

  The communication unit 239 sends and receives data to and from at least one of the hosting device 101, the mobile device 115, and the registration server 130 depending on where the participating application 123 is stored. Communication unit 239 is coupled to bus 220 via signal line 226. In some embodiments, the communication unit 239 includes a communication port for direct physical connection to the network 105 or a separate communication channel. For example, the communication unit 239 includes a USB, SD, CAT-5 or similar communication port for wired communication with the mobile device 115. In some embodiments, the communication unit 239 may use one or more wireless communication methods, including IEEE 802.11, IEEE 802.16, BLUETOOTH®, or another suitable wireless communication method to It includes a wireless transceiver for exchanging data with device 115 or other communication channel.

  In some embodiments, the communication unit 239 includes a short messaging service (SMS), a multimedia messaging service (MMS), a hypertext transfer protocol (HTTP), a direct data connection, WAP, e A cellular communications transceiver for transmitting and receiving data via a cellular communications network, including via mail, or yet another suitable type of electronic communications service function. In some embodiments, the communication unit 239 includes a wired communication port and a wireless transceiver. The communication unit 239 also uses other standard network protocols including TCP / IP, HTTP, HTTPS, SMTP, etc. for the delivery of files and / or media objects to other networks 105. Provides a traditional connection function.

  Storage device 243 may be a non-transitory memory that stores data for providing the functionality described herein. The storage device 243 can be a dynamic random access memory (DRAM) device, a static random access memory (SRAM) device, flash memory, or some other type of memory device. is there. In some embodiments, the storage device 243 further includes a hard disk drive, a floppy disk drive, a CD-ROM device, a DVD-ROM device, a DVD-RAM device, a DVD-RW device, a flash memory device. Or non-volatile memory or similar permanent storage devices and media including any other mass storage device for storing information in a more permanent form.

  In the illustrated embodiment, the storage device 243 is communicatively coupled to the bus 220 via signal line 228. In one embodiment, the storage device 243 includes a video stream that includes one or more video frame images, a data stream that includes images of one or more data screens, one or more detection trigger events, etc. Remember one or more of Storage device 243 may store other data for providing the functionality described herein. For example, the storage device 243 may store a copy of the video conference material, such as a presentation, document, audio clip, video clip, and the like.

  In the embodiment illustrated in FIG. 2, the participating application 123 optionally includes a camera adjustment module 212 in addition to the controller 202, view presentation module 204, screen detection module 206, view switching module 208 and user interface module 210. . The plurality of components of the participating application 123 described above are coupled to each other via the bus 220 so as to communicate with each other. Those skilled in the art will appreciate that some of the components described above may be partially stored on the mobile device 115 and the remaining portion of the components described above may be partially stored on the registration server 130. Will recognize that is possible. For example, the participating application 123 stored on the registration server 130 can include the screen detection module 206 and the participating application 123 stored on the mobile device 115 can include the remaining components. .

  The controller 202 may be software that includes routines for handling communications between the participating application 123 and other components in the computing device 200. In one embodiment, the controller 202 is a set of instructions that can be executed by the processor 235 to implement the functions described below to handle communication between the participating application 123 and other components in the computing arrangement 200. Is possible. In yet another embodiment, the controller 202 can be stored in the memory 237 of the computing device 200 and can be accessible and executable by the processor 235. In any embodiment, controller 202 may be adapted to provide cooperation and communication between processor 235 and other components in computing device 200 via signal line 230.

  In one embodiment, the controller 202 sends and receives data to and from one or more of the mobile device 115, the hosting device 101, and the registration server 130 via the communication unit 239. For example, the controller 202 receives a user input operation from the participant 125 who performs an operation on the mobile device 115 via the communication unit 239 and transmits the user input operation to the view switching module 208. In another example, the controller 202 receives graphical data from the user interface module 210 to provide a user interface to the participant 125, transmits the graphical data to the mobile device 115, and receives it. As an opportunity, the mobile device 115 visually presents the user interface to the participant 125.

  In one embodiment, the controller 202 receives data from other components in the participating application 123 and stores the data in the storage device 243. For example, the controller 202 receives data representing one or more detection trigger events from the screen detection module 206 and stores the data in the storage device 243. In yet another embodiment, the controller 202 retrieves data from the storage device 243 and sends the data to other components in the participating application 123. For example, the controller 202 retrieves the data stream from the storage device 243 and sends the data stream to the view presentation module 204 for visually presenting the data stream to the participant 125.

  View presentation module 204 may be software that includes routines for presenting a video browsing view or a data browsing view on mobile device 115. In one embodiment, the view presentation module 204 can be a set of instructions that can be executed by the processor 235 to implement the functions described below for presenting a data viewing view or a video viewing view on the mobile device 115. is there. In yet another embodiment, the view presentation module 204 can be stored in the memory 237 of the computing device 200 and can be accessible and executable by the processor 235. In any embodiment, view presentation module 204 may be adapted to provide cooperation and communication between processor 235 and other components within computing device 200 via signal line 232.

  Video view mode visually presents video data associated with a multi-user communication session to participant 125. For example, the video view mode visually presents the video streams of other participants in the multi-user communication session to the participant 125 in full screen on the mobile device 115. In another example, the video view mode visually presents the video stream on the mobile device 115 at full resolution.

  In one embodiment according to the present invention, the view presentation module 204 receives data from the mobile device 115 associated with the participant 125 that indicates that the participant 125 will participate in the multi-user communication session. In response, the mobile device 115 transitions to the video view mode. The view presentation module 204 receives a video stream including one or more video frame images directly from the camera 103 or via the hosting device 101, and the video stream is received by the display device of the mobile device 115 as a participant. 125 is presented visually.

  In some examples, one or more data screens in the same hosting environment 137 as the camera 103 are captured in one or more video frame images in a video stream, and the one or more video The frame image includes sub-images that depict one or more data screens. For example, the one or more video frame images may include at least a portion of the hosting device 101 data screen, a portion of the screen projected on the wall, and / or a portion of the electronic whiteboard data screen. take in. In another example, the one or more video frame images capture the entire data screen of the hosting device 101, the entire screen projected on the wall, and / or the entire electronic whiteboard data screen.

  Data view mode is a mode in which a data stream associated with a multi-user communication session is visually presented to participant 125. For example, the data view mode visually presents a data stream with slides that are visually presented during the duration of the multi-user communication session to the participant 125 in full screen on the mobile device 115. To do. In another example, the data view mode visually presents the data stream on the mobile device 115 at full resolution.

  In one embodiment of the present invention, the view presentation module 204 should switch the view being displayed on the mobile device 115 from the video viewing view to the data browsing view in addition to the detected data screen identifier (ID). A view switching signal indicating the presence is received from the view switching module 208. In some embodiments, the view presentation module 204 receives a data stream associated with the detected data screen directly from the display device 107 associated with the data screen. In some other embodiments, view presentation module 204 receives a data stream via hosting device 101. In response to receiving the view switch signal, the view presentation module 204 stops the visual presentation of the video stream on the mobile device 115 and the data associated with the data screen on the mobile device 115. Starts the action of visually presenting the stream. Those skilled in the art will recognize that the description describing the visual presentation of a video stream or data stream in this specification is via the communication unit 239 for display. It is intended to represent how the view presentation module 204 instructs the user interface module 210 to generate graphical data that is transmitted to the mobile device 115.

  In some examples, an embedded data stream is included in the data stream. The view presentation module 204 receives from the view switching module 208 a view switching signal that instructs the view presentation module 204 to switch the view being displayed on the mobile device 115 from the data browsing view to the embedded data browsing view. The embedded data view mode is a mode in which the embedded data stream is visually presented to the participant 125 on the mobile device 115 at full resolution or full screen display. In response to receiving the view switch signal, the view presentation module 204 stops the operation of visually presenting the data stream on the mobile device 115 and the operation of visually presenting the embedded data stream. To start. The embedded data stream can be a video conference meeting, presentation, video clip, text document, presentation slide, or other type of data embedded in the data stream.

  The view presentation module 204 receives from the view switching module 208 a view switching signal that instructs the view presentation module 204 to switch the screen display from the embedded data browsing view in which another view is embedded to the normal data browsing view again. To do. The view presentation module 204 then stops the operation of visually presenting the embedded data stream and starts the operation of visually presenting the data stream again on the mobile device 115. In one embodiment, the view presentation module 204 receives a view switching signal from the view switching module 208 that instructs the view presentation module 204 to switch the view being displayed on the mobile device 115 from the data viewing view to the video viewing view. In response to the view switching signal, the view presentation module 204 stops the operation of visually presenting the data stream on the mobile device 115 and starts the operation of visually presenting the video stream.

  The screen detection module 206 may be software that includes routines for performing detection of data screens contained in the video frame image. In one embodiment, the screen detection module 206 can be a set of instructions that can be executed by the processor 235 to implement the functions described below to perform detection of data screens contained in a video frame image. is there. In yet another embodiment, the screen detection module 206 can be stored in the memory 237 of the computing device 200 and can be accessible and executable by the processor 235. In any embodiment, screen detection module 206 may be adapted to provide cooperation and communication between processor 235 and other components within computing device 200 via signal line 234.

  In one embodiment according to the invention, the screen detection module 206 registers one or more display devices 107 with the registration server 130. For example, the screen detection module 206 can record a device identifier, a user, and the like associated with the display device 107 together with the display device 107, and store registration information in the storage device 243. Each display device 107 periodically transmits a newly updated image of its data screen to the registration server 130. For example, each display device 107 periodically transmits its latest screen shot image to the registration server 130. In some examples, the display device 107 sends a newly updated screen shot image of its data screen to the registration server 130 via the hosting device 101.

  In one embodiment according to the present invention, the screen detection module 206 detects the occurrence of a trigger event. For example, such an event can be a detection trigger event that triggers the detection of one or more data screens in the video frame image. For example, the detection trigger event causes the screen detection module 206 to perform an operation of detecting whether the video frame image includes a data screen. Specific examples of the detection trigger event include movement of the camera 103 (for example, pan operation of the camera 103, zoom operation or tilt operation, movement of the camera 103, etc.) and / or movement of the object in the video frame image ( For example, the image of the screen projected on the wall has appeared in the video frame image, or the electronic whiteboard has moved), but is not limited thereto. In yet another example, the trigger event may occur based on a timer.

  In response to the occurrence of the detection trigger event, the screen detection module 206 receives the latest video frame image in the video stream, either directly from the camera 103 or via the hosting device 101. In some examples, the screen detection module 206 receives the latest video frame image in the video stream from the mobile device 115 or a video server that provides the video stream. The screen detection module 206 performs detection of the data screen included in the latest video frame image in response to the occurrence of the detection trigger event. For example, the screen detection module 206 determines whether the data screen appears in the latest video frame image by matching the latest screen shot image of the data screen with the latest video frame image. .

  In some implementations, for each data screen registered with the registration server 130, the screen detection module 206 causes the sub-image that matches the latest screen shot image of the data screen to appear in the latest video frame image. It is determined whether or not. For example, the screen detection module 206 determines whether the latest video frame image includes a sub-image that depicts the data screen (eg, the screen detection module 206 captures the data screen with the latest video frame image. Determine whether or not In a further embodiment, the screen detection module 206 executes an image matching algorithm to find the correspondence between the latest video frame image and the latest screen shot image of the data screen. When the screen detection module 206 finds a match between the latest video frame image and the latest screen shot image of the data screen, the screen detection module 206 will match the region in the video frame image on the mobile device 115. Is highlighted. For example, the screen detection module 206 highlights the detected data screen in the video frame image on the mobile device 115.

  In one embodiment, the screen detection module 206 executes an image matching algorithm in real time. A specific example of the image matching algorithm includes a scale invariant feature amount conversion (SIFT) algorithm. The SIFT algorithm extracts the feature points of both the latest video frame image and the latest screen shot image of the data screen, and the feature points from both images are matched based on the k-nearest neighbor (KNN) method. Using a random sample consensus (RANSAC) algorithm, a consensus is found and a homography matrix is determined. Additional information on how to use SIFT, KNN, and RANSAC for image matching can be found in "Hess, R., An Open-Source SIFT Library, Proceedings of the International Conference on Multimedia, Octoper, Octopus." 1.493-96 "can be understood. One skilled in the art will recognize that other image matching algorithms other than those described above can also be used.

  When the image detection module 206 detects one or more data screens present in the video frame image, the screen detection module 206 performs one or more matchings between the one or more data screens and the video frame image. Generate matching results that contain. The screen detection module 206 notifies the mobile device 115 of one or more matching results, and establishes a direct connection between the mobile device 115 and each display device 107 having one matched data screen. . The screen detection module 206 highlights one or more matched regions in the video frame image, each matching region corresponding to the position of one data screen captured in the video frame image. . The screen detection module 206 displays the highlighted matching area on the mobile device 115.

  In another embodiment according to the present invention, the camera 103 is statically placed and captures one or more data screens within the hosting environment 137 so that the location of the one or more data screens is within the video frame image. It remains unchanged. The screen detection module 206 can determine the presence of one or more data screens based on static placement settings in the hosting environment 137 and pre-calibrate the position of one or more data screens in the video frame image. Can be processed. The screen detection module 206 highlights one or more data screens in the video frame image that are present at pre-calibrated positions in the video frame image.

  The screen detection module 206 sends one or more screen IDs identifying one or more detected data screens and data representing one or more matching regions in the video frame image to the view switching module 208. To send. In yet another embodiment, the screen detection module 206 sends the pre-calibrated positions of one or more data screens to the view switching module 208. In yet another embodiment, the screen detection module 206 stores one or more screen IDs, data representing one or more matching regions, and / or pre-calibrated locations in the storage device 243.

  The view switching module 208 may be software that includes a routine for switching the view being displayed on the mobile device 115 between a video viewing view and a data viewing view. In one embodiment, the view switching module 208 is a set of instructions that can be executed by the processor 235 to implement the functions described below to switch the view being displayed on the mobile device 115 between a video viewing view and a data viewing view. Is possible. In yet another embodiment, the view switching module 208 can be stored in the memory 237 of the computing device 200 and can be accessible and executable by the processor 235. In any embodiment, view switching module 208 may be adapted to provide cooperation and communication between processor 235 and other components in computing device 200 via signal line 236.

  In one embodiment, the view switching module 208 includes one or more detected data in the video frame image in addition to data representing one or more screen IDs that identify one or more detected data screens. Data representing one or more matching regions associated with the screen is received from the screen detection module 206. In the video view mode, the mobile device 115 visually transmits the video stream to the participant 125 with one or more detected data screens highlighted in the matching area of the video frame image. Present. When participant 125 performs a gesture (eg, pinch open gesture or double tap gesture, etc.) within the highlighted matching area of the data screen using the touch screen of mobile device 115, the view switches Module 208 interprets the participant's gesture as a command to switch from the video viewing view to the data viewing view. The view switching module 208 generates a view switching signal representing the command, and transmits the view switching signal to the view presentation module 204. In response to this, the view presentation module 204 sends the data browsing view to the participant 125. It is presented visually. In one embodiment, the view switching module 208 is configured such that a portion of the data screen detected in the video frame image is greater than a predetermined threshold (eg, the majority of the data screen appears in the video frame image. The gesture is interpreted as a command for switching from the video viewing view to the data viewing view.

  For example, if participant 125 can use a gesture to zoom in on the data screen detected in the video frame image, the video viewing view presenting the video frame image accordingly is mobile It is enlarged and displayed on the touch screen of the device 115 (scaled up). When the size of the data screen scaled up as described above in the video frame image reaches a predetermined threshold, the view switching module 208 changes the view being displayed on the mobile device 115 from the video browsing view to the data browsing view. With automatic switching, the view presentation module 204 visually presents on the mobile device 115 a data stream associated with the detected data screen. The mobile device 115 switches from the video view mode to the data view mode accordingly. Participant 125 can perform further operations on the data stream by performing further gestures such as zoom-in to the data stream, copy operation of the data stream, drag operation of the data stream, etc. is there.

  In the data view mode, the mobile device 115 visually presents the data stream to the participant 125. When the participant 125 performs a gesture (e.g., tapping a pinch close gesture or an exit icon) on the data stream displayed on the touch screen of the mobile device 115, the view switching module 208 is The participant's gesture is interpreted as a command for switching from the data browsing view to the video browsing view again. The view switching module 208 generates a view switching signal representing the command, transmits the view switching signal to the view presentation module 204, and the view presentation module 204 uses the video viewing view as the participant 125. To present visually. Again, the screen detection module 206 detects that one or more data screens visible from the camera 103 are present in the video frame image, and highlights one or more data screens in the video frame image. To do. For example, if participant 125 can zoom out of the data stream using gestures, a data browsing view presenting the data stream is accordingly reduced on the touch screen of mobile device 115. (Scale down). When the size of the data stream scaled down as described above reaches a predetermined threshold, the view switching module 208 automatically switches the view being displayed on the mobile device 115 from the data viewing view to the video viewing view, In response, the view presentation module 204 visually presents the video stream on the mobile device 115.

  In the data view mode, if the presented data stream includes an embedded data stream, the participant 125 can perform operations on the embedded data stream using gestures. The view switching module 208 interprets the gesture as a command for switching from a normal data browsing view to an embedded data browsing view. The view switching module 208 generates a view switching signal representing the command, and transmits the view switching signal to the view presentation module 204. In response, the view presentation module 204 receives the embedded data stream. Visually present to participant 125 at maximum resolution. Participant 125 can perform yet another gesture to exit the embedded data browsing view and return to the normal data browsing view. For example, if the data stream contains an embedded video, the participant 125 can issue a tap open operation command for the icon representing the embedded video in data view mode, As a trigger, the view presentation module 204 visually presents the embedded video on the mobile device 115 in full screen display. After viewing the embedded video, the participant 125 may issue a pinch-close operation command that exits the embedded data browsing view and returns to the normal data browsing view.

  The user interface module 210 may be software that includes routines for generating graphical data for providing a user interface. In one embodiment, the user interface module 210 may be a set of instructions that can be executed by the processor 235 to implement the functions described below to generate graphical data for providing a user interface. It is. In yet another embodiment, the user interface module 210 can be stored in the memory 237 of the computing device 200 and can be accessible and executable by the processor 235. In any embodiment, user interface module 210 may be adapted to provide cooperation and communication between processor 235 and other components in computing device 200 via signal line 238. .

  In one embodiment, user interface module 210 receives instructions from view presentation module 204 and provides graphical data for providing a user interface to a user, such as host 135 or participant 125. Generate. The user interface module 210 transmits the graphical data to the hosting device 101 or the mobile device 115, and the hosting device 101 or the mobile device 115 triggers the user interface to the user. Present visually. For example, the user interface module 210 generates graphical data for providing a user interface that depicts a video stream or a data stream. The user interface module 210 transmits the graphical data to the mobile device 115, and the mobile device 115 sends a video stream or data stream to the participant 125 via the user interface. Present visually. In other embodiments, the user interface module 210 may generate graphical data for providing other user interfaces to the user.

  The optional camera adjustment module 212 may be software including routines for adjusting the camera 103. In one embodiment, the camera adjustment module 212 may be a set of instructions that can be executed by the processor 235 to provide the functions described below for adjusting the camera 103. In yet another embodiment, the camera adjustment module 212 can be stored in the memory 237 of the computing device 200 and can be accessible and executable by the processor 235. In any embodiment, camera adjustment module 212 may be adapted to perform cooperation and communication between processor 235 and other components in computing device 200 via signal line 240.

  In one embodiment according to the present invention, the participant 125 may operate the camera 103 using a gesture. For example, participant 125 can perform a gesture and change the viewing angle of camera 103 via a user interface displayed on mobile device 115. The camera adjustment module 212 receives data representing the gesture of the participant, and uses the gesture of the participant for the adjustment operation of the camera 103 such as a pan operation, a tilt operation, a zoom-in operation, or a zoom-out operation of the camera 103. As a command for The camera adjustment module 212 adjusts the posture and position of the camera 103 in accordance with the participant's gesture. Participant 125 adjusts the orientation and position of camera 103 so that camera 103 can capture one or more data screens in the video frame image, thereby allowing one of one or more display devices 107 to The above data screen can be held within the field of view of the camera 103.

  In an example use of the system described herein, a first party (eg, host 135) is in a conference room equipped with a camera 103 and one or more data screens, and a second party ( For example, a video conferencing scenario where a participant 125) participates in a video conference as a mobile user from a remote location using a mobile device 115 such as a smartphone or tablet terminal is included. After participant 125 joins the video conference, participant application 123 receives the video stream from camera 103 and visually presents the video stream to participant 125 via the mobile device 115 touch screen. . Participating application 123 detects one or more data screens captured by the video frame image. Participant 125 issues a gesture such as a pinch-open gesture toward the detected data screen highlighted in the video frame image, and the mobile device 115 receives the video viewing view as a trigger. Switch the screen display from to the data browsing view. The participating application 123 then visually presents the data stream associated with the detected data screen to the participant 125 at maximum resolution. Participant 125 can switch from the data viewing view to the video viewing view again by issuing another gesture, such as a pinch close gesture.

  Yet another example use of the system described herein includes a search application for searching information related to images. For example, the user can capture an image of an advertisement (eg, an automobile brand advertisement) and instruct the search application to search for information related to the advertisement. The advertisement image may include a banner and / or data screen image displaying a commercial video. The search application can instruct the screen detection module 206 to detect a data screen included in the image of the advertisement and identify a product that matches the content displayed in the image of the data screen. . The search application can retrieve information related to the identified product from one or more databases and provide the relevant information to the user. For the systems described herein, other uses other than those described above can also be implemented.

<Graphic representation>
FIG. 3A illustrates one embodiment of a process for performing data screen detection as a graphical representation 300. After participant 125 joins the multi-user communication session using mobile device 115, camera 103 establishes video stream connection 302 with mobile device 115. The camera 103 transmits a video stream to the mobile device 115 via the video stream connection 302, and the mobile device 115 triggers the video stream to the participant 125 in the video view mode. Present the stream visually. The display device 107 registers with the registration server 130 and periodically transmits an updated screen shot image 304 of the data screen associated with the display device 107 to the registration server 130. In the illustrated example, the display device 107 is an electronic whiteboard. In one embodiment, registration server 130 detects a detection trigger event. For example, the registration server 130 detects the movement of the camera 103 such as a pan operation or a tilt operation. The registration server 130 receives the latest video frame image 306 from the camera 103 in response to the detection trigger event. In some examples, registration server 130 receives the latest video frame image 306 from mobile device 115.

  The registration server 130 dynamically detects an active data screen using an image matching method. For example, registration server 130 may use an image matching algorithm between the latest video frame image 306 and the latest screen shot image received from either hosting device 101 or display device 107. Find correspondence. When a matching result 308 between the latest video frame image 306 and the latest screen shot image of the data screen is found, the registration server 130 notifies the mobile device 115 of the matching result 308 and in the video frame image. Highlight the corresponding data screen. For example, the registration server 130 uses the box 310 to highlight the electronic whiteboard data screen included in the video frame image. The display device 107 associated with the data screen establishes a data stream connection 312 with the mobile device 115. Display device 107 may transmit the data stream to mobile device 115 via data stream connection 312.

  FIG. 3B illustrates one embodiment for switching the screen display between a video browsing view and a data browsing view on the mobile device 115 using a gesture as a graphical representation 319. Participating application 123 interprets gestures from participant 125 to achieve a seamless user experience. In this embodiment, the participating application 123 captures the first data stream and the second data stream and transmits them to the mobile device 115 together with the video stream captured from the camera 103. The first data stream includes a high quality screen shot image from a hosting device 101 (eg, a laptop PC) and the second data stream is a display device 107 (eg, an electronic white Including strokes from the board). Both data screens (a laptop PC data screen and a data screen on an electronic whiteboard) are visible from the camera 103. In one embodiment, the screen shot image from hosting device 101 includes an embedded image depicting the data screen of display device 107.

  Initially, a video viewing view is shown on the mobile device 115 to visually present the video frame image 320 to the participant 125. For example, the video frame image 320 is shown at the maximum resolution or full screen of the mobile device 115. As shown in FIG. 3B, both data screens (laptop PC data screen 324 and electronic whiteboard data screen 322) are displayed in the video frame image 320 displayed on the participant's mobile device 115. Visible inside. The participating application 123 intelligently detects the presence of the data screen in the video frame image and notifies the participant 125 of it. For example, the participating application 123 highlights the data screens 322 and 324 in the video frame image 320.

  In phase (1), if the participant 125 wants to obtain further details from the laptop PC data screen 324, he or she will use the laptop PC data shown in the video frame image 320. A gesture 330 can be performed toward the screen 324 to zoom in on the data screen 324 of the laptop PC. Examples of gestures 330 can be pinch gestures or double tap gestures. In response to the gesture 330 being performed, the video viewing view on the mobile device 115 is magnified (scaled up). When the size of the recognized laptop PC data screen 324 reaches a preset threshold, the mobile device 115 automatically switches from the video viewing view to the data viewing view. For example, the view being displayed on the mobile device 115 is a state in which the video frame image 320 is visually presented at the maximum resolution, and the high-quality screen shot image 326 of the data screen 324 of the laptop PC is displayed at the maximum resolution. Switch to the state of visual presentation. Participating application 123 interprets any further pinch or drag gestures performed towards screen shot image 326 as user input operations for screen shot image 326 on laptop PC data screen 324. .

  In the phase (2), the participant 125 performs a gesture 332 such as a pinch gesture toward the screen shot image 326 to zoom out from the data browsing view, and a zoom ratio (scale ratio) by the zoom out operation. ) Reaches a preset threshold, the mobile device 115 switches from the data browsing view to the video browsing view again. Again, participating application 123 visually presents video frame image 320 at maximum resolution on mobile device 115 and detects and marks viewable data screens 322 and 324 in video frame image 320.

  In phase (3), the participant 125 executes a gesture 334 such as a drag gesture toward the highlighted data screen 322 in the video frame image 320, and the mobile device 115 is triggered by the gesture 334. The mobile device 115 enlarges and displays the video browsing view with a scale ratio larger than the threshold amount, and the mobile device 115 starts electronically from the state where the video frame image 320 is displayed at the maximum resolution when the above-described gesture 334 is executed. It switches to the state which displays the data browsing view which shows the screen shot image 328 of a formula white board by the maximum resolution. In phase (4), participant 125 performed a gesture 336, such as a pinch gesture, towards screen shot image 328 to zoom out from the data viewing view and that the amount of zoom out has reached the threshold point Until the mobile device 115 is triggered to switch from the data browsing view to the video browsing view again, the mobile device 115 is caused to perform an operation of reducing the screen display scale of the data browsing view. Again, the mobile device 115 visually presents the video frame image 320 to the participant 125.

  FIG. 4A illustrates one embodiment of a graphical user interface showing a video viewing view on the mobile device 115 as a graphical representation 400. The example user interface shows a video frame image 402 depicting a conference room. Video frame image 402 depicts host 135 and data screen 404 of hosting device 101 projected onto a conference room wall. The data screen 404 includes an embedded data screen 406. When participant 125 performs a gesture on data screen 404 captured in video frame image 402, mobile device 115 switches from the video viewing view to the data viewing view shown in FIG. 4B.

  FIG. 4B illustrates one embodiment of a graphical user interface showing a data browsing view on the mobile device 115 as a graphical representation 420. In this illustrative example, a data stream including a screen shot image of data screen 404 is visually presented on mobile device 115. A data stream is a multi-user communication session that includes an embedded data stream. For example, the data stream is another conference video clip with an embedded slide. An embedded data screen 406 that visually presents the embedded slide is shown in the screen shot image of the data screen 404.

  When participant 125 switches display from the video viewing view shown in FIG. 4A to the data browsing view shown in FIG. 4B, the data stream containing the video clip begins to play. In one embodiment, participant 125 exits the data browsing view shown in FIG. 4B by performing a gesture (eg, a pinch-to-close gesture operation) on the screen shot image of data screen 404. Returning to the video viewing view shown in FIG. 4A. In one embodiment, if the video clip includes embedded presentation slide or embedded whiteboard stroke information, participant 125 may continue to zoom in on the data viewing view. For example, when the participant 125 performs a gesture on the embedded data screen 406, the mobile device 115 switches the display from the data browsing view to the embedded data browsing view shown in FIG. It is possible to visually present the embedded slide.

  FIG. 4C illustrates one embodiment of a graphical user interface showing an embedded data browsing view on the mobile device 115 as a graphical representation 440. In this specific example, the slide shown in the embedded data screen 406 is visually presented to the participant 125. Participant 125 exits the embedded data viewing view by performing a gesture (e.g., pinch-to-close gesture operation) on the screen shot image of embedded data screen 406, and FIG. It is possible to return to the data browsing view shown in FIG.

<Method according to this embodiment>
FIG. 5 is a flow diagram illustrating one embodiment of a method 500 for switching a screen display between a video browsing view and a data browsing view using gestures during a multi-user communication session. In one embodiment, the controller 202 receives data from the mobile device 115 associated with the participant 125 that indicates that the participant 125 has joined the multi-user communication session (step 502). View presentation module 204 visually presents the video stream on the multi-user communication session on mobile device 115 (step 504). For example, the view presentation module 204 instructs the user interface engine 210 to generate graphical data for screen display of the video stream. In one embodiment, the screen detection module 206 determines whether a detection trigger event has occurred. The controller 202 receives a video frame image from the video stream in response to the occurrence of the detection trigger event (step 506). For example, the controller 202 receives the latest video frame image in the video stream from the camera 103. The screen detection module 206 detects the first data screen in the video frame image (step 508). For example, the screen detection module 206 determines that the video frame image is capturing a first data screen.

  The controller 202 receives data representing a first gesture performed on the mobile device 115 (step 510). For example, the controller 202 receives data representing a pinch-to-open gesture operation performed by the participant toward the first data screen in the video frame image. In response to the first gesture, the view switching module 208 switches the view being displayed on the mobile device 115 from the video viewing view to the data viewing view (step 512). View presentation module 204 visually presents a first data stream associated with the first data screen on mobile device 115 (step 514). In one embodiment, the first data stream includes one or more high resolution screen shot images of the first data screen generated by the display device 107 associated with the first data screen.

  6A-6C are a flow diagram illustrating yet another embodiment of a method 600 for switching a screen display between a video browsing view and a data browsing view using gestures during a multi-user communication session. Referring to FIG. 6A, the controller 202 receives data from the mobile device 115 associated with the participant 125 that indicates that the participant 125 has joined the multi-user communication session (step 602). View presentation module 204 visually presents the video stream of the multi-user communication session on mobile device 115 (step 604). The screen detection module 206 registers the display device 107 with the registration server 130 (step 606). Display device 107 includes a data screen for visually presenting a data stream of a multi-user communication session in hosting environment 137. The controller 202 periodically receives the data screen image from the display device 107 (step 608). For example, the controller 202 periodically receives a screen shot image of a data screen from the display device 107.

  The screen detection module 206 detects the occurrence of a detection trigger event (step 610). In response to the occurrence of the detection trigger event, the controller 202 receives the latest video frame image from the camera 103 (step 612). The screen detection module 206 performs an operation of detecting a data screen from the latest video frame image using the latest image of the data screen received from the display device 107 (step 614).

  Referring to FIG. 6B, the screen detection module 206 determines whether a sub-image matching the latest image on the data screen is found in the latest video frame image (step 616). If such a sub-image is found in the latest video frame image, the method 600 proceeds to step 618. Otherwise, method 600 ends the process. Referring to step 618 in FIG. 6B, the screen detection module 206 generates a matching result indicating the matching between the latest image on the data screen and the latest video frame image, and notifies the matching result to the mobile device 115. To do. The screen detection module 206 provides data between the mobile device 115 and the display device 107 associated with the data screen. For example, the screen detection module 206 establishes a direct connection between devices. In one embodiment, the display device 107 can send a data stream associated with the data screen to the mobile device 115 via the direct connection described above.

  Controller 202 receives data representing a first gesture performed by participant 125 on a sub-image depicting a data screen in the video frame image (step 622). Controller 202 receives a data stream associated with the data screen from display device 107 (step 624). The view switching module 208 switches the view being displayed on the mobile device 115 from the video viewing view to the data viewing view in response to the first gesture exceeding the threshold (step 626). For example, the user starts a view with an expanded display size at the center of the screen and moves it by more than half the width of the entire screen. View presentation module 204 visually presents the data stream associated with the data screen on mobile device 115 (step 628).

  Referring to FIG. 6C, the controller 202 receives data representing a second gesture performed by the participant 125 on the data stream (step 630). In response to the second gesture exceeding the threshold, the view switching module 208 switches the view being displayed on the mobile device 115 from the data viewing view to the video viewing view again (step 632). View presentation module 204 visually presents the video stream on mobile device 115 (step 634).

  FIG. 7 illustrates one embodiment of a method 700 for switching a screen display between one of two different data browsing views and a video browsing view using a selection operation performed within a multi-user communication session. It is a flowchart which shows. In one embodiment, the controller 202 receives data indicating that the first participant, the second participant, and the third participant 125 have joined the multi-user communication session (step 702). The view presentation module 204 visually presents the video stream on the multi-user communication session on the mobile device 115 associated with the third participant 125 (step 704). For example, the view presentation module 204 instructs the user interface engine 210 to generate graphical data for displaying the video stream. In one embodiment, the screen detection module 206 determines whether a detection trigger event has occurred. The controller 202 receives a video frame image from a video stream that includes a first device associated with a first participant and a second device associated with a second participant at a source. (Step 706). For example, the controller 202 receives the latest video frame image of the video stream from the camera 103. The screen detection module 206 detects a first data screen from the first device and a second data screen from the second device in the video frame image (step 708).

  The controller 202 receives data representing the selection operation of the first data screen executed on the mobile device 115 (step 710). For example, the controller 202 represents an input operation indicating that the third participant wants to see the first data browsing view, and represents a pressing operation with a finger in the center of the image of the first device. Receive data. In response to the selection operation, the view switching module 208 switches the view being displayed on the mobile device 115 from the video browsing view to the first data browsing view corresponding to the first data screen (step 712). View presentation module 204 visually presents the first data stream on mobile device 115 (step 714).

  The foregoing description of the embodiments according to the present invention has been presented for purposes of illustrating and describing the present invention. The above description is not intended to be exhaustive, nor is it intended to limit the specification to the embodiments as disclosed. Many modifications and variations can be made to the embodiments of the present invention in view of the teachings described above. It is intended that the scope of the embodiments encompassed by the present invention be defined not by the “Detailed Description of the Invention” herein above, but rather by the claims of this application. As will be appreciated by those skilled in the art, embodiments of the present invention may be embodied in other specific forms without departing from the spirit or essential characteristics of the invention. Similarly, specific names and categories of modules, routines, features, attributes, methods, and other aspects are neither required nor significant, and the mechanism or feature that implements the description has different names, categories, and Can have a format. Further, as will be apparent to those skilled in the relevant art, the modules, routines, features, attributes, methods, and other aspects herein are implemented as software, hardware, firmware, or any combination of the three. Can be done. In addition, although a specific example of a component in the present specification is a functional module, when implemented as software, the component may be a stand-alone program, a part of a larger program, and a plurality of separate modules. As a program, as a statically or dynamically linked library, as a kernel loadable module, as a device driver, and / or any other known now or in the future to those skilled in the field of computer programming This method can inevitably be realized. In addition, this description is in no way limited to embodiments in any particular programming language or for any particular operating system or environment. Accordingly, this disclosure is intended to illustrate rather than limit the scope of the invention described herein, and the technical scope of this disclosure is defined in the claims set forth below.

Claims (20)

A computer-implemented method comprising:
The processor receiving data indicating that the participant has joined the multi-user communication session; and
The processor visually presenting a video stream on the multi-user communication session on a mobile device associated with the participant;
The processor receiving a video frame image from the video stream in response to the occurrence of the detection trigger event;
The processor detecting a first data screen in the video frame image;
The processor receiving data representing a first gesture performed by the participant towards the mobile device;
In response to the first gesture exceeding a threshold, the processor switches a view being displayed on the mobile device from a video browsing view to a data browsing view;
Visually presenting on the mobile device a first data stream associated with the first data screen. Detecting the first data screen in the video frame image comprises:
Registering a first display device associated with the first data screen;
Receiving a first image of the first data screen from the first display device;
A detecting step of detecting a second image in the video frame image matching the first image of the first data screen, the second image being captured in the video frame image A detection step, which is yet another image of the first data screen displayed;
Notifying the mobile device of the result of the matching between the first image of the first data screen and the second image in the video frame image;
The method of claim 1, comprising providing data between the mobile device and the first display device.   The method according to claim 1, wherein the detection trigger event includes an event caused by one of a camera movement and an object movement captured in the video frame image. Receiving data representing a second gesture performed on the mobile device;
Responsive to the second gesture exceeding a threshold value, switching the view being displayed on the mobile device from a data browsing view to a video browsing view;
The method of claim 1, further comprising visually presenting the video stream on the mobile device. Detecting a second data screen in the video frame image;
Receiving data representing a third gesture performed toward the mobile device;
In response to the third gesture being above a threshold, switching the view being displayed on the mobile device from a video browsing view to a data browsing view;
5. The method of claim 4, further comprising: visually presenting on the mobile device a second data stream associated with the second data screen.   The method of claim 1, wherein the first data stream includes content presented within the first data screen. The first data stream comprises an embedded data stream;
Receiving data representing a fourth gesture performed toward the mobile device;
The method of claim 1, further comprising visually presenting the embedded data stream on the mobile device in response to the fourth gesture. A system,
One or more processors;
A controller stored in memory and executable by the one or more processors configured to receive data indicating that a participant has joined a multi-user communication session and associated with the participant A controller configured to receive data representing a first gesture performed by the participant towards the device;
A view presentation module stored in the memory and executable by the one or more processors, wherein a video stream of the multi-user communication session is received in response to the participant joining the multi-user communication session. Visually presenting on the mobile device a first data stream configured to visually present on the mobile device and responsive to the first gesture and associated with a first data screen A view presentation module configured to: a screen detection module stored in the memory and executable by the one or more processors, configured to determine whether the detection trigger event has occurred Configured to detect the first data screen in the video frame image. A screen detection module,
A view switching module stored in the memory and executable by the one or more processors, wherein the view being displayed on the mobile device is a video viewing view in response to the first gesture exceeding a threshold. And a view switching module configured to switch from a data browsing view to a data browsing view. The screen detection module includes:
Registering a first display device associated with the first data screen;
Receiving a first image of the first data screen from the first display device;
A detecting step of detecting a second image in the video frame image matching the first image of the first data screen, the second image being captured in the video frame image A detection step, which is yet another image of the first data screen displayed;
Notifying the mobile device of the result of the matching between the first image of the first data screen and the second image in the video frame image;
9. The method of claim 8, configured to detect the first data screen in the video frame image by providing data between the mobile device and the first display device. system.   The system of claim 8, wherein the detection trigger event includes an event caused by either camera movement or object movement captured in the video frame image. The controller is further configured to receive data representative of a second gesture performed toward the mobile device;
The view switching module is further configured to switch the view being displayed on the mobile device from a data viewing view to a video viewing view in response to the second gesture exceeding a threshold value;
The system of claim 8, wherein the view presentation module is further configured to present the video stream at the mobile device. The screen detection module is further configured to detect a second data screen in the video frame image;
The controller is further configured to receive data representing a third gesture performed toward the mobile device;
The view switching module is further configured to switch the view being displayed on the mobile device from a video browsing view to a data browsing view in response to the third gesture exceeding a threshold;
9. The system of claim 8, wherein the view presentation module is further configured to visually present a second data stream associated with the second data screen on the mobile device.   The system of claim 8, wherein the first data stream includes content presented within the first data screen. The first data stream comprises an embedded data stream;
The controller is further configured to receive data representing a fourth gesture performed toward the mobile device;
The system of claim 8, wherein the view presentation module is further configured to visually present the embedded data stream on the mobile device in response to the fourth gesture. A computer-implemented method,
The processor receiving data indicating that the first participant, the second participant, and the third participant have joined the multi-user communication session;
The processor visually presenting a video stream on a multi-user communication session on a mobile device associated with the third participant;
A video frame image from the first video stream in which a first device associated with the first participant and a second device associated with the second participant are included in a source, Receiving by the processor;
The processor detecting a first data screen from the first device and a second data screen from the second device in the video frame image;
The processor receiving data representing a selection operation of the first data screen performed on the mobile device;
In response to the selection operation being performed, switching a view being displayed on the mobile device from a video browsing view to a first data browsing view corresponding to the first data screen;
The processor visually presenting the first data stream on the mobile device. Detecting the first data screen in the video frame image comprises:
Registering a first display device associated with the first data screen;
Receiving a first image of the first data screen from the first display device;
A detecting step of detecting a second image in the video frame image matching the first image of the first data screen, the second image being captured in the video frame image A detection step, which is yet another image of the first data screen displayed;
Notifying the mobile device of the result of the matching between the first image of the first data screen and the second image in the video frame image;
16. The method of claim 15, comprising providing data between the mobile device and the first display device.   Presenting the video stream is performed in response to a detection trigger event that includes an event due to either camera motion or object motion captured in the video frame image. The method of claim 15. Receiving data representing a second gesture performed toward the mobile device;
Responsive to the second gesture exceeding a threshold value, switching the view being displayed on the mobile device from a data browsing view to a video browsing view;
16. The method of claim 15, further comprising visually presenting the video stream on the mobile device. The processor detecting a second data screen in the video frame image;
Receiving data representing a third gesture performed toward the mobile device;
In response to the third gesture being above a threshold, switching the view being displayed on the mobile device from a video browsing view to a data browsing view;
19. The method of claim 18, further comprising: visually presenting a second data stream associated with the second data screen on the mobile device.   The method of claim 15, wherein the first data stream includes content presented within the first data screen.