JP7206481B2 - Information processing device, control method and program for information processing device - Google Patents

Description

The present invention relates to an information processing device, an information processing system, an information processing method, and a program.

In an electronic conference, a function is provided in which a terminal of a certain participant is used as a host terminal and the screen of the host terminal is shared with terminals of other participants (guest terminals) (for example, Patent Document 1).

JP 2013-029922 A

During screen sharing, screen information may be shared unintentionally or in an unintended state. For example, in a mechanism for capturing a window (screen) being shared and sharing it with another terminal, there is a case where another window B overlaps the window A being shared.

In this case, an image in which window A and window B overlap is shared as the screen of window A.

Also, if a portion of Window A protrudes from the desktop screen (monitor), the portion protruding from the monitor is not captured. A part of the window A (image of only the part moved to the monitor) is shared with the user's PC.

An object of the present invention is to provide a mechanism for properly sharing selected windows .

The information processing device of the present invention includes:
receiving means for receiving a selection of a window to be shared with another device among the windows displayed by the display means;
In response to the acceptance of the selection by the acceptance means, at least one of the position and size of the window to be shared on the display means is displayed on the display means before the selection. a control means for controlling to change from
transmission control means for controlling to transmit the display contents of the shared window whose position and/or size have been changed by the control means so as to be shared by the other device;
characterized by having

According to the present invention, it is possible to provide a mechanism for appropriately sharing a selected window .

It is a figure showing an example of a system configuration in an embodiment of the present invention. It is a block diagram showing an example of hardware constitutions of various devices in an embodiment of the present invention. It is a figure showing an example of functional composition of various devices in an embodiment of the present invention. 4 is a flow chart showing the flow of camera image transmission/reception and display processing in the embodiment of the present invention. It is a figure which shows an example of various data structures in embodiment of this invention. It is a figure which shows an example of various data structures in embodiment of this invention. It is a figure which shows an example of a structure of various screens in embodiment of this invention. 4 is a flow chart showing the flow of processing in the embodiment of the present invention; 4 is a flowchart showing details of processing for adjusting the position and size of an application window in the embodiment of the present invention; 4 is a flowchart showing details of image sharing processing of an application window in the embodiment of the present invention; 4A and 4B are diagrams showing various screen configurations and how an application window is selected in the embodiment of the present invention; FIG. It is a figure which shows an example of a structure of various screens in embodiment of this invention. It is a figure which shows an example of a structure of various screens in embodiment of this invention. It is a figure which shows an example of a structure of various screens in embodiment of this invention. It is a figure which shows an example of a structure of various screens in embodiment of this invention. It is a figure which shows an example of a structure of various screens in embodiment of this invention. It is a figure which shows an example of a structure of various screens in embodiment of this invention.

An example of an embodiment of the present invention will be described below with reference to the drawings.

First, with reference to FIG. 1, an example of the configuration of an information processing system according to an embodiment of the present invention will be described.

In the information processing system of the present invention, for example, client terminals 100 (100A, 100B, 100C, and 100D), which are information processing apparatuses to which personal computers can be applied, and a server 200 that enables a web conference to be used by each client terminal 100 are networked. communicatively connected and configured via 150; The server 200 is one of information processing devices.

The server 200 is a server device for realizing a web conference using a web browser. A conference screen (interface/information for displaying a screen expressed in html, for example) is transmitted to each client terminal 100 and displayed.

By providing the conference screen (providing a camera image displayed on the conference screen and a screen sharing function), a service for conducting a web conference by a plurality of client terminals 100 is provided.

The camera image here is an image captured by the camera used by the client terminal 100 to which the user ID of the user participating in the conference is input and transmitted to the server 200. For example, the user An image of your own face.

The application image referred to here is an image transmitted from the client terminal 100 to the server 200, for which designation is received through the input unit of the client terminal 100 in accordance with a user's operation. For example, it is a desktop screen of the client terminal 100, an image of an application screen (application window) running on the client terminal 100, or the like.

The server 200 transmits and shares the application image to other client terminals 100 of other users participating in the same conference as the user of the client terminal that sent the application image.

In other words, the web conferencing system of the present embodiment displays screen images displayed on the client terminal 100 (eg, 100A) on the screens of other client terminals (eg, 100B, 100C, 100D). application image display sharing).

For example, only the screen of an application selected by user operation can be shared as an application image.

Also, by accepting designation of the desktop screen displayed on the client terminal 100, the entire desktop image can be shared as an application image (desktop sharing).

The client terminals 100 are, for example, desktop PCs such as 100A, notebook PCs such as 100B and 100D, and tablet PCs such as 100C. The client terminal 100 displays images and videos (camera images) similarly received from the server 200 and application images on the conference screen received from the server 200 .

In this embodiment, it is assumed that a web browser for accessing the server 200 and a dedicated module are installed in the client terminal 100 . Note that this dedicated module is, for example, an ActiveX component downloaded from the server 200 via a web browser.

Note that when transmitting a moving image of itself in a web conference, a camera 101 (such as a web camera), which is a video device, is connected. The microphone 102 is connected to transmit voice, and the speaker 103 is connected to listen to the voice of the other party. Microphone 102 and speaker 103 are audio devices.

Also, in FIG. 1, each device is exemplified as separate housings, but devices 101 to 103, for example, may be provided in hardware of the same housing.

For example, it is a device with a camera function and a microphone function, a microphone function and a speaker function, and all functions.

In addition, if the client terminal 100 incorporates devices 101 to 103, these built-in devices may be used to input/output images, sounds, and the like.

In addition, this web conference system can generate, store, and manage a plurality of virtual conference rooms called rooms in the server 200 .

The user who operates the client terminal 100 uses his/her own user ID and password to access the server 200 via a web browser, logs in, and uses his/her user ID to enter the room. You can participate in meetings that are held. The above is the description of FIG.

Next, with reference to FIG. 2, an example of hardware configuration applicable to various devices according to the embodiment of the present invention will be described.

In FIG. 2, reference numeral 201 denotes a CPU that controls devices and controllers connected to a system bus 204 in an integrated manner. The ROM 202 or the external memory 211 also stores a BIOS (Basic Input/Output System), which is a control program for the CPU 201, an operating system (OS), and functions necessary for realizing the functions executed by each server or each PC. Various programs and the like, which will be described later, are stored.

A RAM 203 functions as a main memory, a work area, and the like for the CPU 201 . The CPU 201 loads programs and the like necessary for executing processing from the ROM 202 or the external memory 211 to the RAM 203, and executes the loaded programs to realize various operations.

An input controller 205 controls input from the input device 209 . The input device 209 includes a keyboard, a touch panel, and a pointing device such as a mouse.

If the input device 209 is a touch panel, the user can give various instructions by pressing (touching with a finger or the like) an icon, cursor, or button displayed on the touch panel. Also, the touch panel may be a touch panel, such as a multi-touch screen, capable of detecting positions touched by multiple fingers.

A video controller 206 controls display on a display device such as the display 210 . Note that the display 210 refers to a display device such as a CRT, a liquid crystal display, or the like. Alternatively, it may be another display device. It should be noted that the display of the notebook computer integrated with the main body is also included. The external output device is not limited to a display, and may be a projector, for example.

Further, when the device to which the hardware configuration of FIG. 2 is applied is a device capable of receiving the touch operation described above, the display 210 provides a function as the input device 209 .

A memory controller 207 is connected to a hard disk (HDD), flexible disk (FD), or PCMCIA card slot for storing boot programs, various applications, font data, user files, edit files, various data, etc., through an adapter. It controls access to an external memory 211 such as a compact flash (registered trademark) memory that is

A communication I/F controller 208 connects and communicates with an external device via a network, and executes communication control processing in the network. For example, communication using TCP/IP is possible.

It should be noted that the CPU 201 enables display on the display 210 by, for example, performing outline font expansion (rasterization) processing on a display information area in the RAM 203 . The CPU 201 also allows the user to issue instructions using a mouse cursor (not shown) on the display 210 .

Various programs described later for realizing the present invention are recorded in the ROM 202 or the external memory 211, and are executed by the CPU 201 by being loaded into the RAM 203 as necessary.

Further, definition files, various information, tables, and the like used when executing the above programs are also stored in the external memory 211, and detailed description of these will be given later.

A camera controller 212 controls image acquisition by the camera 214 and the like, and an audio controller 213 controls audio acquisition and output to the microphone/speaker 215 and the like.

A camera 214 is a camera built into the device to which the hardware configuration of FIG. 2 is applied. A microphone/speaker 215 is a microphone and speaker built into the device to which the hardware configuration of FIG. 2 is applied. The above is the description of FIG.

Next, with reference to FIG. 3, an example of functional configurations of various devices in the embodiment of the present invention will be described.

The client terminal 100 comprises a conference client application 310 . The conference client application 310 also includes a camera image storage control unit 311, a display control unit 312, a transmission image storage unit 313, an image acquisition unit 314, a position determination unit 315, a size/shape determination unit 316, a transmission control unit 317, and the like. I have.

The client terminal 100 also includes camera driver software, and the driver sequentially stores camera images captured by the camera in a storage section managed by the driver on the memory of the client terminal 100 .

The camera image storage control unit 311 reads and acquires the camera image stored in the memory, and the transmitted image storage unit 313 is a storage unit that stores the image to be transmitted to the server managed by the conference client application 310 itself. (Transmission image queue 530/FIG. 5).

Since the camera 101 continuously performs imaging processing (capturing a moving image and inputting the moving image to the client terminal 100), the image queue stores a plurality of images constituting the moving image, one after another for each frame in the moving image. remembered.

The image acquisition unit 314 is a processing unit that acquires an image of an application window displayed on the client terminal 100 (an application image that is an image of an application screen). The image is acquired by, for example, requesting the OS.

The position determining unit 315 determines and moves the position of the application window selected as the target for sharing the display with another terminal to a position suitable for the sharing. For example, in order to obtain a captured image of the entire window, a position where the entire window fits on the monitor is specified, and the window is moved to that position.

The monitor here is a screen displayed on the entire display device. It is assumed that two monitors are stored and managed by the client terminal 100 when dual display (multi-display) is applied using the extended display (extended monitor) function.

The size/shape determination unit 316 determines the size and shape of the application window selected as a target for sharing the display with other terminals to a size and shape suitable for the sharing, and transforms the window. For example, the size and shape of the window are specified and deformed so that the entire window can be captured on the monitor.

The transmission control unit 317 is a processing unit that controls transmission to the server 200 of a camera image (video) captured and input by the camera 101 and an image of an application window selected as an image to be displayed shared by a user operation or the like. .

The transmitted image is received by the reception control unit 322 of the server 200 and stored in the distribution image storage unit 323 (image queue) on the memory.

The server 200 includes a conference information storage unit 321, a reception control unit 322, a distribution image storage unit 323, a distribution control unit 324, and the like. Each functional unit is a function provided by the conference server application 320 .

The conference information storage unit 321 is a storage unit that stores which user has entered which conference room and is having a conference with which user.

The reception control unit 322 is a reception unit that receives camera images (video), application images, and audio transmitted from the transmission control unit 317 of the client terminal 100 .

The distribution image storage unit 323 is a storage unit that stores camera images (video) received by the reception control unit 322, application images, and audio.

The distribution control unit 324 distributes the images, application images, sounds, etc. stored in the distribution image storage unit 323 to those who are participating in the same conference (eg, are in the same conference room), and when the images and sounds are transmitted. It is a control unit that transmits (distributes) to other client terminals 100 different from the original client terminal 100 .

In this embodiment, the client terminal 100 requests the server 200 to acquire an image from the distribution image storage unit 323, and the distribution control unit 324 of the server 200 responds to the request. It is assumed that the latest image in the queue at the time of accepting the request is transmitted to the client terminal 100 .

After receiving the requested image, the display control unit 312 of the client terminal 100 displays the received image (one frame in the moving image) on the display screen. In parallel with the display, the server 200 is requested for the next image in the moving image. The request and display of the image are repeated until the video display ends, such as when the meeting ends.

Further, when at least a part of the application window selected as a display sharing target is not acquired, the transmission control unit 317 stops transmission of the image of the application window after the at least part is no longer acquired. .

When at least part of the application window is not obtained, all or part of the application window protrudes from the monitor to which the application window belongs, and the captured image of the application window with the protruding part painted black is obtained. is the case. 1510 in FIG. 15 shows an example of the screen when the application window 1104 protrudes from the monitor.

Also, even if all or part of the application window is hidden by other windows, it is determined that at least part of the application window is not acquired. The position and Z order of the other windows are obtained by inquiring of the OS. An example of the list is shown at 630 in FIG.

Note that the character conversion candidate selection window displayed when characters are input is also one of the other windows.

A screen example when the application window 1104 is hidden by other windows is shown in 1500 in FIG. In 1500 of FIG. 15, part of the application window 1104 is hidden by the window of the spreadsheet application.

Then, until the image of the entire application window is obtained, the image of the application window immediately before it protrudes from the screen or is hidden by another window is repeatedly transmitted to the server 200 .

Since the server 200 sequentially distributes the images to the other client terminals 100, the images of the application window showing the entire image are continuously reproduced and displayed like still images on the other client terminals 100. FIG.

For example, even if you scroll the display contents or switch the page of the presentation material, on the terminal to which the application image is sent, before the application window protrudes from the monitor, or before another window overlaps the application window, The image of the entire application window is continuously displayed.

In other words, when the application window protrudes or another window overlaps the application window, even if the display in the application window is switched, the display switching is not reflected on the other client terminals 100. .

The client terminal 100, which is the transmission source of the image of the application window, displays to notify the user whether or not the display of the application window is reflected on the destination client terminal 100 in real time. An example is shown in 1201 in FIG. 12 and 1501 in FIG. The above is the description of FIG.

Specific contents of processing in the present invention will be described with reference to FIGS. 4 to 17. FIG.

The CPU 201 of the client terminal 100 or the server 200 executes the processing of each step of the flow charts shown in each drawing using the functions of the functional units of each device.

First, camera image transmission processing of the electronic conference system (information processing system) of the present invention will be described with reference to FIG. The processing of the client terminal 100 in FIG. 4 can be executed by any of the client terminals 100A to 100D. In the description of FIG. 4, as an example, it is assumed that the client terminal 100A executes.

In order to enjoy the web conference service provided by the server 200, the client terminal 100A accesses a predetermined URL according to a user's operation on the browser software and displays the login screen 700 of FIG. 7 (step S401).

Then, by accepting the input of the user ID and password on the login screen 700 and accepting the pressing of the "login" button, a login request to the server 200 is transmitted (step S402).

The server 200 accepts the login request (step S403), compares it with user information (user ID and password/illustrated at 500 in FIG. 5) pre-stored in the external memory of its own device, and performs authentication processing (login processing). (step S404).

If the login to the server 200 is successful, screen information (for example, html information) for the menu screen 710 shown in FIG. 7 is generated and transmitted to the client terminal 100A (step S405).

The menu screen 710 includes a menu display portion 713 , a “logout” button, a conference room display area 714 , and “participate” buttons 711 and 712 .

The client terminal 100A receives the screen information and displays the menu screen on the display screen based on the received screen information (step S406).

The client terminal 100A receives the selection of the conference to participate in (the room (meeting room) where the conference to participate in is held), and transmits a participation request to the conference to the server 200 (step S407).

Specifically, when the press of the “participate” button 701 is accepted, a request to enter the conference room 1 is transmitted to the server 200 . Further, when the pressing of the “participate” button 702 is accepted, a request to enter the conference room 2 is transmitted to the server 200 .

The number of "participate" buttons increases or decreases according to, for example, the number of conference rooms.

In response to the room entry request, the client terminal 100A sends information such as the user ID of the logged-in user who is logging in with the client terminal 100A, the identification information (room ID) of the room to be entered, and the identification information (for example, IP address) of the client terminal 100A. , and the request is transmitted to the server 200 .

When the CPU 201 of the server 200 receives the room entry request (step S408), it executes room entry processing for allowing the user to enter the requested room (conference room) (step S409). Specifically, as a room entry process, a process of storing which user has entered which conference room with which terminal is performed.

Specifically, the conference room ID 511 of the room entry information 510 (FIG. 5) is the conference room ID of the conference room requested to enter, and the user ID of the user to enter the room is the participant ID 512 of the room entry information 510 (FIG. The terminal ID 513 of the room entry information 510 (FIG. 5) stores the ID of the client terminal 100A that sent the user ID (or requested to enter the room). The room entry information 510 is stored and held in the external memory 211 of the server 200 for each conference room.

Note that different meetings are held in the meeting room at different times. What kind of conference is to be held in which conference room is received by user operation via a conference reservation screen (not shown) displayed on the client terminal 100A, and the server 200 stores the received contents in the conference information 520 of FIG. memorize to

The conference information 520 is a table that stores and manages which conference (conference ID 523) will be held at which time 522 in which conference room (conference room ID 521). The table is stored in the external memory 211 of the server 200. FIG.

A request to enter a conference room is a request to participate in a conference being held at the time when the request to enter the conference room is made.

When the room entry process is completed, the server 200 transmits the screen information (FIG. 7/for example, html information) of the conference screen 720 displaying the camera image captured by the camera of the client terminal 100A to the client terminal 100A (step S410). .

The client terminal 100A receives the screen information and displays the conference screen based on the received screen information (step S411/displayed by the display control unit 312).

After that, the processes of steps S417 to S419 are executed and continued until the conference ends. The processing of steps S417 to S419 will be described later.

Also, the client terminal 100A activates the camera (the camera 101 connected to itself or the camera 214 built into the device itself) and starts capturing an image (step S412). Assume that the camera 101 is activated here.

After that, the processes of steps S413 and S414 are repeatedly executed and continued until the conference ends. The processing of steps S413 and S414 will be described later.

The client terminal 100A receives an input of a camera image (video) from the camera 101 of its own device and activated in step S412, thereby acquiring the camera image captured by the camera 101 and displaying it on the conference screen. .

For example, an image captured by the own camera is displayed in an image display area 723 on the conference screen 720 .

The screen layout of the conference screen 720 in FIG. 7 is a layout in which four image display areas 723 are arranged.

The name of the sender of the image displayed in the image display area 723 is displayed in the participant name display portion 722 . For example, the user name of the user who has logged into the server 200 at the client terminal 100 that has transmitted the image is displayed.

The button display area 721 displays an “application share” button for starting sharing of the application image, a “leave” button 714 for leaving the conference (leaving the conference room), and the like.

In addition, the client terminal 100 displays the input image (video) from the camera 101 on the conference screen, while performing compression processing of the image (video) by the video codec, and sequentially compresses the compressed image frame by frame. Each image is stored in the image queue on the memory of the device itself (step S413).

An example of the image queue is shown in the transmission image queue 530 of FIG. The transmission image queue is a queue for sequentially storing images to be transmitted to an external device.

As for the frame rate of the image when storing and transmitting, the value shown in the frame rate 570 of FIG. 5 is stored in the memory, and follows this. The frame rate value is transmitted/distributed from the server 200 to the client terminal 100 when participating in the conference.

The image queue for transmission 530 contains an identification number (identification information) for each image, an identification number 531 (sequence number) indicating the order of the images, and the substance of the image (image data/one frame image in video). It consists of an item of a certain image 532 .

The client terminal 100 assigns an identification number to the image of each frame of the video image acquired from the camera 101 in the order in which the image was acquired, and stores the image in the transmission image queue 530 .

In step S414, the client terminal 100A refers to the camera function ON/OFF setting 580, and if the camera function is ON, the image saved in the transmission image queue 530 is transmitted.

The ON/OFF setting 580 of the camera function can be freely changed by user operation via a setting screen (not shown) displayed on the client terminal 100A.

The client terminal 100A transmits the information shown in the transmission image queue 530 to the server 200 (step S414).

Specifically, using data compression processing by a video codec, the image is sequentially transmitted together with the identification number at a high frame rate with a small data size.

The server 200 receives the image (compressed data) and converts it into image data using, for example, data decompression processing using a video codec (step S415).

Then, the server 200 stores and manages the received images for each user (step S416).

Specifically, in the user-by-user image queue 540 (shown in FIG. 5) held in the memory of the own device, the image is added to the record of the user logged in to the server 200 at the client terminal 100 that sent the image. and memorize it.

The user-by-user image queue 540 includes a user ID 541 , an image 543 and an identification number 542 of the image, and a transmitted user 544 .

The user-by-user image queue 540 is generated by the CPU 201 of the server 200 for each user who entered the conference room (for each user ID 541 who is a participant in the conference) when processing for entering the conference room by the user is performed. generated and held in memory.

In the identification number 542 of the user-specific image queue 540, the identification number 531 of the transmission image queue 530 received from the client terminal 100 is stored. The same information as the image 532 in the image queue 530 is associated and stored.

User IDs of users who have already transmitted the image 543 are stored in the transmitted user 544 .

In step S417, the client terminal 100A requests acquisition of images of other conference participants (images in the user-specific image queue 540 of the user IDs of other participants in the same room at the time of execution of step S417). to the server 200 (step S417).

That is, it requests images of other users participating in the same conference.

The CPU 201 of the server 200 receives the acquisition request (step S420), and refers to the room entry information 510 for the user ID of the user who is in the same room as the user, other than the user who is logged in at the client terminal 100 of the request source. Then, the user-by-user image queues 540 of the other identified occupants are identified.

Then, among the images in the user-specific image queue 540 of the identified other occupants, the image of the latest identification number (the last number) is acquired from each user-specific image queue 540, and the acquired image of each user is obtained. (a camera image captured by the camera of the client terminal 100) is transmitted to the requesting client terminal 100 together with the user ID corresponding to the image (step S421).

In other words, images of other users participating in the same conference are transmitted.

Thereafter, in the user-by-user image queue 540, the user ID of the user to whom the image 543 has been transmitted is stored in the transmitted user 544 (step S422).

The client terminal 100A receives the image (compressed data) transmitted in step S421, and stores it as a display image in a queue for each user (the user indicated by the user ID 551) who transmitted the image. An example of the queue is shown in display image queue 550 in FIG. 5 (step S418).

The display image queue 550 is generated and stored for each user (the user indicated by the user ID 551 ) who sent the received image 554 . The identification number 552 is the sequence number of the image 553 .

That is, the user ID 551 of the display image queue 550 has the same value as the user ID 541 as the user-specific image queue 540, and the identification number 552 of the display image queue 550 has the same value as the identification number 542 of the user-specific image queue 540. is stored. Further, an image that is the same as the image 543 of the user-specific image queue 540 is additionally registered in the image 553 of the display image queue 550 each time the image is received.

If the user of the transmitted image is a user not assigned to the image display area 723, the client terminal 100A performs a process of assigning the user to the image display area 723 not assigned to the user. Display area information 560 shows an example of a table for managing allocation status.

The display area information 560 is pre-stored in the external memory of the client terminal 100 . The ID of the image display area 723 is stored in the display area 561 . The user ID 562 stores the user ID of the user assigned to each display area.

The client terminal 100A restores the video (image) by decompressing the received image of each user using a video codec, and assigns the restored image of each user to each user indicated in the display area information 560. It is displayed in the display area 561 (723 in FIG. 7) (step S419).

The client terminal 100A updates the display of each user's image (camera image) by displaying the received image in the display area corresponding to the user of each image, and reproduces and displays it as a moving image (video). .

When conducting a web conference, there may actually be multiple client terminals 100 (100A to 100D, etc.). Therefore, the server 200 executes, for example, steps S415 and S416 and steps S420 and 421 for the number of users participating in the conference (eg, the number of users 001 to 004 participating in the client terminals 100A to 100D). It will be done.

Explain closing a meeting. For example, it is determined that the conference has ended for the logged-in user when an operation of pressing the “Exit” button 724 on the conference screen 720 is accepted.

In addition, if the end time of room use is stored in the server 200, the server 200 forcibly terminates the web conference at the end time, and the client of the user who was participating in the forcibly terminated conference notifies that the forcible termination has occurred. The terminal 100 is notified. The client terminal 100A determines that the conference has ended when the notice of forced termination is received. The above is the description of FIG.

Next, with reference to FIG. 8, the flow of processing in the embodiment of the present invention will be described. The processing of each step is executed by the CPU 201 of the client terminal 100 using the function of each functional unit.

The process of FIG. 8 can be executed by any client terminal 100 of 100A-100D. In the description of FIG. 8, as an example, it is assumed that the client terminal 100D executes the process.

It is assumed that the conference screen 720 is already displayed at the start of step S801.

The client terminal 100 starts the process of step S801 in FIG. 8 after executing the camera image transmission and reception processes in FIG.

In step S801, the client terminal 100D accepts pressing of the “application selection” button 1105 (FIG. 11) on the conference screen displayed by the display control unit 312. FIG.

Reference numeral 1100 in FIG. 11 denotes the desktop screen of the client terminal of the user of the application selection function (user 004 in this embodiment). In this embodiment, 1100 indicates one monitor.

An “application selection” button 1105 is included in the conference screen 1101 . Since the image display area 1106 is similar to the image display area 723, the description thereof is omitted.

The display control unit 312 of the client terminal 100D receives the activation instruction of the application selection function by receiving the pressing of the "application selection" button 1105 (step S801).

The display control unit 312 displays the application selection window 1102 (FIG. 11) (step S802). The application selection window 1102 is a window for selecting (designating) and displaying a screen of an application whose screen is to be shared with terminals of participants in the same conference.

An application selection window 1102 displays a clip-type object 1103 used to select an application.

The transmission control unit 317 of the client terminal 100D stops transmitting camera images to the server 200 (step S803). That is, the process of sharing the camera image with other client terminals 100 is stopped.

Specifically, the software module (camera image storage control unit 311) that outputs and stores the camera image in the image queue for transmission is instructed to stop the output process, and the module outputs the image queue. By stopping the processing, the next camera image is no longer stored in the transmission image queue. As a result, the camera image cannot be acquired from the transmission image queue, and the transmission of the camera image is stopped.

The transmission control unit 317 acquires a still image indicating "no image input" preliminarily stored in the external memory of the client terminal 100D itself, and stores it in the image queue for transmission instead of the camera image. That is, instead of the camera image (as the camera image), the process of transmitting the still image to the server is started (step S804).

An example of a still image indicating "no image input" is shown at 1711 in FIG. Reference numeral 1710 in FIG. 17 denotes a desktop screen of the client terminal 100 (in this embodiment, the client terminal 100C operated by the user 003) whose application selection function is not activated.

Through this processing, an image such as 1711 in FIG. 17 is displayed as a camera image by another device other than the client terminal 100D, until the transmission of the image of the application screen (application image) is started by the processing of step S807. is transmitted to the user's client terminal 100 (eg, 100C). The transmission process is executed in steps S414 and S421 of FIG.

The client terminal 100D accepts selection of an application screen for sharing an image instead of a camera image (step S805).

Specifically, by accepting an operation to overlap the clip-type object 1103 displayed in the application selection window 1102 on the application window 1104 displayed on the desktop screen, the screen displayed in the window of the application can be selected. judged to have been accepted. FIG. 11 shows how this operation is performed.

Next, the client terminal 100D executes the selected application screen position/size adjustment process (step S806). Then, sharing processing for sharing the image of the application window (image of the application screen) with other terminals is executed (step S807).

Details of the determination process will be described later with reference to FIG. Details of the sharing process will be described later with reference to FIG.

Here, the details of the determination process will be described with reference to FIG.

The client terminal 100D refers to the aspect ratio setting 610 (FIG. 6) of the application image (step S901).

The setting is a setting value set on an aspect ratio designation screen (not shown) (such as a fixed aspect ratio setting screen), and is stored in the external memory of the client terminal 100D.

When "4:3" is set, it is determined that the aspect ratio is fixed to 4:3 vertically. On the other hand, if "Null" is set, it is determined that the aspect ratio is not set to be fixed.

The size/shape determining unit 316 of the client terminal 100D determines and changes the window size and shape of the application selection window 1102 or the application window 1104 according to the setting of whether or not the aspect ratio is fixed (step S902).

If the aspect ratio is not set, the size and shape of the application selection window 1102 are changed according to the window size and shape of the selected application window 1104 .

The size, shape, and position of each window are acquired by the client conference application by inquiring of the OS.

For example, of the four vertices of a rectangular window, the coordinates of the upper left vertex and the coordinates of the lower right vertex (for example, the XY coordinates of the upper left vertex of the four vertices of the monitor when X, Y = 0, 0) to get This allows the size and shape of the window to be specified. Also, the center point of the window is set as the position of the window.

For example, if the outer dimensions of the application window 1104 are 320×640 pixels, the shape of the application selection window 1102 is changed so that the inner dimensions of the application selection window 1102 are 320×640 pixels.

13, the position determination unit 315 of the client terminal 100D determines and moves the application selection window 1102 to a position where the application window 1104 fits within the window.

That is, as indicated by 1300 in FIG. 13, the size and shape of the application selection window 1102 are changed so that the application window 1104 can fit in, and the position of the application selection window 1102 is moved to match the application window 1104 .

When the aspect ratio is set, the size and shape of the selected application window 1104 are changed according to the window size and shape of the application selection window 1102 .

For example, as indicated by 1200 in FIG. 12, the size and shape of the application window 1104 are changed to fit the application selection window 1102, and the position of the application window 1104 is moved to a position that fits within the application selection window 1102. .
An example of the image of the application window 1104 received from the client terminal 100D and displayed by the client terminal 100C when the aspect ratio is fixed is shown in 1212 in FIG. As indicated by 1212, if the aspect ratio of the fixed image is close to the aspect ratio of the display frame of the image, the image can be displayed in full display frame. A conference screen 1211 of the client terminal 100C used by the user 003 is shown.

On the other hand, 1311 in FIG. 13 shows an example of an image of the application window 1104 that the client terminal 100C receives from the client terminal 100D and displays when the aspect ratio is not fixed. The image can be shared without collapsing the layout of the image in the application window 1104 by changing the aspect ratio.

The client terminal 100D specifies the area (coordinates on the monitor) where the application window 1104 is displayed (step S903).

The client terminal 100D identifies the monitor on which the application window 1104 is displayed and acquires the pixel size (step S904).

The size/shape determining unit 316 of the client terminal 100D determines whether the application window 1104 is larger than the monitor size (step S905).

If the size is equal to or smaller than the monitor size, the process proceeds to step S907. If it is larger than the monitor size, the window size of the application window 1104 is changed to a size that fits in the monitor (step S906).

For example, the size/shape determining unit 316 changes the size and shape of the application window 1104 to the same size and shape as the monitor. In other words, it has the same size and shape as the full-screen display.

The position determination unit 315 of the client terminal 100D changes the position of the application window 1104 to the center of the monitor (step S907).

For example, the coordinates of the center of the application window 1104 are matched with the coordinates of the center of the monitor.

Also, the position of the application selection window 1102 is moved to a position where the application window can be accommodated. That is, the application selection window 1102 is attached to the application window 1104 and moved to the center of the monitor (eg, 1104 and 1102 in FIGS. 12 and 13). The above is the description of FIG.

Next, with reference to FIG. 10, the details of the application window image sharing process will be described.

The client terminal 100D repeats steps S1001 to S1011 until the conditions for deselecting the application window are satisfied.

The deselection condition is that the overlap between the clip-type object 1103 and the application window 1104 is eliminated, or the application window 1104 itself is closed.

Here, the condition for eliminating the overlap between the clip-type object 1103 and the application window 1104 indicates that the application window 1104 is minimized or closed.

The position determination unit 315 of the client terminal 100D identifies the area (coordinates on the monitor) where the application window 1104 is displayed (step S1001).

The client terminal 100D identifies the monitor on which the application window 1104 is displayed and acquires the pixel size of the monitor (step S1002).

The size/shape determining unit 316 of the client terminal 100D determines whether the application window 1104 fits on the monitor (step S1003).

For example, if at least one of the upper left and lower right vertex coordinates of the application window 1104 is not within the monitor area, it is determined that the application window 1104 does not fit within the monitor, and the process proceeds to step S1011.

If both the upper left and lower right vertex coordinates of the application window 1104 are located within the monitor area, it is determined that the application window 1104 is within the monitor, and the process proceeds to step S1004.

In step S1011, the display control unit 312 of the client terminal 100D displays information indicating that sharing of the image of the application window 1104 is stopped on the application selection window 1102 (step S1011).

For example, as indicated by 15 1 0 in FIG. 15, the application selection window 1102 displays "imaging stopped". The display of “image stopped” indicates that the process of storing the captured image of the application window 1104 in the transmission image queue 530 is stopped. In other words, the image (eg 1611) shared by the other client terminals 100 (eg 100A to 100C) is not synchronized with the display update of the application window 1104 and is in a stopped state.

That is, in the case of "image stop", even if the display of the application window on the client terminal 100D is updated from the state 1500 in FIG. 15 to 1104 in 1500 in FIG. 1611 (the application window 1104 before the spreadsheet software overlaps) is not updated.

In step S1004, the transmission control unit 317 of the client terminal 100D inquires of the OS for a list of identification information of windows that are higher in the Z order than the application window 1104 and that are being displayed on the monitor (step S1004). to step S1005.

The Z order above the application window 1104 means that the application window 1104 is located on the front side.

The transmission control unit 317 determines whether the application window 1104 is hidden by another window indicated by the identification information acquired in step S1004 (step S1005).

In this embodiment, the state in which the application window 1104 is hidden by another window refers to the state in which at least a part of the application window 1104 is hidden by another window, as indicated by 1500 in FIG. 15 .

If the application window 1104 is hidden by another window, the process proceeds to step S1011. If the application window 1104 is not hidden by other windows, the process proceeds to step S1006.

In step S1006, the image acquisition unit 314 of the client terminal 100D instructs the OS to capture the image of the selected application window 1104. FIG. Then, the image of the application window 1104 obtained by the capture process executed by the OS is received from the OS (step S1006).

The image acquisition unit 314 stores the received image in the dedicated capture image queue 600 (FIG. 6) created on the RAM 203 of the client terminal 100D (step S1007). Since the configuration of the captured image queue 600 is the same as that of 530 in FIG. 5, the description is omitted.

The display control unit 312 of the client terminal 100D displays information indicating that the real-time image of the application window 1104 is being shared on the application selection window 1102 (step S1008).

For example, as indicated by 1300 in FIG. 13, the application selection window 1102 displays "updating image". The display of “updating image” indicates that the captured image stored in the transmission image queue 530 is being updated in accordance with the update of the display of the application window 1104 . That is, it indicates that the shared image (eg 1311) is updated in conjunction with the update of the display of the application window 1104 also in the other client terminals 100 (eg 100A to 100C).

In the case of "updating image", if the application window display on the client terminal 100D is 1300 in FIG. 13, the image display on the client terminal 100C is 1311. FIG. Also, if the display of the application window on the client terminal 100D is updated like 1104 in 1400 in FIG. 14, the image display on the client terminal 100C is also updated like 1411. FIG.

The transmission control unit 317 of the client terminal 100 acquires the latest captured image from the captured image queue 600 (step S1009), and starts processing to transmit the captured image to the server 200 instead of the camera image (step S1010). ).

Specifically, the process of storing the captured image in the transmission image queue 530 is started. The latest captured images of the application window 1104 are sequentially stored in the transmission image queue 530 until the selection of the application window 1104 is cancelled. As a result, the captured image (application image) in the transmission image queue 530 is continuously transmitted to the server 200 in step S414.

The server 200 receives the capture image in the reception control unit 322 and stores it in the image queue in the delivery image storage 323 . Then, the captured images are sequentially transmitted to other client terminals (for example, 100A to 100C) participating in the same conference as the transmission source terminal (here, the client terminal 100D). Each client terminal 100 that has received the captured image stores the received image in a display image queue and sequentially displays the image to display the image as if playing a moving image.

In other words, until the selection of the application image is canceled, the application image can be continuously acquired and transmitted, and the application image corresponding to the change of the screen of the selected application can be displayed as a video. Can be shared between client terminals.

If the conditions for deselecting the application window 1104 are satisfied, the client terminal 100 ends the processing of steps S1001 to S1010.

Next, the image acquisition unit 314 instructs the OS to finish capturing the application window 1104 . In other words, the process of obtaining the captured image and storing it in the captured image queue 600 ends. Then, the process proceeds to step S808 in FIG.

Returning to the description of FIG. The client terminal 100D determines whether the application selection function has ended (step S808). For example, when the application selection window 1102 is closed, it is determined that the application selection function has ended.

If the application selection function has not ended, the process returns to step S804.

When the application selection function is terminated, the transmission control unit 317 instructs the camera image storage control unit 311 to resume storing camera images in the transmission image queue 530 (step S809). In response to the instruction, the process of storing camera images in the transmission image queue 530 is resumed. In other words, the process of transmitting the camera image in step S414 resumes. The above is the description of FIG.

In the above description, only the client terminal 100D activates the application selection function and transmits the image of the application window to the other client terminals 100D. Transmission/sharing can be performed simultaneously on a plurality of client terminals 100 .

For example, when the processes of FIGS. 4, 8, 9, and 10 are executed on the client terminals 100 of the users 003 and 004, respectively, and the image of the application screen is selected, transmitted, and shared, the user 001 and the user 002 The screen is brought to a state as indicated by 1700 in FIG.

For example, the client terminal 100A of the user 001 displays both the image 1212 (same as in FIG. 12) received from the client terminal 100D of the user 004 and the image 1702 received from the client terminal 100C of the user 003 on the conference screen 1701. FIG.

The above is the description of the embodiment of the present invention.

According to the present invention, it is possible to provide a mechanism for appropriately sharing a selected screen.

For example, the position of the selected screen can be set to a position suitable for sharing.

Also, the size of the selected screen can be set to a size suitable for sharing.

Also, sharing of information different from the selected screen can be prevented.

Also, it is possible to make the user confirm whether or not the selected screen is being shared.

Also, for example, it is possible to prevent images other than the selected screen from being reflected.

In addition, it is possible to prevent sharing of an insufficient screen image, such as a partially missing image.

Furthermore, by continuing to share the image immediately before part or all of the image cannot be obtained, the shared image may appear and disappear on the device of the person to whom the image is shared, similar to the method in which the image is no longer shared. can be reduced, and the user's burden of confirming the cause of the display change can be reduced.

In the above-described embodiments, the present invention has been described by taking a web conference as an example, but the present invention can be applied to systems and services that transmit and receive image data and share it, such as video conferences and video telephony. . A conference using electronic data, such as a web conference, a video conference, or a video call, is called an electronic conference.

The present invention can be embodied as, for example, a system, device, method, program, recording medium, or the like. Specifically, it may be applied to a system composed of a plurality of devices, or may be applied to an apparatus composed of a single device.

For example, each client terminal 100 is assumed to execute the processes of FIGS. may be executed on the server 200. In this case, the client terminal 100 serves as a terminal that remotely operates software on the server 200 .

Further, the program in the present invention is a program that allows a computer to execute the processing method of each flowchart shown in the figure, and the storage medium of the present invention stores the program that allows the computer to execute the processing method of each flowchart shown in the figure. ing. Note that the program in the present invention may be a program for each processing method of each device in FIG.

As described above, a recording medium recording a program for realizing the functions of the above-described embodiments is supplied to a system or device, and the computer (or CPU or MPU) of the system or device reads the program stored in the recording medium. Needless to say, the object of the present invention can also be achieved by reading and executing.

In this case, the program itself read from the recording medium implements the novel functions of the present invention, and the recording medium recording the program constitutes the present invention.

Examples of recording media for supplying programs include flexible disks, hard disks, optical disks, magneto-optical disks, CD-ROMs, CD-Rs, DVD-ROMs, magnetic tapes, non-volatile memory cards, ROMs, EEPROMs, silicon A disk or the like can be used.

Further, by executing the program read by the computer, not only the functions of the above-described embodiments are realized, but also based on the instructions of the program, the OS (operating system) and the like running on the computer are actually executed. Needless to say, a case where part or all of the processing is performed and the functions of the above-described embodiments are realized by the processing are included.

Furthermore, after the program read from the recording medium is written in the memory provided in the function expansion board inserted into the computer or the function expansion unit connected to the computer, the function expansion board is read according to the instruction of the program code. It goes without saying that a case where a CPU or the like provided in a function expansion unit performs part or all of the actual processing and the functions of the above-described embodiments are realized by the processing.

Furthermore, by downloading and reading out the program for achieving the present invention from a server, database, etc. on the network using a communication program, the system or device can enjoy the effects of the present invention.

100A client terminal 100B client terminal 100C client terminal 100D client terminal 101 camera 102 microphone 103 speaker 150 network 200 server

Claims (15)

receiving means for receiving a selection of a window to be shared with another device among the windows displayed by the display means;
In response to the acceptance of the selection by the acceptance means, at least one of the position and size of the window to be shared on the display means is displayed on the display means before the selection. a control means for controlling to change from
transmission control means for controlling to transmit the display contents of the shared window whose position and/or size have been changed by the control means so as to be shared by the other device;
An information processing device comprising: 2. The information processing apparatus according to claim 1, wherein said control means changes at least one of the position and size of said window to be shared so that it falls within a range shared with said other device. 3. The information processing apparatus according to claim 1, wherein the control means controls to change the position of the window to be shared to the center of the display means. When the size of the window to be shared before the selection is larger than the size of the display means, the control means controls to change the size of the window to be shared to a size that fits in the display means. 4. The information processing apparatus according to any one of claims 1 to 3. 5. The control means controls to change the shape and size of the window to be shared in the same manner as when the window to be shared is displayed on the display means on a full screen. 1. The information processing apparatus according to 1. 6. The method according to any one of claims 1 to 5, wherein, in response to receiving the selection, the control means performs control so that the aspect ratio of the window to be shared is changed to a predetermined aspect ratio. The information processing device described. 7. The information processing apparatus according to any one of claims 1 to 6, wherein said accepting means can accept selection of a window for displaying a screen of specific application software. 3. The apparatus further comprises processing means for performing processing such that the display content of the shared window, of which at least one of position and size has been changed by the control means, is continuously shared with the other device. 8. The information processing apparatus according to any one of 1 to 7. The processing means stops sharing the display content of the shared window with the other device when the shared window does not fit in the display area of the display means while the display content of the shared window is being shared with the other device. 9. The information processing apparatus according to claim 8, wherein processing is performed so as to The processing means stops sharing with the other device when at least part of the shared window is hidden by another window while the display content of the shared window is being shared with the other device. 10. The information processing apparatus according to claim 8, wherein processing is performed so as to stop. 11. The window according to any one of claims 1 to 10, wherein the window to be shared is a window to be screen-shared with the other device in at least one of a video conference, a video phone, and a web conference. information processing equipment. receiving means for receiving a selection of a window to be shared with another device among the windows displayed by the display means;
performing processing so that the display content of the window that has received the selection by the receiving means and has become the shared window is continuously shared with the other device;
When the display content of the shared window becomes out of the display area of the display means while the display content of the shared window is being shared with the other device, the sharing with the other device is stopped. a first stop process; and when at least a portion of the shared window is hidden by another window while the display content of the shared window is being shared with the other device. and a processing means for performing at least one of a second stop process for stopping sharing. a receiving step of receiving a selection of a window to be shared with another device among the windows displayed by the display means;
At least one of the position and size of the window to be shared on the display means is displayed on the display means before the selection in response to the selection being received in the receiving step. a control step that controls to change from
a transmission control step of controlling to transmit the display content of the shared window, of which at least one of the position and size has been changed in the control step, so as to be shared by the other device;
A control method for an information processing device, comprising: a receiving step of receiving a selection of a window to be shared with another device among the windows displayed by the display means;
a sharing step of performing processing so that the display content of the window that has become the shared window after receiving the selection in the receiving step is continuously shared with the other device;
When the display content of the shared window becomes out of the display area of the display means while the display content of the shared window is being shared with the other device, the sharing with the other device is stopped. a first stop process; and when at least a portion of the shared window is hidden by another window while the display content of the shared window is being shared with the other device. and a stopping step of performing at least one of a second stopping process of stopping sharing. A program for causing a computer to function as each means of the information processing apparatus according to any one of claims 1 to 12.

Images