JP2021064833A - Server device and video conferencing system - Google Patents

Abstract

To materialize smooth progress of video conference.SOLUTION: A server device 10 comprises the following configuration. An acquisition unit 111 acquires, from each of a plurality of user devices, an image of each user, individual information indicating the sound of the user, and picture information regarding the picture size of a user device corresponding to each user. An image generation unit 112 generates image information corresponding to the picture size of each of the user devices for the user device on the basis of a plurality of pieces of individual information corresponding to the plurality of user devices in one-to-one manner and a plurality of pieces of picture information corresponding to the plurality of user devices in one-to-one manner. A communication device 130 transmits the image information generated for each of the plurality of user devices to the corresponding user device.SELECTED DRAWING: Figure 2

Description

The present invention relates to a server device and a video conferencing system.

Patent Document 1 discloses a conference support system that supports a conference by identifying a speaker among a plurality of participants who participate in the conference. In the conference support system, an imaging device using a fisheye lens is placed on the conference table. The conference support system extracts the movement of the participant's lips from the imaging information captured by the imaging device, and identifies the speaker based on the extracted lip movement. Further, the conference support system generates an image including the faces of a plurality of participants and a microphone icon indicating a speaker based on the captured information and displays the image on the display.

Japanese Unexamined Patent Publication No. 2015-19162

However, the conventional conference support system does not support video conferencing in which a plurality of users in different locations participate. The user device used by the user who participates in the video conference includes, for example, a smartphone, a personal computer, or a clock-type restable device. That is, the screen size of the user device includes the size of a small screen and the size of a large screen. When a video conference is held using a plurality of user devices having different screen sizes, if an image for a large screen is distributed to a plurality of user devices, the image may not be displayed on a certain user device. On the other hand, when an image for a small screen is distributed to a plurality of user devices, the display capability may not be sufficiently exhibited by other user devices. Therefore, the conventional technique has a problem that the video conference cannot proceed smoothly.

In order to solve the above problems, the server device according to the preferred embodiment of the present disclosure is a server device that communicates with a plurality of users participating in a video conference and a plurality of user devices having a one-to-one correspondence. An acquisition unit that acquires an image of a user, individual information indicating the sound of the user, and screen information regarding a screen size of a user device corresponding to each user from each of the plurality of user devices, the plurality of user devices, and 1 Based on a plurality of individual information corresponding to one-to-one and a plurality of screen information corresponding to one-to-one with the plurality of user devices, image information corresponding to the screen size of each user device is provided for each of the plurality of user devices. It includes an image generation unit to be generated, and a transmission unit to transmit image information generated for each of the plurality of user devices to the corresponding user device.

According to the present disclosure, the server device transmits image information according to the screen size to a plurality of user devices, so that the video conference can proceed smoothly.

It is a block diagram which shows an example of the structure of a video conferencing system 1. It is a perspective view which shows an example of the structure of the user apparatus 20_1. It is a top view which shows an example of the structure of the user apparatus 20_2. It is a block diagram which shows an example of the structure of the server apparatus 10. It is explanatory drawing which shows an example of the screen layout when the number of participants of a video conference is four. It is explanatory drawing which shows an example of the screen layout when the number of participants of a video conference is six. It is explanatory drawing which shows an example of the image of a video conference. It is explanatory drawing which shows an example of the image of a video conference. It is explanatory drawing which shows an example of the image which image information G2 shows. It is a block diagram which shows an example of the structure of the user apparatus 20_j. It is a flowchart which shows an example of the operation of the server apparatus 10 concerning the generation of image information in a video conference. It is a flowchart which shows an example of the operation of the server apparatus 10 about a secret talk. It is explanatory drawing which shows an example of the image displayed on the user apparatus 20_1 which is the other party of a secret talk. It is explanatory drawing which shows an example of the image corresponding to image information G3. It is explanatory drawing which shows an example of the image displayed on the user apparatus 20_1 during a chat.

1. 1. Embodiment 1-1: Overall configuration FIG. 1 is a block diagram showing a configuration example of the video conferencing system 1 according to the embodiment. The video conferencing system 1 provides a video conferencing service in which a plurality of users U1, U2, ... Un, who are located at different locations, participate. Further, the video conferencing system 1 provides a service for conducting a secret talk during the video conferencing. The secret talk can be performed by some users of a plurality of users U1, U2, ... Un. A typical example of a secret talk is performed by two users out of a plurality of users U1, U2, ... Un. The content of the secret talk is shared only with the participants of the secret talk and is kept secret to users other than the participants of the secret talk. n is an integer of 2 or more.

The video conferencing system 1 includes a server device 10, and a plurality of user devices 20_1, 20_2, ... 20_n. These components are connected to a communication network NET such as the Internet. The plurality of user devices 20_1 to 20_n have a one-to-one correspondence with the plurality of users U1 to Un. In the following description, an arbitrary integer of 1 or more and n or less is defined as j. Of the plurality of user devices 20_1 to 20_n, any user device is referred to as user device 20_j. Of the plurality of users U1 to Un, any user is referred to as user Uj. Further, the user device 20_1 is an example of the first user device. The user device 20_2 is an example of the second user device.

As the user device 20_j, a device having a call function, an image pickup function, and a communication function can be used. As the user device 20_j, for example, a watch-type wearable device, a personal computer, or a smartphone is used. In the following description, a personal computer is used as the user device 20_1 used by the user U1, and a watch-type wearable device is used as the user device 20_2 used by the user U2.

FIG. 2 is a perspective view showing a configuration example of the user device 20_1. The user device 20_1 includes a display device 240, an input device 250, an image pickup device 260, and a microphone 270. The display device 240 displays the image of the video conference. The image pickup apparatus 260 takes an image of the subject and outputs the image pickup information indicating the image pickup result. The microphone 270 converts sound into an electric signal and outputs an audio signal indicating the conversion result. The image of the video conference includes an image of the voice button B1 in addition to the image of the user's face. The speak button B1 will be described later.

FIG. 3 is a plan view showing a configuration example of the user device 20_2. The user device 20_2 includes a display device 240, an input device 250, an image pickup device 260, and a microphone 270, similarly to the user device 20_1. The user device 20_2 has a belt. The user device 20_2 is used by winding a belt around the wrist of the user U1.

1-2: Server device FIG. 4 is a block diagram showing an example of the configuration of the server device 10. The server device 10 is an example of an information processing device. The server device 10 includes a processing device 110, a storage device 120, and a communication device 130. Each element of the server device 10 is connected to each other by a single bus or a plurality of buses for communicating information. The term "device" in the present specification may be read as another term such as a circuit, a device, or a unit. Further, each element of the server device 10 may be composed of a single or a plurality of devices, and some elements of the server device 10 may be omitted.

The processing device 110 is a processor that controls the entire server device 10, and is composed of, for example, a single chip or a plurality of chips. The processing device 110 is composed of, for example, a central processing unit (CPU) including an interface with peripheral devices, an arithmetic unit, registers, and the like. In addition, a part or all of the functions of the processing device 110 are realized by hardware such as DSP (Digital Signal Processor), ASIC (Application Specific Integrated Circuit), PLD (Programmable Logic Device), FPGA (Field Programmable Gate Array). You may. The processing device 110 executes various processes in parallel or sequentially.

The storage device 120 is a recording medium that can be read by the processing device 110, and can store a plurality of programs including the control program PR1 executed by the processing device 110 and various information used by the processing device 110, such as management information K. Remember. The storage device 120 may be composed of at least one such as a ROM (Read Only Memory), an EPROM (Erasable Programmable ROM), an EPROM (Electrically Erasable Programmable ROM), and a RAM (Random Access Memory). The storage device 120 may be called a register, a cache, a main memory (main storage device), or the like.
The management information K identifies a service ID that identifies a video conference, a user ID that identifies a user who uses the service, screen information that indicates the screen size of the user device used by the user, a speaker flag that indicates a speaker, and a secret talk. This is information for associating a secret talk ID. The management information K provides a video conferencing service by the processing device 110 referring to and updating the management information K.

The communication device 130 is hardware (transmission / reception device) for communicating with another device. The communication device 130 is also called, for example, a network device, a network controller, a network card, a communication module, or the like.

The processing device 110 functions as an acquisition unit 111, an image generation unit 112, a recognition unit 113, and a management unit 114 by reading and executing the control program PR1 from the storage device 120.

The acquisition unit 111 acquires various types of information from each of the plurality of user devices 20_1 to 20_n. The information acquired from the user device 20_j includes individual information Xj, screen information Yj, designated information Zj, and detection information Dj. The individual information Xj, the screen information Yj, the designated information Zj, and the detection information Dj are generated by the user device 20_j. Further, these pieces of information may be transmitted directly from the user device 20_j to the server device 10 or may be transmitted to the server device 10 via another device. For example, when the user device 20_j is a clock-type restable device, such information may be transmitted to the server device 10 via a smartphone possessed by the user Uj.

The individual information Xj is information indicating the image of the user Uj and the sound of the user Uj. The screen information Yj is information regarding the screen size of the user device 20_j. The screen information Yj may indicate, for example, the screen resolution, the model number of the user device 20_j, or the type of the device of the user device 20_j. The model number of the user device 20_j is not information that directly indicates the screen size. However, the screen size of the user device 20_j is uniquely specified by the model number of the user device 20_j. Therefore, the model number of the user device 20_j is information that indirectly indicates the screen size. The type of device of the user device 20_j may include a watch-type restable device, a smartphone, a tablet terminal, and a personal computer. The designated information Zj is information that specifies the other party of the secret talk and the communication method of the secret talk.

The detection information Dj is information indicating that the user device 20_j has detected that the speech button B1 included in the image of the video conference has been operated. The remark button B1 is an example of an operator for conveying the intention of wishing to remark. In a video conference, when a plurality of users speak at the same time, the progress of the video conference is hindered. Therefore, in the video conferencing system 1, the plurality of user devices 20_1 to 20_n display an image that conveys the intention of speaking by operating the speaking button B1.

The image generation unit 112 generates image information Gj according to the screen size of the user device 20_j based on the individual information Xj and the screen information Yj. That is, the image generation unit 112 uses the image information G1 to Gn according to the screen size of each user device based on the plurality of individual information X1 to Xn and the plurality of screen information Y1 to Yj of the plurality of user devices 20_1 to 20_n. Generate for each. The image information G1 is an example of the first image information. The image information G2 is an example of the second image information.

The image generation unit 112 extracts a plurality of face image information Gf1 corresponding to one-to-one with a plurality of users U1 to Un by extracting an image of a portion including a user's face based on a plurality of individual information X1 to Xn. ~ Gfn is generated. The shape of the image of the portion including the user's face is, for example, an ellipse, a circle, or a rectangle. The image of the part including the user's face is composed of the background and the user's face.

The image generation unit 112 determines the screen layout of the image indicated by the image information Gj to be transmitted to the user device 20_j based on the number of users participating in the video conference and the screen information Yj. The image generation unit 112 determines the screen layout by referring to the layout information in which the number of users and the screen size indicated by the screen information Yj are associated with the screen layout, for example.

The image generation unit 112 determines the screen layout of the image indicated by the image information Gj to be transmitted to the user device 20_j based on the number of users participating in the video conference and the screen information Yj. The image generation unit 112 determines the screen layout by referring to the layout information in which the number of users and the screen size indicated by the screen information Yj are associated with the screen layout, for example.

For example, when the screen size of the screen information Yj indicates a personal computer and the number of participants in the video conference is 4, the screen layout is allocated, for example, four face image areas Ra1 to Ra4 as shown in FIG. 5A. .. Further, when the screen size of the screen information Yj indicates a personal computer and the number of participants in the video conference is 6, the screen layout is allocated, for example, four face image areas Rb1 to Rb6 as shown in FIG. 5B. .. On the other hand, when the screen size of the screen information Yj indicates a watch-type wearable device, one face image area Rc1 is allocated as shown in FIG. 3, regardless of the number of participants in the video conference.

The image generation unit 112 generates image information Gj by inserting a face image into the face image area of the screen layout. In the case of the screen layout shown in FIG. 5B, the image generation unit 112 inserts the face images of the users U1 to U6 corresponding to one-to-one into the face image areas Rb1 to Rb6 to generate the image information Gj. In this case, there are 6 participants in the video conference. On the other hand, in the case of the screen layout shown in FIG. 3, the image generation unit 112 inserts the face image of any one of the users U1 to Un into the face image area Rc1 to generate the image information Gj.

When the acquisition unit 111 acquires the detection information Dj, the image generation unit 112 identifies the user Uj of the user device 20_j, which is the source of the detection information Dj, as a user who wants to speak. By updating the image information Gj, the image generation unit 112 includes an image in which the face image of the specified user can be distinguished from the face image of another user in the image of the video conference. For example, assume that the image of the video conference shown in FIG. 6A is displayed on the user device 20_j. In the example shown in FIG. 6A, the face images of the users U1 to U6 are arranged one-to-one in the six face image areas Rb1 to Rb6. Also, the size of the speaker's face image is larger than the size of the face image of another user. That is, the user U2 of the face image displayed in the face image area Rb2 is the speaker. Here, when the user U6 operates the speak button B1 in the image of the video conference displayed on the user device 20_6, the user device 20_6 transmits the detection information D6 to the server device 10. When the acquisition unit 111 acquires the detection information D6, the image generation unit 112 updates the image information Gj. As a result, the image shown by the image information Gj becomes the image of the video conference shown in FIG. 6B. As shown in FIG. 6B, the icon A1 is arranged at a position overlapping the boundary of the face image area Rb6 on which the face image of the user U6 is displayed. The icon A1 is an image indicating that he / she wants to speak. By the icon A1, the users U1 to U5 other than the user U6 know that the user U6 wants to speak.

By the way, the user devices 20_1 to 20_n may include a watch-type wearable device. For example, the user device 20_2 shown in FIG. 3 is a watch-type wearable device. The image generation unit 112 generates image information G2 that can identify a user who wants to speak in a watch-type wearable device. FIG. 6C is an explanatory diagram showing an example of the image shown by the image information G2. As shown in FIG. 6C, the face image of the user U6 who wants to speak is arranged in the face image area Rc2. The image generation unit 112 generates image information G2 using the face image information Gf6. In this example, since the face image of the user U6 is wiped and displayed, the user U2 can assume that the user U6 wants to speak.

Returning to FIG. 4, the recognition unit 113 identifies the speaker based on the plurality of individual information X1 to Xn. For example, the recognition unit 113 compares the loudness of the sounds indicated by the plurality of individual information X1 to Xn, and based on the comparison result, the user corresponding to the individual information indicating the loudest sound is set as the speaker.

When a secret talk is conducted, the management unit 114 determines the user device 20_j of the source of the designated information Zj and the user of the other party of the secret talk specified by the designated information Zj, according to the communication method used for the secret talk specified by the designated information Zj. Switch the communication method with the device.

1-3: User device Next, the user device 20_j will be described. FIG. 7 is a block diagram showing an example of the configuration of the user device 20_j. The user device 20_j includes a processing device 210, a storage device 220, a communication device 230, a display device 240, an input device 250, an image pickup device 260, and a microphone 270. Each element of the user device 20_j is connected to each other by a single bus or a plurality of buses for communicating information. Each element of the user device 20_j is composed of a single device or a plurality of devices, and some elements of the user device 20_j may be omitted.

The processing device 210 is a processor that controls the entire user device 20_j, and is composed of, for example, a single chip or a plurality of chips. The processing device 210 is composed of, for example, a central processing unit including an interface with peripheral devices, an arithmetic unit, registers, and the like. The processing device 210 executes various processes in parallel or sequentially.

The storage device 220 is a recording medium that can be read by the processing device 210, and can store a plurality of programs including the control program PR2 executed by the processing device 210 and various information used by the processing device 210, such as screen information Yj. Remember. The storage device 220 may be composed of at least one such as a ROM (Read Only Memory), an EPROM (Erasable Programmable ROM), an EPROM (Electrically Erasable Programmable ROM), and a RAM (Random Access Memory). The storage device 220 may be called a register, a cache, a main memory (main storage device), or the like.

The communication device 230 is hardware (transmission / reception device) for communicating with another device.

The display device 240 displays an image. The display device 240 displays various images under the control of the processing device 210. For example, various display panels such as a liquid crystal display panel and an organic EL (Electro Luminescence) display panel are preferably used as the display device 240.

The input device 250 is a device that receives an input from the outside. For example, the input device 250 accepts an operation for inputting a code such as a number and a character into the processing device 110 and an operation for selecting an image such as an icon displayed on the display device 240. For example, a touch panel that detects contact with the display surface of the display device 240 is suitable as the input device 250. The input device 250 may include one or more controls that can be operated by the user. The input device 250 outputs input information according to the user's operation.

The image pickup device 260 takes an image of the subject and outputs the image pickup information indicating the image pickup result to the processing device 210. When using the video conferencing service, the subject is the user Uj who uses the user device 20_j. The user Uj images himself / herself using the user device 20_j. The image indicated by this imaging information includes the face of the user Uj.

The microphone 270 converts sound into an electric signal, and outputs sound information indicating the conversion result to the processing device 210.

The processing device 210 reads the control program PR2 from the storage device 220 and executes the read control program PR2 to function as the first generation unit 211, the second generation unit 212, and the transmission control unit 213.

The first generation unit 211 generates individual information Xj based on the imaging information and the sound information. The imaging information indicates an image in which the user Uj is a subject. The sound information indicates the voice of the user Uj when the user Uj speaks. The user ID may be included in the individual information Xj. The communication device 230 transmits the individual information Xj to the server device 10.

The second generation unit 212 generates the detection information Dj and the designated information Zj based on the input information.
The second generation unit 212 generates the detection information Dj when the input information indicates the operation of the speak button B1.

The second generation unit 212 generates the designated information Zj according to the operation of the input information on the face image of the user other than the user Uj among the users U1 to Un. The operation on the face image may be any operation as long as the operation can be identified. For example, the operation on the face image includes a tap operation and a long press operation. The long press operation means an operation of continuously pressing the face image for a predetermined time or longer. When the input information is a tap operation on the face image, the second generation unit 212 generates the designated information Zj that specifies the secret talk using the first communication method for the user of the face image. When the input information is a long press operation on the face image, the second generation unit 212 generates the designated information Zj that specifies the secret talk using the second communication method for the user of the face image.

The first communication method is, for example, a voice call. The second communication method is, for example, chat and non-verbal communication. Non-verbal communication means communication that does not depend on language. Non-verbal communication includes sending concrete icons such as so-called stamps and photographs, and effects on facial images. The face image targeted by the effect is a face image of the user corresponding to the user device of the source of the specified information in the image of the video conference displayed on the user device of the other party of the secret talk. Further, the effect may include, for example, shaking the face image from side to side, changing the size of the face image periodically, and the like.

The transmission control unit 213 transmits the individual information Xj, the screen information Yj, the designated information Zj, and the detection information Dj to the server device 10 by controlling the communication device 230.

1-4: Operation of the server device The operation of the server device 10 will be described separately for a video conference and a secret talk. FIG. 8 is a flowchart showing the operation of the server device 10 regarding the generation of image information in a video conference. The processing device 110 detects the start of the video conference (step S1). A specific URL is notified in advance to the user devices 20_1 to 20_n participating in the video conference. The user devices 20_1 to 20_n can use the video conferencing service by accessing a specific URL. The processing device 110 detects the activation of the video conference by detecting, for example, that the user device has accessed a specific URL.

Next, the processing device 110 functions as the acquisition unit 111, and acquires the individual information X1 to Xn and the screen information Y1 to Yn from the user devices 20_1 to 20_n participating in the video conference (step S2).

In step S3, the processing device 110 functions as an image generation unit 112 and extracts a face image based on the individual information X1 to Xn. By extracting an image of a portion including a user's face based on the individual information X1 to Xn, the processing device 110 obtains a plurality of face image information Gf1 to Gfn corresponding to one-to-one with the plurality of users U1 to Un. Generate.

In step S4, the processing device 110 functions as the image generation unit 112, and determines the screen layout according to the number of participants in the video conference for each screen size of the user devices 20_1 to 20_n. The processing device 110 may use the number of acquired individual information X1 to Xn as the number of participants in the video conference. Alternatively, the processing device 110 may specify the number of users participating in the video conference by counting the plurality of face images extracted in step S2.

In step S5, the processing device 110 functions as an image generation unit 112 to generate image information G1 to Gn. Specifically, the processing device 110 inserts the image information G1 to Gn into the determined screen layout for each of the user devices 20_1 to 20_n by inserting a plurality of face images indicated by the face image information Gf1 to Gfn. Generate.

In step S6, the processing device 110 functions as the recognition unit 113, and determines whether or not the speaker can recognize the individual information X1 to Xn. If the determination result is negative, the processing device 110 repeats the determination until the determination result becomes affirmative.

If the determination result in step S6 is affirmative, the processing device 110 functions as the image generation unit 112 and emphasizes the face image of the speaker (step S7). In step S8, the processing device 110 functions as an image generation unit 112, and updates the image information indicating the image in which the speaker's face image is not emphasized to the image information indicating the image in which the speaker's face image is emphasized. To do.

In step S9, the processing device 110 functions as the image generation unit 112, and determines whether or not the acquisition unit 111 has acquired the detection information Dj.
If the determination result in step S9 is negative, the processing device 110 returns the processing to step S6. If the determination result in step S9 is affirmative, the processing device 110 functions as the image generation unit 112, and identifies the user Uj of the user device 20_j, which is the source of the detection information Dj, as the user who wants to speak (step S10). ).

In step S11, the processing device 110 functions as an image generation unit 112, and updates the image information G1 to Gn by including an image that can identify the face image of the specified user in the image of the video conference. The image that can identify the face image of the specified user corresponds to, for example, the image of the icon A1 shown in FIG. 6B.

After that, the processing device 110 determines whether or not the end condition of the video conference is satisfied (step S12). If the determination result is negative, the processing device 110 returns the processing to step S6. On the other hand, if the determination result is affirmative, the processing device 110 ends the processing. The termination condition of the video conference may be, for example, that the communication connection is terminated for all the user devices 20_1 to 20_n that are communication-connected to the server device 10 in order to participate in the video conference.

Image information G1 to Gn are generated by the above processing. The generated image information G1 to Gn are transmitted to the user devices 20_1 to 20_n corresponding to one-to-one.

Next, the operation of the server device 10 in the secret talk will be described. FIG. 9 is a flowchart showing the operation of the server device 10 regarding the secret talk.

The processing device 110 determines whether or not the designated information has been acquired from any of the plurality of user devices 20_1 to 20_n (step S21). If the determination result in step S21 is negative, the processing device 110 repeats the process in step S21 until the determination result becomes affirmative. In the following description, the server device 10 determines the case where the designated information Z3 is acquired from the user device 20_3. Further, it is assumed that the designation information Z3 specifies a user U1 who uses the user device 20_1 as the other party of the secret talk. That is, the user U3 is the requester of the secret talk, and the user U1 is the other party of the secret talk.

If the determination result in step S21 is affirmative, the processing device 110 functions as the image generation unit 112, and determines whether or not the communication method indicated by the designated information Z3 is the first aspect. In this example, there are two communication methods for secret talk, the first aspect and the second aspect. The communication method of the first aspect is a voice call. The communication method of the second aspect is chat.

If the determination result in step S21 is affirmative, the processing device 110 functions as an image generation unit 112 to generate image information G1 corresponding to the other party of the secret talk and corresponding to the first aspect (step S24). After that, the processing device 110 transmits the image information G1 to the user device 20_1 used by the user U1 who is the other party of the secret talk (step S24).

FIG. 10 is an explanatory diagram showing an example of an image displayed on the user device 20_1 which is the other party of the secret talk. As shown in FIG. 10, in the image, the icon A2 in which the telephone is embodied is arranged in the face image area Rb3 in which the face image of the user U3 requesting a secret talk is arranged. The user U1 can know that the user U3 requests a secret talk by voice call by arranging the icon A2 corresponding to the communication method of the first aspect in the face image area Rb3.

In step S25, the processing device 110 determines whether or not the other party of the secret talk has accepted the secret talk. Specifically, the processing device 110 determines whether or not the user U1 who is the other party of the secret talk has tapped the icon A2 shown in FIG. 10 during a predetermined period. In the user device 20_1, the detection information D1 is generated when the tap operation is performed. The processing device 110 determines whether or not the other party of the secret talk has accepted the secret talk, depending on whether or not the detection information D1 has been acquired during the predetermined period.

If the determination result in step S25 is affirmative, the processing device 110 starts a voice call between the user device 20_1 and the user device 20_3 (step S26). In step S26, the processing device 110 transfers the voice information acquired from the user device 20_1 to the user device 20_3, and transfers the voice information acquired from the user device 20_3 to the user device 20_1. This process enables a voice call between the user U1 and the user U3.

If the determination result in step S25 is negative, the processing device 110 generates the image information G3 corresponding to the request source of the secret talk (step S27). After that, the processing device 110 transmits the image information G3 to the user device 20_3, which is the request source of the secret talk (step S28). FIG. 11 is an explanatory diagram showing an example of an image corresponding to the image information G3. As shown in FIG. 11, in the image, an icon A3 indicating that the call is not possible is arranged in the face image area Rb1 where the face image of the user U1 who is the other party of the secret talk is arranged. The user U3 knows that the icon A3 is arranged in the face image area Rb1 so that the user U1 does not accept the secret talk by the voice call.

Next, when the determination result in step S22 is negative, the communication method designated by the designated information Z3 is the second aspect. In this case, the processing device 110 generates image information G1 and G3 corresponding to the chat, which is the second aspect (step S29). The image indicated by the image information G1 and the image indicated by the image information G3 are the same. The processing device 110 transmits the image information G3 to the user device 20_3, and transmits the image information G1 to the user device 20_1 (step S30).

After that, the processing device 110 starts a chat between the user device 20_1 and the user device 20_3 (step S31). The processing device 110 during chat generates image information G1 and G3 indicating an image reflecting the text information acquired from the user device 20_1 and the text information acquired from the user device 20_3, and generates the generated image information G1 and G3. It is transmitted to the user device 20_1 and the user device 20_3.

FIG. 12 is an explanatory diagram showing an example of an image displayed on the user device 20_1 during a chat. As shown in FIG. 12, the image is provided with a chat area Rd. Text is displayed in the chat area Rd. Further, in the chat area Rd, an end button B2 for inputting an end instruction is arranged. The chat area Rd of this example is arranged in an area that does not overlap with the face image of the user U3 who is the requester of the chat, the face image of the user U1 who is the other party of the chat, and the face image of the user U2 who is the speaker. According to this screen layout, the user can recognize the requester of the secret talk and the other party of the secret talk at a glance. Further, according to this screen layout, since the state of the speaker can be recognized, the requester and the other party of the secret talk can talk while grasping the progress of the video conference.

After that, the processing device 110 determines whether or not the end condition of the secret talk is satisfied (step S32). If the determination result in step S32 is negative, the processing device 110 repeats the determination process in step S32 until the determination result becomes affirmative. When the determination result becomes affirmative, the processing device 110 ends the processing of the secret talk and returns to the processing of the normal video conference.

As described above, the server device 10 communicates with the plurality of users U1 to Un participating in the video conference and the plurality of user devices 20_1 to 20_n corresponding to one-to-one. The acquisition unit 111 acquires the image of each user, the individual information Xj indicating the sound of the user, and the screen information Yj regarding the screen size of the user device corresponding to each user from each of the plurality of user devices 20_1 to 20_n. The image generation unit 112 provides a plurality of individual information X1 to Xn corresponding to one-to-one with the plurality of user devices 20_1 to 20_n and a plurality of screen information Y1 to Yn corresponding to one-to-one with the plurality of user devices 20_1 to 20_n. Based on this, image information Gj corresponding to the screen size of each user device is generated for each of the plurality of user devices 20_1 to 20_n. The communication device 130, which is an example of the transmission unit, transmits the image information Gj generated for each of the plurality of user devices 20_1 to 20_n to the corresponding user device 20_j.
According to the above configuration, even if the screen sizes of the plurality of user devices 20_1 to 20_n are partially different, the server device 10 has an image corresponding to the screen size of each of the plurality of user devices 20_1 to 20_n. Since the information Gj is transmitted, the video conference can proceed smoothly.

Further, the plurality of user devices 20_1 to 20_n include a user device 20_1 which is an example of the first user device and a user device 20_2 which is an example of the second user device. The screen size indicated by the screen information Y1 corresponding to the user device 20_1 is larger than the screen size indicated by the screen information Y2 corresponding to the user device 20_2. The image generation unit 112 generates the image information G1 corresponding to the user device 20_1 and the image information G2 corresponding to the user device 20_2. The image indicated by the image information G1 includes the faces of a plurality of users U1 to Un. The image indicated by the image information G2 includes the face of the speaker among the plurality of users U1 to Un. The number of user faces included in the image indicated by the image information G2 is smaller than the number of user faces included in the image indicated by the image information G1. The communication device 130 transmits the image information G1 to the user device 20_1 and the image information G2 to the user device 20_2.

According to the above configuration, the number of user faces displayed on the user device having a small screen size and the number of faces of the user displayed on the user device having a large screen size are the users displayed on the user device having a small screen size. Since the number of faces is larger than that of the user, the number of faces of users participating in the video conference can be changed according to the screen size. When the number of faces of the user to be displayed decreases, the amount of information decreases, but the face of the speaker is displayed even on a user device having a small screen size, which enhances the convenience of video conferencing.

Further, the image generation unit 112 extracts a plurality of face images corresponding to one-to-one with a plurality of users U1 to Un by extracting an image of a portion including a user's face based on a plurality of individual information X1 to Xn. Information Gf1 to Gfn are generated. The image generation unit 112 generates information including face image information Gf1 to Gfn of a plurality of users U1 to Un as the image information G1. The image generation unit 112 includes the face image information of the speaker among the face image information Gf1 to Gfn of the plurality of users U1 to Un as the image information G2, and the face image information Gf1 to Gfn of the plurality of users U1 to Un. Generate information that does not include some face image information.

According to the above configuration, the image generation unit 112 generates a plurality of face image information Gf1 to Gfn, and uses the generated plurality of face image information Gf1 to Gfn to generate a plurality of image information G1 to Gn. .. Therefore, the image generation unit 112 can easily generate a plurality of image information G1 to Gn as compared with the case where a necessary user's face image is extracted for each of the plurality of image information G1 to Gn.

The server device 10 includes a recognition unit 113 that recognizes a speaker among a plurality of users U1 to Un based on the voices indicated by the plurality of individual information X1 to Xn. The image generation unit 112 identifies the face image information corresponding to the speaker among the plurality of face image information Gf1 to Gfn based on the recognition result of the recognition unit 113.

According to the above configuration, since the speaker is identified based on the voice, the image is analyzed to generate the feature amount of the movement of the lips, and the speaker is specified from the generated feature amount. The processing load is reduced.

When the number of faces of the user included in the image indicated by the image information G2 is "1", the image generation unit 112 uses the image information G1 by emphasizing the face image indicated by the face image information corresponding to the speaker. Generate. The image generation unit 112 generates the image information G2 without emphasizing the face image indicated by the face image information corresponding to the speaker.

The reason for emphasizing the speaker's face image is to facilitate the identification of the speaker when the image contains a plurality of face images. Since the number of user's faces included in the image indicated by the image information G2 is "1", only the speaker is displayed. Therefore, it is not necessary to emphasize the face image included in the image indicated by the image information G2. According to the above configuration, when the number of faces of the user included in the image indicated by the image information G2 is "1", the face image of the speaker is not emphasized, so that the processing load for generating the image information G2 is reduced. Will be done.

The image indicated by each of the plurality of image information G1 to Gn includes an image of the remark button B1 which is an example of an operator for transmitting a desire to remark. The acquisition unit 111 can acquire the detection information Dj indicating that the image of the speak button B1 has been operated from any of the plurality of user devices 20_1 to 20_n. The image generation unit 112 includes an image in which each of the plurality of image information G1 to Gn can identify the user Uj who uses the user apparatus 20_j which is the source of the detection information Dj.

According to the above configuration, the user who wants to speak can convey the intention to speak to other users who participate in the video conference by operating the voice button B1, so that the video conference can be smoothly performed. You can proceed.

The acquisition unit 111 can acquire the designated information Z1 to Zn that specifies the user who is the other party of the secret talk and the communication method used for the secret talk from the plurality of user devices 20_1 to 20_n. When a secret talk is conducted, the management unit 114 determines the user device 20_j of the source of the designated information Zj and the user of the other party of the secret talk specified by the designated information Zj, according to the communication method used for the secret talk specified by the designated information Zj. Switch the communication method with the device.

According to the above configuration, the communication method can be specified according to the designated information Zj generated by the requester of the secret talk, so that the secret talk can be performed according to the situation as compared with the case where there is only one communication method of the secret talk.

The designated information Zj is generated by one of the plurality of user devices 20_1 to 20_n, the user device 20_j. The user who is the other party of the secret talk is specified by the face image to be operated among the plurality of face images displayed on one user device 20_j. The communication method in the secret talk is specified by the operation on the face image.

According to the above configuration, the designation of the other party of the secret talk and the designation of the communication method in the secret talk can be specified by the operation on one face image, so that the user's convenience is compared with the case of specifying by the operation on different images. Improves sex.

When the faces of a plurality of users U1 to Un are displayed in the image displayed on the user device 20_j of the other party of the secret talk in the video conference, the image generation unit 112 emphasizes the face of the user who is the source of the specified information. Image information Gj indicating an image is generated.
According to the above configuration, it is possible to notify the user of the other party of the secret talk to the user who requested the secret talk.

2. Modifications The present disclosure is not limited to the embodiments exemplified above. A specific mode of modification is illustrated below. Two or more aspects arbitrarily selected from the following examples may be merged.

2-1: Modification 1
In the embodiment, the image shown by each of the image information G1 to Gn is composed of one page, but the present disclosure is not limited to this. For example, the maximum value of the face image to be displayed on one page is determined according to the screen size, and when the number of participants in the video conference exceeds the maximum value, the image generation unit 112 displays the image of the video conference on a plurality of pages. The constituent image information Gj may be generated. For example, in the case of a wearable device having a screen size of a watch type, the maximum value may be set to "1". The image generation unit 112 generates image information Gj composed of n pages of the image of the video conference. For example, when the screen size is a smartphone, the maximum value may be set to "3". If the number of participants in the video conference is 6, the image generation unit 112 may generate image information Gj in which the image of the video conference is composed of two pages. When the image of the video conference is composed of a plurality of pages, the page displayed by the flick operation can be changed on the user device 20_j.

2-2: Modification example 2
In the present disclosure, the image information G1 to Gn may be a moving image or a still image. Further, the image information about the speaker may be a moving image, and the image information about a user other than the speaker may be a still image. The image information G1 to Gn may be used as a moving image, and the frame rate of the image information of the speaker may be higher than the frame rate of the image information related to a user other than the speaker. In this case, the processing device 110 may instruct each of the user devices 20_1 to 20_n to specify the frame rate of the individual information.

3. 3. Others (1) In the above-described embodiment, the storage device 120 or 220 is a recording medium that can be read by the processing device 110 or 210, and examples thereof include a ROM and a RAM. Discs, digital versatile discs, Blu-ray® discs), smart cards, flash memory devices (eg cards, sticks, key drives), CD-ROMs (Compact Disc-ROMs), registers, removable discs, hard disks , Floppy® discs, magnetic strips, databases, servers and other suitable storage media. The program may also be transmitted from the network via a telecommunication line. The program may also be transmitted from the communication network via a telecommunication line.

(2) In the above-described embodiment, the described information, signals, etc. may be represented using any of a variety of different techniques. For example, data, instructions, commands, information, signals, bits, symbols, chips, etc. that may be referred to throughout the above description are voltages, currents, electromagnetic waves, magnetic fields or magnetic particles, light fields or photons, or any of these. It may be represented by a combination of.

(3) In the above-described embodiment, the input / output information and the like may be stored in a specific place (for example, a memory) or may be managed by using a management table. Input / output information and the like can be overwritten, updated, or added. The output information and the like may be deleted. The input information or the like may be transmitted to another device.

(4) In the above-described embodiment, the determination may be made by a value represented by 1 bit (0 or 1) or by a boolean value (Boolean: true or false). , May be done by numerical comparison (eg, comparison with a given value).

(5) The order of the processing procedures, sequences, flowcharts, etc. exemplified in the above-described embodiment may be changed as long as there is no contradiction. For example, the methods described in the present disclosure present elements of various steps using exemplary order, and are not limited to the particular order presented.

(6) Each function illustrated in FIG. 1 is realized by any combination of at least one of hardware and software. Further, the method of realizing each functional block is not particularly limited. That is, each functional block may be realized by using one device that is physically or logically connected, or directly or indirectly (for example, by two or more devices that are physically or logically separated). , Wired, wireless, etc.) and may be realized using these plurality of devices. The functional block may be realized by combining the software with the one device or the plurality of devices.

Further, the communication devices 130 and 230 are hardware (transmission / reception devices) for performing communication between computers via at least one of a wired network and a wireless network, and are, for example, a network device, a network controller, a network card, a communication module, and the like. Also called. The communication device 130 includes, for example, a high frequency switch, a duplexer, a filter, a frequency synthesizer, or the like in order to realize at least one of FDD (Frequency Division Duplex) and Time Division Duplex (TDD). It may be configured to include.

(7) In the program illustrated in the above-described embodiment, the software is an instruction, an instruction set, a code, regardless of whether the software is called software, firmware, middleware, microcode, hardware description language, or another name. It should be broadly interpreted to mean code segments, program code, programs, subprograms, software modules, applications, software applications, software packages, routines, subroutines, objects, executable files, execution threads, procedures, features, etc. ..

Further, software, instructions, information and the like may be transmitted and received via a transmission medium. For example, a website, where the software uses at least one of wired technology (coaxial cable, fiber optic cable, twisted pair, Digital Subscriber Line (DSL), etc.) and wireless technology (infrared, microwave, etc.). When transmitted from a server, or other remote source, at least one of these wired and wireless technologies is included within the definition of transmission medium.

(8) In each of the above-mentioned forms, the terms "system" and "network" are used interchangeably.

(9) The information, parameters, etc. described in the present disclosure may be expressed using absolute values, relative values from predetermined values, or other corresponding information. May be represented as.

(10) In the above-described embodiment, the store device and the user device may be a mobile station (MS: Mobile Station). Mobile stations can be used by those skilled in the art as subscriber stations, mobile units, subscriber units, wireless units, remote units, mobile devices, wireless devices, wireless communication devices, remote devices, mobile subscriber stations, access terminals, mobile terminals, wireless. It may also be referred to as a terminal, remote terminal, handset, user agent, mobile client, client, or some other suitable term. Further, in the present disclosure, terms such as "mobile station", "user terminal", "user equipment (UE)", and "terminal" can be used interchangeably.

(11) In the embodiments described above, the terms "connected", "coupled", or any variation thereof, are direct or indirect between two or more elements. It means any connection or connection and can include the presence of one or more intermediate elements between two elements that are "connected" or "connected" to each other. The connection or connection between the elements may be physical, logical, or a combination thereof. For example, "connection" may be read as "access". As used in the present disclosure, the two elements use at least one of one or more wires, cables and printed electrical connections, and, as some non-limiting and non-comprehensive examples, the radio frequency domain. Can be considered to be "connected" or "coupled" to each other using electromagnetic energies having wavelengths in the microwave and light (both visible and invisible) regions.

(12) In the above embodiments, the phrase "based on" does not mean "based on" unless otherwise stated. In other words, the statement "based on" means both "based only" and "at least based on".

(13) The terms "determining" and "determining" used in the present disclosure may include a wide variety of actions. "Judgment" and "decision" are, for example, judgment, calculation, computing, processing, deriving, investigating, looking up, search, inquiry. (For example, searching in a table, database or another data structure), ascertaining may be regarded as "judgment" or "decision". Also, "judgment" and "decision" are receiving (for example, receiving information), transmitting (for example, transmitting information), input (input), output (output), and access. (Accessing) (for example, accessing data in memory) may be regarded as "judgment" or "decision". In addition, "judgment" and "decision" mean that the things such as solving, selecting, choosing, establishing, and comparing are regarded as "judgment" and "decision". Can include. That is, "judgment" and "decision" may include considering some action as "judgment" and "decision". Further, "judgment (decision)" may be read as "assuming", "expecting", "considering" and the like.

(14) When "include", "including" and variants thereof are used in the embodiments described above, these terms are similar to the term "comprising". , Intended to be inclusive. Furthermore, the term "or" used in the present disclosure is intended not to be an exclusive OR.

(15) In the present disclosure, if articles are added by translation, for example a, an and the in English, the disclosure also includes the plural nouns following these articles. Good.

(16) In the present disclosure, the term "A and B are different" may mean "A and B are different from each other". The term may mean that "A and B are different from C". Terms such as "separate" and "combined" may be interpreted in the same way as "different".

(17) Each of the embodiments / embodiments described in the present disclosure may be used alone, in combination, or switched with execution. Further, the notification of predetermined information (for example, the notification of "being X") is not limited to the explicit one, but is performed implicitly (for example, the notification of the predetermined information is not performed). May be good.

Although the present disclosure has been described in detail above, those skilled in the art are not limited to the embodiments described in the present disclosure. The present disclosure may be implemented as an amendment or modification without departing from the purpose and scope of the present disclosure, which is determined by the description of the scope of claims. Therefore, the description of the present disclosure is for the purpose of exemplary explanation and does not have any limiting meaning to the present disclosure.

1 ... Video conferencing system, 10 ... Server device, 20_1 to 20_n ... User device, 111 ... Acquisition unit, 112 ... Image generation unit, 113 ... Recognition unit, 114 ... Management unit, DJ ... Detection information, Gf1 to Gf6 ... Face image Information, Gj ... image information, Uj ... user, Xj ... individual information, Yj ... screen information, Zj ... designated information.

Claims (10)

A server device that communicates with a plurality of users participating in a video conference and a plurality of user devices having a one-to-one correspondence.
An acquisition unit that acquires an image of each user, individual information indicating the sound of the user, and screen information regarding the screen size of the user device corresponding to each user from each of the plurality of user devices.
Based on the plurality of individual information corresponding to one-to-one with the plurality of user devices and the plurality of screen information corresponding to one-to-one with the plurality of user devices, the image information corresponding to the screen size of each user device is obtained. An image generator generated for each of a plurality of user devices,
A transmission unit that transmits image information generated for each of the plurality of user devices to the corresponding user device, and a transmission unit.
A server device that comprises. The plurality of user devices include a first user device and a second user device.
The screen size indicated by the screen information corresponding to the first user device is larger than the screen size indicated by the screen information corresponding to the second user device.
The image generation unit generates first image information corresponding to the first user device and second image information corresponding to the second user device.
The image indicated by the first image information includes the faces of the plurality of users.
The image indicated by the second image information includes the face of the speaker among the plurality of users.
The number of user's faces included in the image indicated by the second image information is smaller than the number of user's faces included in the image indicated by the first image information.
The transmission unit transmits the first image information to the first user device, and transmits the second image information to the second user device.
The server device according to claim 1. The image generation unit
By extracting an image of a portion including the face of the user based on the plurality of individual information, a plurality of face image information corresponding to one-to-one with the plurality of users is generated.
As the first image information, information including the face image information of the plurality of users is generated, and the information is generated.
As the second image information, among the face image information of the plurality of users, information including the face image information of the speaker and not including a part of the face image information of the plurality of users. To generate,
The server device according to claim 2. A recognition unit that recognizes a speaker among the plurality of users based on the voice indicated by the plurality of individual information is further provided.
The image generation unit identifies the face image information corresponding to the speaker among the plurality of face image information based on the recognition result of the recognition unit.
The server device according to claim 3. When the number of user's faces included in the image indicated by the second image information is 1, the image generation unit may perform the image generation unit.
The first image information is generated by emphasizing the face image indicated by the face image information corresponding to the speaker.
The second image information is generated without emphasizing the face image indicated by the face image information corresponding to the speaker.
The server device according to claim 3 or 4.
The image indicated by each of the plurality of image information includes an image of an operator for conveying the intention to make a statement.
The acquisition unit can acquire detection information indicating that the image of the operator has been operated from any of the plurality of user devices.
The image generation unit
The image indicated by each of the plurality of image information includes an image that can identify the user who uses the user device that is the source of the detection information.
The server device according to any one of claims 1 to 5. The acquisition unit can acquire designated information for designating the user who is the other party of the secret talk and the communication method used for the secret talk from the plurality of user devices.
When the secret talk is performed, the user device of the source of the designated information and the user device of the other party of the secret talk designated by the designated information are used according to the communication method used for the secret talk specified by the designated information. Equipped with a management unit that switches communication methods
The server device according to any one of claims 1 to 6. The specified information is
Generated by one of the plurality of user devices
The user who is the other party of the secret talk is designated by the face image to be operated among the plurality of face images displayed on the one user device.
The communication method in the secret talk is specified by the operation on the face image.
The server device according to claim 7. The image generation unit
In the video conference, when the faces of the plurality of users are displayed on the image displayed on the user device of the other party of the secret talk, the image showing the image emphasizing the face of the user who is the transmission source of the designated information. The server device according to claim 7 or 8, which generates information. The server device according to any one of claims 1 to 9, and the server device.
Multiple user devices that have a one-to-one correspondence with multiple users participating in a video conference,
Video conferencing system with.

Images