JP2023176774A - Information processing apparatus, display method, communication system, and program - Google Patents

Abstract

To provide an information processing apparatus configured to display text data on a second screen which is different from a first screen on which text data converted from sound data is displayed.SOLUTION: An information processing apparatus which communicates with a plurality of terminals over a network includes: a communication unit which transmits display instruction information to instruct displaying text data converted from sound data transmitted by the terminals on a first display region, to a first terminal and a second terminal; and a processing unit which stores, in a storage unit, operation information regarding the text data, received from the terminals, in association with the text data. The communication unit is configured to transmit, to the plurality of terminals, the display instruction information to instruct displaying at least a part of the text data to which an operation of the operation information has been performed, in a second display region different from the first display region.SELECTED DRAWING: Figure 29

Description

The present invention relates to an information processing device, a display method, a communication system, and a program.

2. Description of the Related Art Communication systems that transmit and receive video and audio to and from remote locations via networks are known. In a communication system, for example, at a base where one of the participants in a conference is located, a communication terminal is used to capture images of the surroundings and collect voices (also referred to as sound data) such as speech, and convert these into digital data. and send it to the other party's communication terminal. The other party's communication terminal displays an image on the display and outputs audio from the speaker, allowing participants between the bases to make a video call.

In order to support video conferences, a technology has been devised to convert audio from video conferences into text using voice recognition (for example, see Patent Document 1). Patent Document 1 discloses a technique for using this text for creating minutes.

However, the conventional technology has a problem in that text data is not displayed on a second screen that is separate from the first screen on which text data converted from sound data is displayed. For example, during a meeting, text data converted from audio data is displayed one after another, and even if a participant asks a question, the presenter may not notice and the discussion proceeds, leaving the person asking the question behind. be.

In view of the above problems, an object of the present invention is to provide an information processing device in which text data is displayed on a second screen that is separate from a first screen on which text data converted from sound data is displayed. shall be.

In view of the above problems, the present invention provides an information processing device that communicates with a plurality of terminals via a network, and which displays text data converted from sound data transmitted by the terminals in a first display area together with display instruction information. , a communication unit that transmits to the first terminal and the second terminal, and a processing unit that associates operation information regarding text data received from the terminal with the text data and stores it in a storage unit, transmitting to the plurality of terminals display instruction information for displaying at least a portion of the text data on which the operation information has been manipulated in a second display area different from the first display area; It is characterized by

It is possible to provide an information processing device in which text data is displayed on a second screen that is separate from a first screen on which text data converted from sound data is displayed.

1 is a schematic diagram of a communication system according to this embodiment. FIG. 2 is a hardware configuration diagram of a video conference terminal. FIG. 2 is a hardware configuration diagram of a car navigation device. It is a hardware configuration diagram of a PC, a display terminal, and a server. FIG. 2 is a functional block diagram of a communication system. (A) is a conceptual diagram of a user authentication management table, (B) is a conceptual diagram of an access management table, and (C) is a conceptual diagram of a schedule management table. It is a conceptual diagram of a content management table. FIG. 3 is a conceptual diagram of a reaction type management table. FIG. 3 is a conceptual diagram showing a reaction press management table. (A) is a conceptual diagram of a user authentication management table, (B) is a conceptual diagram of a user management table, and (C) is a conceptual diagram of a union management table. (A) is a conceptual diagram of a union reservation management table, and (B) is a conceptual diagram of an event management table. (A) is a conceptual diagram of a server authentication management table, and (B) is a conceptual diagram of an execution event history management table. (A) A conceptual diagram of an execution event management table, and (B) a conceptual diagram of a related information management table. It is a conceptual diagram of a text information management table. FIG. 3 is a sequence diagram showing schedule registration processing. It is a figure showing a sign-in screen. This is an example of the initial screen of a PC. It is a figure showing a schedule input screen. FIG. 3 is a sequence diagram showing event start processing. FIG. 3 is a diagram showing a sign-in screen displayed on the electronic blackboard. FIG. 3 is a diagram showing a reservation list screen for a union. FIG. 3 is a sequence diagram showing event start processing. It is a figure showing a detailed information screen of an event. FIG. 3 is a diagram showing a display screen displayed on the electronic blackboard at the start of an event. FIG. 3 is a sequence diagram showing execution event history registration processing. 3 is a flowchart showing speech-to-text conversion processing. FIG. 3 is a sequence diagram showing execution event history registration processing. FIG. 3 is a diagram showing an example of a conference screen displayed by a PC. FIG. 3 is a diagram showing an example of a conference screen displayed by a PC when a participant clicks or taps a question button. This is an example of a sequence diagram in which the PC accepts the pressing of the question button, and the communication terminal and the display terminal share information that the question button has been pressed. FIG. 3 is a diagram showing an example of a reaction button registration screen displayed by a PC. FIG. 7 is an example of a sequence diagram illustrating a process in which a participant registers a reaction button in a shared support server.

EMBODIMENT OF THE INVENTION Hereinafter, as an example of a form for carrying out the present invention, a communication system and a display method performed by the communication system will be described with reference to the drawings.

<Processing overview>
As transcription progresses in real time using voice recognition, text data that has been voice recognized is displayed one after another, and the oldest text data scrolls out. When a user asks a question based on text data that interests him, the question itself scrolls out, or even if he does not ask a question, the text data that interests him scrolls out. For this reason, it was sometimes difficult for users to ask questions.

The shared support server of this embodiment displays various reaction buttons for the text data of the part of the statement that is the subject of the question. When the question button is pressed, the shared support server displays a message indicating that the text data and the question button were pressed in a separate area from the voice-recognized text data so as not to be buried in the voice-recognized text data. Display.

By doing this, the text data that was the subject of the question and the fact that the question button was pressed are not scrolled out, so it is possible to reduce the likelihood that the answerer will overlook the question. Furthermore, it is possible to reduce situations in which a hearing-impaired person who has difficulty following the flow of a conversation hesitates even when asked a question, or a situation in which a hearing-impaired person is unable to express his or her own intentions.

<About terms>
Sound data is sound converted into data that can be processed as a signal. Sound data may be analog or digital, but it is converted to digital on a computer. The content of the sound data is assumed to be mainly voice, but any sound may be included.

Text data is characters, alphabets, numbers, symbols, etc. expressed by character codes.

Real-time means that if a certain process is currently being executed, the result of the process can be obtained within a certain delay.

The operation information is information indicating that an operation indicating a certain reaction has been performed on text data converted from sound data. In this embodiment, the predetermined operation information will be explained using a question operation as an example.

In addition to a question, the predetermined operation information may be an operation that indicates some kind of reaction to limit scrolling out of text data, such as "wait a moment" or "objection."

<Summary of system configuration>
First, the outline of the configuration of the communication system 1 will be explained with reference to FIG. FIG. 1 is a schematic diagram of a communication system according to this embodiment.

As shown in FIG. 1, the communication system 1 of this embodiment includes a communication terminal 2, a video conference terminal 3, a car navigation device 4, a PC (Personal Computer) 5, a shared support server 6, a schedule management It has a server 8, a speech-to-text conversion server 9, and a display terminal 10.

Furthermore, the communication terminal 2, video conference terminal 3, car navigation device 4, PC 5, shared support server 6, schedule management server 8, speech-to-text conversion server 9, and display terminal 10 can communicate via the communication network N. . The communication network N is constructed by the Internet, a mobile communication network, a LAN (Local Area Network), and the like. The communication network N may include not only wired communication but also wireless communication networks such as 3G (3rd Generation), WiMAX (Worldwide Interoperability for Microwave Access), 4G, 5G, or LTE (Long Term Evolution). .

Further, the communication terminal 2 is used in the conference room X. Video conference terminal 3 is used in conference room Y. An electronic blackboard may be placed in conference rooms X and Y. Furthermore, the union is subject to reservation by each user. The car navigation device 4 is used in the vehicle α. The vehicle α in this case is a vehicle to be car-shared. Vehicles include cars, motorcycles, bicycles, wheelchairs, etc.

"Common entity" refers to goods, services, spaces (rooms), places, or information that are used jointly by multiple people or groups. The conference room X, the conference room Y, and the vehicle α are examples of a shared entity shared by a plurality of users. An example of information given to a union is an account. For example, in a particular service provided on the Web, the number of accounts that can be used is limited to one.

The communication terminal 2 is a general-purpose computer such as a notebook PC, a tablet terminal, or a smartphone. Display terminal 10, PC 5, video conference terminal 3, and car navigation device 4 are also examples of communication terminals. These "communication terminals" are terminals that can be used for video conferences between bases, for example, when a user signs in (see S32 described later). Note that the communication terminal used in the vehicle α includes not only the car navigation device 4 but also a wearable device such as a smartphone or a smart watch in which a car navigation application is installed. The display terminal 10 is a portable terminal such as a smartphone or a general-purpose computer such as a PC (an example of a second terminal). There may be a plurality of display terminals 10, such as display terminals 10a, 10b, etc., and any display terminal is referred to as a "display terminal 10."

The PC 5 is a general-purpose computer, and is an example of a registration device that registers reservations for use of each union and events scheduled to be executed by each user with respect to the schedule management server 8. The event is, for example, a conference, meeting, gathering, getting together, consultation, meeting, driving, riding, or moving. There may be a plurality of PC5s such as PC5a, 5b, etc., and any PC is referred to as "PC5".

The shared support server 6 is a computer, and provides support when sharing a shared entity remotely to each communication terminal (an example of an information processing device).

The schedule management server 8 is a computer that manages reservations for each union and schedules for each user.

The speech-to-text conversion server 9 is a computer, and converts sound (voice) data received from an external computer (for example, the shared support server 6) into text data.

Here, the shared support server 6, schedule management server 8, and speech-to-text conversion server 9 are referred to as a management system. Note that the management system may be, for example, a computer in which all or part of the functions of the shared support server 6, schedule management server 8, and speech-to-text conversion server 9 are integrated. Moreover, each of the shared support server 6, the schedule management server 8, and the speech-to-text conversion server 9 may have a configuration in which each function is distributed to a plurality of computers and realized. Furthermore, although the shared support server 6, schedule management server 8, and speech-to-text conversion server 9 will be described as server computers that exist in a cloud environment, the shared support server 6 and speech-to-text conversion server 9 exist in an on-premises environment. It may also be a server that does Further, the schedule management server 8 may be a server existing in an on-premises environment.

<Hardware configuration>
Next, the hardware configuration of the device or terminal that constructs the communication system 1 will be explained using FIGS. 2 to 4.

<Hardware configuration of video conference terminal>
FIG. 2 is a diagram showing a hardware configuration diagram of a video conference terminal. As shown in FIG. 2, the video conference terminal 3 includes a CPU 301, ROM 302, RAM 303, flash memory 304, SSD 305, media I/F 307, operation button 308, power switch 309, bus line 310, network I/F 311, CMOS (Complementary Metal Oxide Semiconductor) sensor 312, image sensor I/F 313, microphone 314, speaker 315, sound input/output I/F 316, display I/F 317, external device connection I/F 318, short-range communication circuit 319, and short-range communication circuit 319 antennas 319a. Among these, the CPU 301 controls the entire operation of the video conference terminal 3. The ROM 302 stores programs used to drive the CPU 301, such as IPL. RAM 303 is used as a work area for CPU 301. The flash memory 304 stores various data such as communication programs, image data, and sound data. The SSD 305 controls reading and writing of various data to the flash memory 304 under the control of the CPU 301. Note that the SSD may be replaced by an HDD. The media I/F 307 controls reading or writing (storage) of data to a recording medium 306 such as a flash memory. The operation button 308 is a button that is operated when selecting the destination of the video conference terminal 3, for example. The power switch 309 is a switch for switching the power of the video conference terminal 3 ON/OFF.

Further, the network I/F 311 is an interface for data communication using a communication network N such as the Internet. The CMOS sensor 312 is a type of built-in imaging means that images a subject and obtains image data under the control of the CPU 301. The image sensor I/F 313 is a circuit that controls driving of the CMOS sensor 312. The microphone 314 is a type of built-in sound collecting means for inputting sound. The sound input/output I/F 316 is a circuit that processes input/output of sound signals between the microphone 314 and the speaker 315 under the control of the CPU 301 . Display I/F 317 is a circuit that transmits image data to external display 320 under the control of CPU 301. External device connection I/F 318 is an interface for connecting various external devices. The short-range communication circuit 319 is a communication circuit such as NFC or Bluetooth (registered trademark).

Further, the bus line 310 is an address bus, a data bus, etc. for electrically connecting each component such as the CPU 301 shown in FIG. 2.

The display 320 is a type of display unit (display means) configured with a liquid crystal or organic EL that displays an image of a subject, operation icons, and the like. Further, the display 320 is connected to the display I/F 317 via a cable 320c. This cable 320c may be a cable for analog RGB (VGA) signals, a cable for component video, DisplayPort, HDMI (High Definition Multimedia Interface (registered trademark)), or DVI (Digital It may also be a cable for Video Interactive) signals.

Note that the CMOS sensor 312 may be an image sensor such as a CCD (Charge Coupled Device) sensor. External devices such as an external camera, an external microphone, and an external speaker can be connected to the external device connection I/F 318 using a USB cable or the like. When an external camera is connected, the external camera is driven with priority over the built-in CMOS sensor 312 under the control of the CPU 301. Similarly, when an external microphone or external speaker is connected, the external microphone is given priority over the built-in microphone 314 and the built-in speaker 315, respectively, under the control of the CPU 301. or an external speaker.

Furthermore, the recording medium 306 is configured to be detachable from the video conference terminal 3. Furthermore, as long as the flash memory 304 is a nonvolatile memory that reads or writes data under the control of the CPU 301, an EEPROM or the like may be used.

<Hardware configuration of car navigation device>
FIG. 3 is a hardware configuration diagram of the car navigation device. As shown in FIG. 3, the car navigation device 4 includes a CPU 401, a ROM 402, a RAM 403, an EEPROM 404, a power switch 405, an acceleration/direction sensor 406, a media I/F 408, and a GPS receiving section 409.

Among these, the CPU 401 controls the operation of the car navigation device 4 as a whole. The ROM 402 stores programs used to drive the CPU 401, such as IPL. RAM 403 is used as a work area for CPU 401. The EEPROM 404 reads or writes various data such as programs for the car navigation device 4 under the control of the CPU 401 . The power switch 405 is a switch for switching the power of the car navigation device 4 ON/OFF. The acceleration/direction sensor 406 is a variety of sensors such as an electronic magnetic compass, a gyro compass, and an acceleration sensor that detect geomagnetism. A media I/F 408 controls reading or writing (storage) of data to a recording medium 407 such as a flash memory. GPS receiving section 409 receives GPS signals from GPS satellites.

The car navigation device 4 also includes a long-distance communication circuit 411, an antenna 411a of the long-distance communication circuit 411, a CMOS sensor 412, an image sensor I/F 413, a microphone 414, a speaker 415, a sound input/output I/F 416, a display 417, a display It includes an I/F 418, an external device connection I/F 419, a short-range communication circuit 420, and an antenna 420a of the short-range communication circuit 420.

Of these, the long-distance communication circuit 411 receives traffic congestion information, road construction information, traffic accident information, etc. provided from infrastructure outside the vehicle, and conversely transmits information such as the vehicle's location and emergency life-saving signals to the outside. This is a circuit for The external infrastructure is, for example, a road information guidance system such as a VICS (registered trademark) (Vehicle Information and Communication System) system. The CMOS sensor 412 is a type of built-in imaging means that images a subject and obtains image data under the control of the CPU 401. The image sensor I/F 413 is a circuit that controls driving of the CMOS sensor 412. The microphone 414 is a type of built-in sound collection means for inputting audio. The sound input/output I/F 416 is a circuit that processes input/output of sound signals between the microphone 414 and the speaker 415 under the control of the CPU 401 . The display 417 is a type of display unit (display means) such as a liquid crystal or organic EL display that displays an image of a subject, various icons, etc., and has a touch panel function. The touch panel is a type of input means for a user to operate the car navigation device 4. Display I/F 418 is a circuit that causes display 417 to display an image. The external device connection I/F 419 is an interface for connecting various external devices. The short-range communication circuit 420 is a communication circuit such as NFC or (registered trademark). The car navigation device 4 also includes a bus line 410. The bus line 410 is an address bus, a data bus, etc. for electrically connecting each component such as the CPU 401 shown in FIG. 3.

<Hardware configuration of communication terminal, PC, display terminal, and server>
FIG. 4 is a hardware configuration diagram of the communication terminal 2, the PC 5, the display terminal 10, and various servers. The communication terminal 2, the PC 5, and the display terminal 10 are constructed by a computer, and as shown in FIG. It includes a display 508, a network I/F 509, a keyboard 511, a mouse 512, a CD-RW drive 514, a speaker 515, a camera 516, a microphone 517, and a bus line 510.

Among these, the CPU 501 controls the overall operation of the communication terminal 2, the PC 5, and the display terminal 10. The ROM 502 stores programs used to drive the CPU 501 such as IPL. RAM 503 is used as a work area for CPU 501. The HD 504 stores various data such as programs. The HDD controller 505 controls reading and writing of various data to the HD 504 under the control of the CPU 501. A media I/F 507 controls reading or writing (storage) of data to a recording medium 506 such as a flash memory. The display 508 displays various information such as a cursor, menu, window, characters, or images. Display 508 is an example of a display unit (display means). The network I/F 509 is an interface for data communication using the communication network N. The keyboard 511 is a type of input means that includes a plurality of keys for inputting characters, numbers, various instructions, and the like. The mouse 512 is a type of input means for selecting and executing various instructions, selecting a processing target, moving a cursor, and the like. A CD-RW drive 514 controls reading and writing of various data on a CD-RW (Compact Disc ReWritable) 513, which is an example of a removable recording medium. The speaker 515 outputs a sound signal under the control of the CPU 501. The camera 516 captures an image within the viewing angle range under the control of the CPU 501 and generates image data. Microphone 517 collects sound signals under the control of CPU 501.

Furthermore, the communication terminal 2, the PC 5, and the display terminal 10 are equipped with a bus line 510. The bus line 510 is an address bus, a data bus, etc. for electrically connecting each component such as the CPU 501 shown in FIG. 4.

In addition, the shared support server 6 is constructed by a computer, and as shown in FIG. It includes an I/F 609, a keyboard 611, a mouse 612, a CD-RW drive 614, and a bus line 610. These are, in the shared support server 6, a CPU 501, ROM 502, RAM 503, HD 504, HDD controller 505, recording medium 506, media I/F 507, display 508, network I/F 509, keyboard 511, mouse 512, CD-RW drive 514. and the bus line 510, the explanation will be omitted.

Furthermore, the schedule management server 8 is constructed by a computer, and as shown in FIG. It includes an I/F 809, a keyboard 811, a mouse 812, a CD-RW drive 814, and a bus line 810. These are, in the shared support server 6, a CPU 501, ROM 502, RAM 503, HD 504, HDD controller 505, recording medium 506, media I/F 507, display 508, network I/F 509, keyboard 511, mouse 512, CD-RW drive 514. and the bus line 510, the explanation will be omitted.

The speech-to-text conversion server 9 is constructed by a computer, and includes a CPU 901, ROM 902, RAM 903, HD 904, HDD controller 905, recording media 906, media I/F 907, display 908, It includes a network I/F 909, a keyboard 911, a mouse 912, a CD-RW drive 914, and a bus line 910. These are, in the shared support server 6, a CPU 501, ROM 502, RAM 503, HD 504, HDD controller 505, recording medium 506, media I/F 507, display 508, network I/F 509, keyboard 511, mouse 512, CD-RW drive 514. and the bus line 510, the explanation will be omitted.

Note that each of the above programs may be an installable or executable file and may be recorded on a computer-readable recording medium and distributed. Examples of recording media include CD-R (Compact Disc Recordable), DVD (Digital Versatile Disk), Blu-ray Disc, SD card, and the like. Furthermore, the recording medium can be provided domestically or internationally as a program product. For example, the communication terminal 2, the PC 5, and the display terminal 10 implement the display control method according to the present invention by executing the program according to the present invention.

Furthermore, the shared support server 6 may be constructed by a single computer, or may be constructed by a plurality of computers in which each part (function, means, or storage part) is divided and arbitrarily assigned. The same applies to the schedule management server 8 and speech-to-text conversion server 9.

<<Functional configuration of communication system>>
Next, the functional configuration of this embodiment will be explained using FIGS. 5 to 15. FIG. 5 is a functional block diagram of the communication system. Note that, in FIG. 5, among the terminals, devices, and servers shown in FIG. 1, those related to the processing or operation described below are shown.

<Functional configuration of communication terminal>
As shown in FIG. 5, the communication terminal 2 includes a communication section 21, a reception section 22, an image/sound processing section 23, a display control section 24, a judgment section 25, and a storage/read processing section 29. . Each of these units is a function or a means of functioning that is realized when any of the components shown in FIG. be. Furthermore, the communication terminal 2 has a storage unit 2000 constructed by a RAM 503 and an HD 504 shown in FIG.

(Functional configuration of communication terminal)
Next, each component of the communication terminal 2 will be explained. The communication unit 21 is realized by instructions from the CPU 501 and the network I/F 509 shown in FIG. 4, and sends and receives various data (or information) to and from other terminals, devices, or systems via the communication network N. .

The receiving unit 22 is mainly realized by commands from the CPU 501 shown in FIG. 4, a keyboard 511, a mouse 512, and a display 508 with a touch panel, and receives various inputs from the user.

The image/sound processing unit 23 performs image processing on image data obtained by capturing an image of a subject with the camera 516. Furthermore, after the user's voice is converted into an audio signal by the microphone 517, the image/sound processing unit 23 performs audio processing on the sound data related to this audio signal. Further, the image/sound processing unit 23 outputs an audio signal related to the sound data to the speaker 515, and causes the speaker 515 to output audio.

The display control unit 24 is realized by instructions from the CPU 501 shown in FIG. 4, and displays a drawn image on the display 508, accesses the shared support server 6 using a web browser, and displays various screen data. or

The determination unit 25 is realized by instructions from the CPU 501 shown in FIG. 4, and performs various determinations.

The storage/read processing unit 29 is executed by the instructions from the CPU 501 and the HD 504 shown in FIG. I do. Furthermore, image data and sound data received when communicating with other communication terminals or video conference terminals are overwritten and stored in the storage unit 2000 each time they are received. Among these, an image is displayed on the display 508 using the image data before being overwritten, and sound is output from the speaker 515 using the sound data before being overwritten.

Note that the video conference terminal 3 and the car navigation device 4 have the same functions as the communication terminal 2, so a description thereof will be omitted here.

<Functional configuration of PC>
The PC 5 includes a communication section 51, a reception section 52, a display control section 54, a generation section 56, a sound control section 58, and a storage/reading processing section 59. Each of these units is a function or functioning means that is realized when any of the components shown in FIG. . Furthermore, the PC 5 has a storage unit 5000 constructed by the HD 504 shown in FIG.

(Each functional configuration of PC)
Next, each component of the PC 5 will be explained. The communication unit 51 is realized by commands from the CPU 501 and network I/F 509 shown in FIG. 4, and sends and receives various data (or information) to and from other terminals, devices, or systems via the communication network N.

The receiving unit 52 is mainly realized by commands from the CPU 501 shown in FIG. 4, as well as a keyboard 511 and a mouse 512, and receives various inputs from the user.

The display control unit 54 is realized by instructions from the CPU 501 shown in FIG. 4, and displays images on the display 508, accesses the shared support server 6 using a web browser, and displays various screen data. do. For example, the display control unit 54 downloads a Web APP (Web Application) that includes at least HTML (HyperText Markup Language), CSS (Cascading Style Sheets), JavaScript (registered trademark), etc., and displays various image data generated by the Web APP. is displayed on the display 508. The display control unit 54 displays image data generated by HTML5, including data in XML (Extensible Markup Language), JSON (JavaScript (registered trademark) Object Notation), or SOAP (Simple Object Access Protocol) format, on the display 508. to be displayed.

The generation unit 56 is realized by instructions from the CPU 501 shown in FIG. 4, and has a function of generating various image data to be displayed on the display 508. The generation unit 56 generates various image data using the content data received by the communication unit 51. For example, the generation unit 56 generates image data related to the text data, which is content image data, in order to render text data, which is content data, and display the rendered data. Rendering is a process of interpreting data written in a language for web page description (HTML, CSS, XML, etc.) and calculating the arrangement of characters, image data, etc. that are actually displayed on the screen.

The sound control unit 58 is realized by instructions from the CPU 501 shown in FIG. 4, and has a function of outputting a sound signal from the speaker 515. The sound control unit 58 sets sound data to be output from the speaker 515, and reproduces the sound data by outputting a sound signal related to the set sound data from the speaker 515.

The storage/read processing unit 59 is executed by the instructions from the CPU 501 and the HDD controller 505 shown in FIG. 4, and stores various data in the storage unit 5000 and reads various data from the storage unit 5000. Perform processing.

<Functional configuration of shared support server>
The shared support server 6 includes a communication section 61 , an authentication section 62 , a creation section 63 , a generation section 64 , a judgment section 65 , a processing section 66 , and a storage/read processing section 69 . Each of these units has a function or function that is realized when any of the components shown in FIG. It is a means. Further, the shared support server 6 has a storage unit 6000 constructed from the HD 604 shown in FIG. 4 and the like.

(User authentication management table)
FIG. 6(A) is a conceptual diagram showing a user authentication management table. In the storage unit 6000, a user authentication management DB 6001 is constructed, which includes a user authentication management table as shown in FIG. 6(A). In this user authentication management table, a user ID for identifying a user, a user name, an organization ID for identifying an organization to which the user belongs, and a password are managed in association with each other. Note that the organization ID also includes a domain name representing a group or organization for managing multiple computers on a communication network.

(Access management table)
FIG. 6(B) is a conceptual diagram showing the access management table. In the storage unit 6000, an access management DB 6002 is constructed, which includes an access management table as shown in FIG. 6(B). In this access management table, organization IDs, access IDs and access passwords necessary for authentication when accessing the schedule management server 8 are managed in association with each other. This access ID and access password are provided by the shared support server 6 to the service provided by the schedule management server 8 via Web API or the like using the HTTP (Hypertext Transfer Protocol) or HTTPS (Hypertext Transfer Protocol Secure) protocol. functions). The schedule management server 8 manages a plurality of schedulers, and different organizations may use different schedulers, so management using an access management table is required.

(Schedule management table)
FIG. 6(C) is a conceptual diagram showing the schedule management table. In the storage unit 6000, a schedule management DB 6003 is constructed, which includes a schedule management table as shown in FIG. 6(C). In this schedule management table, for each scheduled event ID and execution event ID, the organization ID, the user ID of the person making the reservation, whether or not this person is participating, the name of the person making the reservation, the scheduled start time, the scheduled end time, the event name, etc. The user ID of the participant, presence or absence of participation of other participants, names of other participants, and file data are managed in association with each other.

Among these, the scheduled event ID is identification information for identifying a reserved event. The scheduled event ID is an example of scheduled event identification information for identifying an event scheduled to be executed. The execution event ID is identification information for identifying an event that has actually been executed or an event that is being executed among the reserved events. The execution event ID is an example of execution event identification information for identifying an event that has been executed or an event that is being executed. The reservation person name is the name of the person who reserved the common entity, and is, for example, the name of the organizer when the common entity is a conference room, and is, for example, the name of the driver when the common entity is a vehicle. The scheduled start time indicates the scheduled time to start using the union. The scheduled end time indicates the scheduled end time of use of the union. The event name indicates the name of the event scheduled to be executed by the reservation person. The user IDs of other participants are identification information for identifying participants other than the reservation person. Other participant names are names of participants other than the reservation person, and include unions. In other words, users in this case include not only the person making the reservation and other participants, but also the union. The file data is file data of a material file used in the event corresponding to the scheduled event ID, which is registered by user A using a schedule input screen 550 (see FIG. 18), which will be described later. Further, the file data is data in a predetermined file format created using various applications. The file format of the file data is, for example, a PowerPoint file, an Excel file, or the like.

(Content management table)
FIG. 7 is a conceptual diagram showing a content management table. In the storage unit 6000, a content management DB 6005 is constructed, which includes a content management table as shown in FIG. In this content management table, a content processing ID, content processing type, content content, and start date and time and end date and time of content processing are managed in association with each execution event ID. Here, the content is the content of an executed event that occurs at an event such as a conference, or materials used at the event. Types of content processing include recording, snapshots, speech-to-text conversion, action item generation, and document sending. Further, the content processing ID is identification information for identifying the content processing that occurred in each event.

Here, the content includes history information indicating the execution details of the event and action items caused by the executed event. The history information is data such as recorded data, snapshots, audio text, or materials. A snapshot is a process of acquiring a display screen at a certain point in time of an event being executed as image data. Note that a snapshot is also called, for example, capture or image recognition.

When the content processing type is "recording", the content includes a URL indicating the storage location of the recorded sound data. Further, when the type of content processing is "snapshot", the content includes a URL indicating the storage location of the image data of the screen acquired by the snapshot (capture). Capturing means saving an image (still image, moving image) displayed on the display 508 as image data. Further, when the type of content processing is "speech-to-text conversion", the content includes a URL indicating a storage location of text data of the received speech text.

Here, an action item indicates the content of an action that occurs at an event such as a meeting and that a person related to the event should respond to. When the type of content processing is "Action item generation", the contents include the user ID of the person executing the action item, the deadline for completing the action item, and the URL indicating the storage location of the image data representing the action item. including.

(Reaction type management table)
FIG. 8 is a conceptual diagram showing a reaction type management table. In the storage unit 6000, a reaction type management DB 6006 is constructed, which is composed of a reaction type management table as shown in FIG. In this reaction type management table, an organization ID, registrant ID, execution event ID, target, reaction type, and button image are managed in association with each reaction registration ID. Here, the reaction is a reaction shown by the user such as "like", and more specifically, a reaction button that has been pressed. The reaction registration ID is identification information given to the act of registering a reaction. The target is the range to which the reaction of the reaction registration ID is applied. When the target is an "organization", a reaction button is displayed for all events executed in the organization with the organization ID. When the target is an "event", a reaction button is displayed only for the event executed by the event with the execution event ID. The reaction type indicates what kind of reaction it is, and includes, for example, question and good. The button image is an image of each button forming a reaction button displayed by the communication terminal 2 or the like. The reaction type can also serve as reaction identification information.

(Reaction press management table)
FIG. 9 is a conceptual diagram showing a reaction press management table. In the storage unit 6000, a reaction press management DB 6007 is constructed, which is constituted by a reaction press management table as shown in FIG. The reaction press management table stores operation information regarding reaction button operations. In this reaction press management table, a content processing ID, a user ID, and a reaction type are associated and managed for each reaction press ID. The reaction press ID is identification information given to the act of pressing the reaction button. The content processing ID is the content processing ID of the content whose reaction button was pressed. The user ID is identification information of the user who pressed the reaction button. The reaction type is the type of reaction button that was pressed.

(Each functional configuration of the shared support server)
Next, each functional configuration of the shared support server 6 will be explained in detail. In the following, when explaining each functional configuration of the shared support server 6, the main components for realizing each functional configuration of the shared support server 6 among the components shown in FIG. Also explain the relationship.

The communication unit 61 of the shared support server 6 shown in FIG. 5 is realized by the command from the CPU 601 shown in FIG. 4 and the network I/F 609 shown in FIG. Sends and receives various data (or information) to and from other terminals, devices, or systems.

The authentication unit 62 is realized by instructions from the CPU 601 shown in FIG. Authentication is performed by determining whether the information is authentic.

The creation unit 63 is realized by instructions from the CPU 601 shown in FIG. 4, and creates the information as shown in FIG. Various screens such as the reservation list screen 230 are created.

The generation unit 64 is realized by instructions from the CPU 601 shown in FIG. 4, and generates an execution event ID, a content processing ID, and a storage destination URL.

The determination unit 65 is realized by instructions from the CPU 601 shown in FIG. 4, and performs various determinations. This determination will be described later.

When a reaction button is pressed on the PC 5, the processing unit 66 manages the reaction by associating identification information of the reaction button, text data on which the reaction button was pressed, user ID who pressed the reaction button, type of reaction, etc. Save to table. Furthermore, when the type of reaction is a question, the processing unit 66 requests the PC 5 to display text data indicating that the question button was pressed and a message to that effect.

The storage/read processing unit 69 is realized by instructions from the CPU 601 shown in FIG. 4 and the HDD controller 605 shown in FIG. Performs processing such as reading out various stored data.

<Functional configuration of schedule management server>
The schedule management server 8 includes a communication section 81 , an authentication section 82 , a generation section 83 , and a storage/read processing section 89 . Each of these units has a function or function that is realized when any of the components shown in FIG. It is a means. Furthermore, the schedule management server 8 has a storage unit 8000 constructed by the HD 804 shown in FIG.

(User authentication management table)
FIG. 10(A) is a conceptual diagram showing a user authentication management table. In the storage unit 8000, a user authentication management DB 8001 is constructed, which includes a user authentication management table as shown in FIG. 10(A). In this user authentication management table, a user ID for identifying a user is managed in association with an organization ID and password for identifying an organization to which the user belongs.

(User management table)
FIG. 10(B) is a conceptual diagram showing the user management table. In the storage unit 8000, a user management DB 8002 is constructed, which includes a user management table as shown in FIG. 10(B). In this user management table, a user ID and a user name (user name) indicated by the user ID are associated and managed for each organization ID.

(union management table)
FIG. 10C is a conceptual diagram showing the union management table. In the storage unit 8000, a union management DB 8003 is constructed, which is composed of a union management table as shown in FIG. 10(C). In this union management table, a union ID for identifying a union and a name of the union (union name) are managed in association with each organization ID.

(Union reservation management table)
FIG. 11A is a conceptual diagram showing a union reservation management table. In the storage unit 8000, a union reservation management DB 8004 is constructed, which is composed of a union reservation management table as shown in FIG. 11(A). In this union reservation management table, reservation information in which each piece of information is associated is managed. The reservation information includes, for each organization ID, a union ID, a union name, a user ID of the person making the reservation, a scheduled start date and time of use, a scheduled end date and time of use, and an event name. Among these, the scheduled start date and time of use indicates the scheduled start date and time of use of the union. The scheduled end date and time of use indicates the scheduled end date and time of use of the union. Each date and time indicates year, month, day, hour, minute, second, and time zone, but in FIG. 11(A), due to space limitations, year, month, day, hour, and minute are also indicated.

(Event management table)
FIG. 11(B) is a conceptual diagram showing the event management table. In the storage unit 8000, an event management DB 8005 is constructed, which is composed of an event management table as shown in FIG. 11(B). This event management table manages schedule information in which each piece of information is associated. The schedule information is managed in association with an organization ID, user ID, user name, scheduled event start date and time, scheduled event end date and time, and event name for each scheduled event ID. Among these, the scheduled event start date and time indicates the scheduled start date and time when the event is to be executed. The scheduled event end date and time indicates the scheduled end date and time when the event is executed. Each date and time indicates year, month, day, hour, minute, second, and time zone, but in FIG. 11(B), due to space limitations, year, month, day, hour, and minute are also indicated. Further, in the event management table, file data of material files used in the event indicated in the schedule information is managed in association with the schedule event ID.

(Server authentication management table)
FIG. 12(A) is a conceptual diagram showing the server authentication management table. In the storage unit 8000, a server authentication management DB 8006 is constructed, which includes a server authentication management table as shown in FIG. 12(A). In this server authentication management table, access IDs and access passwords are managed in association with each other. The access ID and access password have the same concept as the access ID and access password managed by the access management DB 6002 of the shared support server 6.

(Execution event history management table)
FIG. 12(B) is a conceptual diagram showing the execution event history management table. In the storage unit 8000, an execution event history management DB 8008 is constructed, which is composed of an execution event history management table as shown in FIG. 12(B). In this execution event history management table, a content processing ID, content processing type, content content, and start date and time and end date and time of the content processing are managed in association with each execution event ID. This execution event history management DB 8008 manages some of the same data as the content management DB 6005. The same data includes an execution event ID, a content processing ID, a content processing type, a start date and time, and an end date and time of the content processing. The contents of the contents are the same except for the notation method (http:// or c://) of the storage destination of the content data.

(Execution event management table)
FIG. 13(A) is a conceptual diagram showing an execution event management table. In the storage unit 8000, an execution event management DB 8009 is constructed, which is composed of an execution event management table as shown in FIG. 13(A). In this execution event management table, an execution event ID is managed in association with an event name, a start date and time, and an end date and time of the event. This execution event management DB 8009 manages information regarding actually executed events among the schedule information managed in the event management DB 8005.

(Related information management table)
FIG. 13(B) is a conceptual diagram showing the related information management table. In the storage unit 8000, a related information management DB 8010 is constructed, which includes a related information management table as shown in FIG. 13(B). In this related information management table, related information in which each piece of information (data) is associated is managed for each execution event ID. The related information includes the time period of content occurrence, sound data, audio text data, and screen data, which are managed in association with each other. The content occurrence time zone indicates the elapsed time from the start date and time of the event at the time when the content occurred in the executed event. Here, the generation time period of the content is generated by the generation unit 83 based on the start date and time of the event stored in the event management DB 8005 and the start date and time and end date and time of the content processing stored in the execution event history management DB 8008. Ru. The time period in which content occurs is an example of time information. The sound data also includes a content processing ID and a content type. The audio text data and screen data include a content processing ID, content type, and sequence number. The sequence number indicates the chronological order in which content processing occurred for each of audio text data and screen data.

(Text information management table)
FIG. 14 is a conceptual diagram showing the text information management table. In the storage unit 8000, a text information management DB 8012 is constructed, which is composed of a text information management table as shown in FIG. In this text information management table, text information including voice text data generated in an executed event is managed for each execution event ID. This text information is associated with a content processing ID, a text ID for identifying the text data, and a transcript indicating the content of the text data. Among these, the content processing ID identifies content processing whose type of content processing is "speech-to-text conversion." Further, transcript (text data) is text data indicating the content of the content associated in the execution event history management DB 8008 with the same content processing ID as the associated content processing ID as text information.

(Functional configuration of schedule management server)
Next, each functional configuration of the schedule management server 8 will be explained in detail. In the following, when explaining each functional configuration of the schedule management server 8, the main components for realizing each functional configuration of the schedule management server 8 among the components shown in FIG. Also explain the relationship.

The communication unit 81 of the schedule management server 8 shown in FIG. 5 is realized by the command from the CPU 801 shown in FIG. 4 and the network I/F 809 shown in FIG. Send and receive various data (or information) with other terminals, devices, or systems.

The authentication unit 82 is realized by instructions from the CPU 801 shown in FIG. Authentication is performed by determining whether the information is authentic. Further, the authentication unit 82 performs authentication by determining whether the information (access ID and access password) sent from the shared support server 6 is information registered in advance in the server authentication management DB 8006.

The generation unit 83 is realized by instructions from the CPU 801 shown in FIG. 4, and has a function of generating related information to be registered in the related information management DB 8010.

The storage/read processing unit 89 is realized by instructions from the CPU 801 shown in FIG. 4 and the HDD controller 805 shown in FIG. Performs processing such as reading out various stored data. The storage/read processing unit 89 is an example of storage control means.

<Functional configuration of speech-to-text conversion server>
The speech-to-text conversion server 9 includes a communication section 91, a conversion section 93, and a storage/read processing section 99. Each of these units is a function or functioning means that is realized when any of the components shown in FIG. . The speech-to-text conversion server 9 also has a storage unit 9000 constructed by the HD 904 shown in FIG.

<Functional configuration of the speech-to-text conversion server>
Next, each functional configuration of the speech-to-text conversion server 9 will be explained in detail. In the following, when explaining each functional configuration of the speech-to-text conversion server 9, the main components for realizing each functional configuration of the speech-to-text conversion server 9 among the components shown in FIG. Also explain the relationship.

The communication unit 91 of the speech-to-text conversion server 9 shown in FIG. 5 is realized by the command from the CPU 901 shown in FIG. 4 and the network I/F 909 shown in FIG. Sends and receives various data (or information) to and from other terminals, devices, or systems via the computer.

The conversion unit 93 is realized by a command from the CPU 901 shown in FIG. 4, and converts the sound data received by the communication unit 91 via the communication network N into text data.

The storage/read processing unit 99 is realized by the commands from the CPU 901 shown in FIG. 4 and the HDD controller 905 shown in FIG. Performs processing such as reading out various stored data.

Note that each of the above IDs is an example of identification information. Further, the organization ID includes a company name, a business office name, a department name, a region name, and the like. User IDs include employee numbers, driver's license numbers, My Numbers in Japan's social security and tax number system, etc.

<Functional configuration of display terminal>
Next, each component of the display terminal 10 will be explained. The display terminal 10 includes a communication section 11, a reception section 12, a display control section 13, and a storage/read processing section 19. Each of these units is a function or a means of functioning that is realized when any of the components shown in FIG. be. Furthermore, the display terminal 10 has a storage unit 1000 constructed by a RAM 503 and an HD 504 shown in FIG.

The communication unit 11 is realized by instructions from the CPU 501 and the network I/F 509 shown in FIG. 4, and sends and receives various data (or information) to and from other terminals, devices, or systems via the communication network N. .

The receiving unit 12 is mainly realized by commands from the CPU 501 shown in FIG. 4, a keyboard 511, a mouse 512, and a display 508 with a touch panel, and receives various inputs from the user.

The display control unit 13 is realized by instructions from the CPU 501 shown in FIG. 4, and displays a drawn image on the display 508, accesses the shared support server 6 using a web browser, and displays various screen data. or

The storage/read processing unit 19 is executed by the instructions from the CPU 501 and the HD 504 shown in FIG. I do. Further, image data and sound data received when communicating with other communication terminals or video conference terminals are overwritten and stored in the storage unit 1000 each time they are received. Among these, an image is displayed on the display 508 using the image data before being overwritten, and sound is output from the speaker 515 using the sound data before being overwritten.

Note that although the display terminal 10 may have the same functions as the communication terminal 2, only the main functions are shown in FIG. 5 for convenience of explanation.

<Processing or operation of embodiment>
Hereinafter, the processing or operation of each embodiment will be described.

<Schedule registration process>
Here, a process in which reservation person A (Rikotaro) registers his own schedule from PC 5 to schedule management server 8 will be explained using FIGS. 15 to 18. FIG. 15 is a sequence diagram showing the schedule registration process. FIG. 16 is a diagram showing the sign-in screen. FIG. 17 is an example of the initial screen of the PC. FIG. 18 shows a schedule input screen. First, when user A operates the keyboard 511 of the PC 5, the display control unit 54 of the PC 5 displays a sign-in screen on the display 508 as shown in FIG. A sign-in screen 530 for performing the following is displayed (step S11). This sign-in screen 530 includes an input field 531 for entering the user ID and organization ID of the user, an input field 532 for entering the password, a sign-in button 538 to be pressed when signing in, and a sign-in button 538 for signing in. A cancel button 539 is included to be pressed when canceling. Here, the user ID and organization ID are user A's e-mail address. The user name part of the e-mail address is the user ID, and the domain name part is the organization ID. Note that the input field 531 may be an input field for inputting a user ID and an organization ID separately, instead of an e-mail address.

Next, when User A enters his or her user ID and organization ID in the input field 531, enters his or her password in the input field 532, and presses the sign-in button 538, the reception unit 52 prompts the user to sign in. The request is accepted (step S12). Then, the communication unit 51 of the PC 5 transmits sign-in request information indicating a request for sign-in to the schedule management server 8 (step S13). This sign-in request information includes the information (user ID, organization ID, and password) accepted in step S12. Thereby, the communication unit 81 of the schedule management server 8 receives the sign-in request information.

Next, the authentication unit 82 of the schedule management server 8 authenticates the user A using the user ID, organization ID, and password (step S14). Specifically, in the user authentication management DB 8001 (see FIG. 10(A)), the storage/read processing unit 89 stores the user information corresponding to the set of user ID, organization ID, and password received in step S13. Search for a set of ID, organization ID, and password. If there is a corresponding pair, the authentication unit 82 determines that the requesting user A is an authorized user. If there is no corresponding pair, the authentication unit 82 determines that user A is an unauthorized (unauthorized) user. If it is not valid, the communication unit 81 notifies the PC 5 that it is not valid, but here we will continue to explain the case where it is valid.

Next, the communication unit 81 transmits the authentication result to the PC 5 (step S15). Thereby, the communication unit 51 of the PC 5 receives the authentication result.

Next, when the generation unit 56 of the PC 5 receives the authentic authentication result in step 15, it generates an initial screen 540 as shown in FIG. 17 (step S16). Then, the display control unit 54 of the PC 5 displays an initial screen 540 as shown in FIG. 17 on the display 508 (step S17). This initial screen 540 includes a "schedule registration" button 541 that is pressed when registering a schedule, and a "view execution event history" button 543 that is pressed when viewing the execution event history. Here, when the user presses the "schedule registration" button 541, the reception unit 52 accepts schedule registration (step S18). The communication unit 51 then transmits a schedule registration request to the schedule management server 8 (step S19). Thereby, the communication unit 81 of the schedule management server 8 receives the schedule registration request.

Next, the storage/read processing unit 89 of the schedule management server 8 searches the user management DB 8002 (see FIG. 10(B)) using the organization ID received in step S13 as a search key, thereby searching all the corresponding The user ID and all user names are read (step S20). Then, the communication unit 81 transmits schedule input screen information to the PC 5 (step S21). This schedule input screen information includes all user IDs and all user names read out in step S20. All the user names also include the name of the reservation person who is the user A who made the input for sign-in in step S12. Thereby, the communication unit 51 of the PC 5 receives the schedule input screen information.

Next, in the PC 5, the generation unit 56 generates the schedule input screen 550 using the schedule input screen information received in step S21 (step S22). Then, the display control unit 24 of the PC 5 displays a schedule input screen 550 as shown in FIG. 18 on the display 508 (step S23).

On this schedule input screen 550, an input field 551 for inputting an event name, an input field 552 for inputting a union ID or a union name, and a scheduled start date and time of event execution (union use) are input. an input field 553 for inputting the scheduled end date and time of the event execution (use of the union), an input field 555 for inputting notes such as an agenda, and a display area for displaying the reservation person's name. 556, a selection menu 557 for selecting the participant name of a participant other than the person making the reservation, an "OK" button 558 that is pressed when registering a reservation, and a button 558 that is pressed when registering a reservation, and when canceling the entered contents. It includes a "CANCEL" button 559 that can be pressed. The reservation user name is the name of the user who made the input to sign in to the PC 5 in step S12. A mouse pointer p1 is also displayed.

Note that an e-mail address may be entered in the input field 552. Furthermore, when a union name is selected in the selection menu 557, the union is also added as another participant.

Next, user A enters predetermined items in input fields 551 to 555, uses mouse pointer p1 to select the name of the user who wants to participate in the event (user name) from selection menu 557, When the "OK" button 558 is pressed, the receiving unit 52 receives input of schedule information (step S24). The communication unit 51 then transmits the schedule information to the schedule management server 8 (step S25). This schedule information includes the event name, union ID (or union name), scheduled start date and time, scheduled end date and time, user ID of each participant, and memo. In addition, on the schedule input screen 550, if a union ID is input in the input field 552, this union ID is transmitted, and if a union name is input in the input field 552, this union name is transmitted. is sent. Further, on the schedule input screen 550, a user name is selected in the selection menu 557, but since the user ID has also been received in step S21, the user ID corresponding to the user name is transmitted. Thereby, the communication unit 81 of the schedule management server 8 receives the schedule information.

Next, the schedule management server 8 searches the union management DB 8003 (see FIG. 10(C)) using the union ID (or union name) received in step S25 as a search key to find the corresponding union. name (or union ID) is read out (step S26).

Next, the storage/read processing unit 89 stores the reservation information in the union reservation management DB 8004 (see FIG. 11(A)) (step S27). In this case, the storage/read processing unit 89 adds one record of reservation information to the union reservation management table of the union reservation management DB 8004 managed by the scheduler registered in advance. The reservation information is configured based on the schedule information received in step S25 and the union name (or union ID) read out in step S26. Note that the scheduled usage start date and time in the union reservation management DB 8004 corresponds to the scheduled start date and time in the schedule information. Further, the scheduled usage end date and time in the union reservation management DB 8004 corresponds to the scheduled end date and time in the schedule information.

Further, the storage/read processing unit 89 stores the schedule information in the event management DB 8005 (see FIG. 11(B)) (step S28). In this case, the storage/read processing unit 89 adds one record of schedule information to the event management table of the event management DB 8005 managed by the scheduler registered in advance. The schedule information is configured based on the schedule information received in step S25. Note that the scheduled event start date and time in the event management DB 8005 corresponds to the scheduled start date and time in the schedule information. Furthermore, the scheduled end date and time of the event in the event management DB 8005 corresponds to the scheduled end date and time in the schedule information.

As described above, user A can register his or her own schedule in the schedule management server 8. Although FIGS. 15 to 18 describe the case in which the schedule is registered using the PC 5, a user who operates the communication terminal 2, video conference terminal 3, or car navigation device 4 may register the schedule using the same process. Good too.

<Event start processing>
Here, with reference to FIGS. 19 to 27, a process of holding a conference with other participants using the communication terminal 2 in a conference room X reserved by a reservation person A (Rikotaro) will be described. 19 and 22 are sequence diagrams showing event start processing. FIG. 20 is a diagram showing a sign-in screen displayed on a communication terminal. FIG. 21 is a diagram showing a reservation list screen for a union. FIG. 23 is a diagram showing a detailed event information screen. FIG. 24 is a diagram showing a display screen displayed on the communication terminal after signing in.

First, when user A presses the power switch 222 of the communication terminal 2, the reception unit 22 of the communication terminal 2 receives a power-on request (or activation of an application) from the user (step S31). Then, the display control unit 24 of the communication terminal 2 displays a sign-in screen 110 for signing in on the display 508, as shown in FIG. 20 (step S32). This sign-in screen 110 includes a selection icon 111 that is pressed when user A signs in using his or her own IC card, and a selection icon 111 that user A presses to sign in using his or her own IC card. It includes a selection icon 113 that is pressed when signing in by entering a password, and a power icon 115 (or an app end button) that is pressed when turning off the power without signing in.

Next, when the user A presses the selection icon 113 and inputs his/her e-mail address and password, the reception unit 22 accepts the sign-in request. (Step S33). Then, the communication unit 21 transmits sign-in request information indicating a sign-in request to the shared support server 6 (step S34). This sign-in request information includes the information accepted in step S33 (user ID, organization ID, and password), time zone information of the country or region where the communication terminal 2 is installed, and the user ID of the communication terminal 2. , organization ID and password. Thereby, the communication unit 61 of the shared support server 6 receives the sign-in request information.

Next, the authentication unit 62 of the shared support server 6 authenticates the user A using the user ID, organization ID, and password of the user A received in step S34 (step S35). Specifically, the storage/reading processing unit 69 uses the user ID, organization ID, and password of the user A received in step S34 as search keys in the user authentication management DB 6001 (see FIG. 6(A)). , searches for a set of user ID, organization ID, and password that corresponds to the above-received set of user ID, organization ID, and password of user A. If there is a corresponding pair, the authentication unit 62 determines that the requesting user A is an authorized user. If there is no corresponding pair, the authentication unit 62 determines that the requesting user A is an unauthorized (unauthorized) user. If it is not valid, the communication unit 61 notifies the communication terminal 2 that it is not valid, but here we will continue to explain the case where it is valid.

Next, the storage/read processing unit 69 of the shared support server 6 searches the access management DB 6002 (see FIG. 6(B)) using the organization ID of user A received in step S34 as a search key. The access ID and access password are read out (step S36).

Next, the communication unit 61 transmits reservation request information indicating a request for reservation information of the union and schedule request information indicating a request for user schedule information to the schedule management server 8 (step S37). The reservation request information and schedule request information include the time zone information, the user ID and organization ID of the communication terminal received in step S34, and the access ID and password read out in step S36. Thereby, the communication unit 81 of the schedule management server 8 receives the reservation request information and the schedule request information.

Next, the authentication unit 82 of the schedule management server 8 authenticates the shared support server 6 using the access ID and the access password (step S38). Specifically, the storage/read processing unit 89 stores the set of access ID and access password corresponding to the set of access ID and access password received in step S37 in the server authentication management DB 8006 (see FIG. 12(A)). Search for. If there is a corresponding pair, the authentication unit 82 determines that the request source shared support server 6 is an authorized accesser. If there is no corresponding pair, the authentication unit 82 determines that the requesting shared support server 6 is an unauthorized (unauthorized) accessor. If it is not valid, the communication unit 81 notifies the shared support server 6 that it is not valid, but here we will continue to explain the case where it is valid.

The storage/read processing unit 89 of the schedule management server 8 uses the user ID of the communication terminal received in step S37 as a search key to search the union reservation management DB 8004 (see FIG. 11(A)) managed by the scheduler. By searching, the corresponding reservation information is read (step S39). In this case, the storage/reading processing unit 89 reads the reservation information whose usage start date and time is today.

Furthermore, the storage/read processing unit 89 uses the user ID of the communication terminal received in step S37 as a search key to search the event management DB 8005 (see FIG. 11(B)) managed by the scheduler. The corresponding schedule information is read (step S40). In this case, the storage/reading processing unit 89 reads out schedule information in which the scheduled event start date and time is today. If the schedule management server 8 is located in a country or region different from the communication terminal 2, the time zone is adjusted based on the time zone information to match the country or region where the communication terminal is installed. .

Next, the communication unit 81 transmits the reservation information read out in step S39 and the schedule information read out in step S40 to the shared support server 6 (step S41). Thereby, the communication unit 61 of the shared support server 6 receives the reservation information and schedule information.

Next, the creation unit 63 of the shared support server 6 creates a reservation list based on the reservation information and schedule information received in step S41 (step S42). Then, the communication unit 61 transmits reservation list information indicating the contents of the reservation list to the communication terminal 2 (step S43). Thereby, the communication unit 21 of the communication terminal 2 receives the reservation list information.

Next, in the communication terminal 2, the display control unit 24 displays a reservation list screen 230 as shown in FIG. 21 on the display 508 (step S44). This reservation list screen 230 includes a display area 231 that displays a union name (here, a place name) and a display area 232 that displays today's date and time. Further, on the reservation list screen 230, event information 235, 236, 237, etc. indicating the event using today's main community (here, conference room X) is displayed. Each event information includes, for each event, the scheduled start time and end time of use of the union, the event name, and the name of the person who reserved the union (reserver name). The event information includes start buttons 235s, 236s, 237s, etc. that are pressed when the user specifies an event to start.

Next, in FIG. 22, when the user A presses the start button 235s, the reception unit 22 accepts the selection of the event indicated by the event information 235 (step S51). Next, the communication unit 21 of the communication terminal 2 transmits the scheduled event ID indicating the scheduled event selected in step S51 above to the shared support server 6 (step S52). The process in step S52 is a process for requesting transmission of execution event identification information. Thereby, the communication unit 61 of the shared support server 6 receives the selected scheduled event ID.

Next, in the shared support server 6, the generation unit 64 generates a unique execution event ID (step S53). Then, the storage/reading processing unit 69 manages the execution event ID generated in step S53, the reservation event ID received in step S52, the user ID and organization ID of the reservation person, and the event information in association with each other (step S54).

Note that the user ID and organization ID of the reservation person, and the event information are IDs and information based on the reservation information and schedule information received in step S41. Note that, at this point, there is no entry in the participation/non-participation column in the reservation management table (see FIG. 6(C)).

Next, in the shared support server 6, the communication unit 61 transmits file data transmission request information indicating a transmission request for file data registered in the schedule management server 8 to the schedule management server 8 (step S55). The file data transmission request information includes the scheduled event ID received in step S52, the user ID and organization ID of the communication terminal received in step S34, and the access ID and access password read out in step S36. ing. Thereby, the communication unit 81 of the schedule management server 8 receives the file data transmission request information.

Next, the storage/read processing unit 89 of the schedule management server 8 searches the event management DB 8005 (see FIG. 11(B)) using the scheduled event ID received in step S55 as a search key. The associated file data is read (step S56). The communication unit 81 then transmits the file data read in step S56 to the shared support server 6 (step S57). Thereby, the communication unit 61 of the shared support server 6 receives the file data.

Next, the storage/read processing unit 69 of the shared support server 6 associates the scheduled event ID received in step S52 with the execution event ID generated in step S53, and stores the file data received in step S57. It is stored and managed in the schedule management DB 6003 (see FIG. 6(C)) (step S58).

The communication unit 61 then transmits the execution event ID generated in step S53 and the file data received in step S57 to the communication terminal 2 (step S59). Thereby, the communication unit 21 of the communication terminal 2 receives the execution event ID and the file data.

Next, in the communication terminal 2, the storage/read processing unit 29 stores the execution event ID and file data in the storage unit 2000 (step S60). Here, the file data transmitted from the shared support server 6 is stored in a specific storage area of the storage unit 2000. Thereby, in the communication terminal 2, by accessing the specific storage area during the execution of the event, the display control unit 24 causes the display 508 to display the file data stored in the specific storage area. Here, the specific storage area is a temporary storage destination for data provided for each event being executed, and is a storage area specified by an arbitrary path (character string) indicating the location within the storage unit 2000. It is. Note that the specific storage area is not limited to the configuration provided inside the communication terminal 2, but may be an external storage device connected to the communication terminal 2, or a local server, etc. that exists in an on-premises environment and can communicate with the communication terminal 2. The structure may be provided.

Next, the display control unit 24 displays a detailed information screen 250 of the selected event on the display 508, as shown in FIG. 23 (step S61). The event detailed information screen 250 includes a display area 251 for the event name, a display area 252 for the scheduled event time (scheduled start time and scheduled end time), and a display area 253 for the reservation person's name. Further, the event detailed information screen 250 includes a display area 256 for displaying the contents of the memo, a display area 257 for displaying the names of prospective participants, and files stored in a specific storage area of the storage unit 2000. A display area 258 is displayed for displaying identification information (eg, file name) for identifying data. In the display area 257, the name of the person making the reservation shown in FIG. 18 and the names of other selected participants are displayed, and each person who is scheduled to participate is checked to see who is actually participating in the conference. A check box is displayed. Further, the display area 258 displays the file name of the file data stored in a specific storage area of the storage unit 2000, that is, the file data downloaded from the shared support server 6, and also displays the file name of the file data downloaded from the shared support server 6. The file name of the file data inside is also displayed. Further, the lower right side of the event detailed information screen 250 includes a "close" button 259 that is pressed to close the detailed information screen 250.

Next, when the user checks the checkbox of the user who is actually participating among the names of the planned participants and presses the "Close" button 259, the reception unit 22 accepts the selection of the participant. (Step S62). Then, the communication unit 21 transmits the user ID of each prospective participant and information indicating whether or not to participate to the shared support server 6 (step S63). Thereby, the communication unit 61 of the shared support server 6 receives the user ID of each prospective participant and information indicating whether or not they will participate.

Next, the shared support server 6 stores and manages the presence/absence of participation in the uninputted participation/presence column in the schedule management DB 6003 (step S64).

As described above, user A starts an event (here, a policy-setting meeting) using the shared entity (here, conference room X) and the communication terminal 2. When the event starts, the display control unit 24 of the communication terminal 2 causes the display 508 to display a display screen 100a as shown in FIG. 24.

FIG. 24 is a diagram showing a display screen displayed on a communication terminal when an event starts. The display screen 100a shown in FIG. 24 includes a menu bar 121, time information 124 indicating the elapsed time from the start of the event or the remaining time until the end of the event, and a power icon 117 that is pressed when turning off the power of the communication terminal 2. It is included. This menu bar 121 includes a plurality of operation icons 125 (125a to 125j) that are selected (pressed) when performing various processes during event execution. The operation icon 125a is selected (pressed) to view detailed information of the event being executed. The operation icon 125b is selected (pressed) when starting various external applications 103. The operation icon 125c is selected (pressed) when viewing file data stored in a specific storage area of the storage unit 2000. The operation icon 125d is selected (pressed) when switching the display of the application display screen of the external application 103 that is currently running. The operation icon 125e is selected (pressed) when changing the screen size of the application display screen of the external application 103. The operation icon 125f is selected (pressed) when performing various operations related to the event being executed. The operation icon 125g is selected (pressed) when capturing the display screen 100 displayed on the display 508. The operation icon 125h is selected (pressed) when ending an event in progress. The operation icon 125i is selected (pressed) when starting the browser application 103c to perform a search using the browser. The operation icon 125j is selected (pressed) when inputting characters, numbers, or the like.

Here, various icons included in the display screen displayed on the communication terminal 2 are an example of a "reception area". The reception area may include not only images such as icons and buttons, but also characters such as "change" or a combination of images and characters. The images here may be not only symbols and figures, but also illustrations, patterns, and other images that can be visually recognized by the user. Further, selection (pressing) of various icons is an example of an operation on various icons. Operations on various icons include, for example, input operations on the display 508 using the keyboard 511 and mouse 512.

Then, user A can hold a conference in conference room X using communication terminal 2. For example, when the user A of the communication terminal 2 presses the operation icon 125c, the reception unit 22 receives the selection of the operation icon 125c, and the display control unit 24 displays the information stored in a specific storage area of the storage unit 2000. Display file data of a file. Note that the display control unit 24 of the communication terminal 2 displays not only the file data received in step S59 but also the file data stored in advance in the storage unit 2000 or the file data newly generated in the event that is being started. It may be displayed. In this case, the storage/read processing unit 29 of the communication terminal 2 stores the file data generated or updated in the event that has been started in a specific storage area of the storage unit 2000.

<Execution event history registration process>
Next, the execution event history registration process will be described using FIGS. 25 to 27. FIG. 25 is a sequence diagram showing execution event history registration processing. FIG. 26 is a flowchart showing the speech-to-text conversion process.

First, the determining unit 25 of the communication terminal 2 determines the type of content processing in the event that is being started (step S71). Specifically, when the content is sound data generated by recording by the image/sound processing unit 23, the determining unit 25 determines that the type of content processing is “recording”. If the content is image data acquired by a snapshot (capture) by the image/sound processing unit 23, the determining unit 25 determines that the content processing type is “snapshot”. If the content is material file data transmitted by the communication unit 21, the determining unit 25 determines that the content processing type is “material transmission”.

Next, the communication unit 21 transmits registration request information indicating the generated content registration request to the shared support server 6 (step S72). In this case, the communication unit 21 automatically transmits registration request information every time content occurs. This registration request information includes an execution event ID, a content transmission source user ID, content data, and content processing type information. Thereby, the communication unit 61 of the shared support server 6 receives the registration request information.

Next, the determination unit 65 of the shared support server 6 determines the type of content processing received based on the type information included in the registration request information received by the communication unit 61 (step S73). Then, when the determination unit 65 determines that the type of content processing is “recording”, the communication unit 61 of the shared support server 6 transmits sound data, which is content data, to the voice-to-text conversion server 9 (step S74). Thereby, the communication unit 91 of the speech-to-text conversion server 9 receives the sound data. Note that if the type of content processing is other than "recording", the shared support server 6 moves to the process of step S77 without performing the processes of steps S74 to S76.

Next, the conversion unit 93 of the speech-to-text conversion server 9 converts the sound data received by the communication unit 91 into text data (step S75). Here, the speech-to-text conversion process in the speech-to-text conversion server 9 will be explained using FIG. 26. First, the conversion unit 93 obtains information indicating the date and time when the sound data was received by the communication unit 91 (step S75-1). Note that the information acquired in step S75-1 may be information indicating the date and time when the shared support server 6 received the sound data or the date and time when the shared support server 6 transmitted the sound data. In this case, the communication unit 91 of the speech-to-text conversion server 9 receives the sound data and information indicating the date and time transmitted from the shared support server 6 in step S74.

Next, the conversion unit 93 executes text conversion processing of the sound data received by the communication unit 91 (step S75-2). Then, when the text conversion process of the sound data is completed (YES in step S75-3), the conversion unit 93 moves the process to step S75-4. On the other hand, the conversion unit 93 repeats the process of step S75-2 until the audio data text conversion process is completed. Here, in step S75-3, if a predetermined amount of text conversion has been performed based on the sound data received by the communication unit 91, the conversion unit 93 determines that the text conversion process of the sound data is completed. do. For example, when the text conversion for one sentence has been performed, the conversion unit 93 determines that the text conversion process of the sound data is completed. The conversion unit 93 then generates text data converted from the sound data (step S75-4). Thereby, the speech-to-text conversion server 9 converts the sound data sent from the shared support server 6 into text data. Since the speech-to-text conversion server 9 receives sound data transmitted from the shared support server 6 at any time, it repeatedly executes the process shown in FIG. 26.

Returning to FIG. 25, the description of the execution event history registration process will be continued. The communication unit 91 transmits the text data converted by the conversion unit 93 to the shared support server 6 (step S76). Here, the communication unit 91 transmits information indicating the date and time acquired in step S75-1 to the shared support server 6 along with the text data.

Next, the generation unit 64 generates a unique content processing ID for identifying the content processing that occurred due to the event (step S77). Furthermore, the generation unit 64 generates a URL of content data indicating the contents of the content (step S78). Then, the storage/read processing unit 69 stores the type of content processing, the start date and time of the content processing, and the end date and time of the content processing in step S77 for each execution event ID received in step S72 for the content management DB 6005 (see FIG. 7). The content processing ID generated in step S78 and the URL indicating the storage location of the content generated in step S78 are managed in association with each other (step S79).

Here, when the type of content processing is "speech-to-text conversion", the start date and time and end date and time of content processing are the dates and times when sound data was converted to text data. Here, the date and time when the sound data was converted into text data is the date and time when the sound data was transmitted by the communication unit 61 of the shared support server 6 and the date and time when the text data was received by the communication unit 61 of the shared support server 6. be. Note that the date and time when the sound data was converted to text data is the date and time when the sound data was received by the communication unit 91 of the speech-to-text conversion server 9 and the date and time when the text data was sent by the communication unit 91 of the speech-to-text conversion server 9. There may be. Furthermore, if the content processing type is "speech-to-text conversion", the start date and time and end date and time of the content processing may be the same as the start date and time and end date and time of the content processing related to the audio data to be converted into text. .

In addition, when the type of content processing is "recording", "snapshot", and "material transmission", the start date and time and end date and time of content processing are determined by the communication unit 61 of the shared support server 6 for content data (sound data, This is the date and time when the image data, file data) was received. In addition, when the content processing type is "recording", "snapshot", and "material transmission", the start date and time and end date and time of the content processing are the date and time when the content data was transmitted by the communication unit 21 of the communication terminal 2. It may be. Furthermore, when the type of content processing is "recording", the start date and time and end date and time of recording by the image/sound processing unit 23 may be used. Furthermore, when the type of content processing is "snapshot", the start date and time and end date and time of content processing may be the date and time when a snapshot (capture) was performed by the image/sound processing unit 23.

Furthermore, the communication unit 61 transmits text data to the communication terminal 2, the display terminal 10, and the PC 5 with which the session is established (S80). Thereby, the communication terminal 2 and the display terminal 10 can display text data converted from sound data in real time.

Next, in FIG. 27, the storage/read processing unit 69 of the shared support server 6 searches the user authentication management DB 6001 (see FIG. 6(A)) using the user ID received in step S72 as a search key. The corresponding organization ID is read out (step S91).

Next, the storage/read processing unit 69 reads out the corresponding access ID and access password by searching the access management DB 6002 (see FIG. 6(B)) using the organization ID read in step S91 as a search key. (Step S92).

Next, the communication unit 61 transmits execution event history registration request information indicating a content data registration request to the schedule management server 8 (step S93). This execution event history registration request information includes the execution event ID received in step S72, the sender's user ID, the content data, the content processing ID generated in step S77, and the content data generated in step S78. The URL, the access ID and access password read in step S92, and the start date and time and end date and time of content processing are included. Thereby, the communication unit 81 of the schedule management server 8 receives the execution event history registration request information.

Next, in the schedule management server 8, the authentication unit 82 authenticates the shared support server 6 using the access ID and the access password (step S94). This authentication is the same process as step S38, so the explanation will be omitted. Here, we will continue to explain cases where it is legal.

Next, the storage/read processing unit 89 stores and manages the various data (information) received in step S93 in the execution event history management DB 8008 (see FIG. 12(B)) (step S95). The storage/read processing unit 89 stores various data (information) in the execution event history management DB 8008 in association with the execution event ID received in step S93. Thereby, the schedule management server 8 manages the same data as the shared support server 6 side.

Furthermore, the generation unit 83 of the schedule management server 8 generates related information that associates the content data received in step S93 for each content generation time slot (step S96). Here, the occurrence time period of the content included in the related information is generated using the start date and time of the event stored in the event management DB 8005 and the start date and time and end date and time of the content processing stored in the execution event history management DB 8008. Ru. That is, the content occurrence time zone indicates the elapsed time from the start date and time of the event at the time when the content occurred in the executed event. Then, the storage/reading processing unit 89 of the schedule management server 8 generates the data generated by the generation unit 83 in association with the execution event ID received in step S93 in the related information management DB 8010 (see FIG. 13(B)). The associated information is stored and managed (step S97). Thereby, the schedule management server 8 manages content data of different types of content processing in association with each content generation time.

Then, the storage/read processing unit 89 of the schedule management server 8 stores the text data received in step S93 in association with the execution event ID received in step S93 in the text information management DB 8012 (see FIG. 14). The included text information is stored and managed (step S98). Specifically, the generation unit 83 generates text information including the text data and content processing ID received in step S93, a text ID that identifies the text data received in step S93, and status information. Then, the storage/read processing unit 89 stores the text information generated by the generation unit 83 in the text information management DB 8012 in association with the execution event ID received in step S93.

As described above, the communication terminal 2 can transmit the execution event ID of the event being executed and the content generated by the corresponding event to the schedule management server 8. Furthermore, the schedule management server 8 can store the received content in the execution event history management DB 8008 for each execution event ID. Thereby, the communication system 1 can store content generated by an executed event for each event.

<Processing related to reaction buttons>
Next, pressing the reaction button, etc. will be explained with reference to FIGS. 28 to 30. Note that, among the users of the communication system 1, those who participate in an event such as a conference will be referred to as participants, and those who ask questions among the participants will be referred to as questioners.

FIG. 28 is an example of a conference screen 1100 displayed by the PC 5. It is assumed that the conference screen 1100 in FIG. 28 is displayed by a dedicated application running on the PC 5. The conference screen can also be displayed using a web browser. Further, the conference screen 1100 is also displayed on the communication terminal 2 and the display terminal 10. As shown in FIG. 28, text data is displayed almost in real time on the PC 5, communication terminal 2, and display terminal 10.

The conference screen 1100 has a bibliographic information field 1021 (an example of a second display area) and a text data field 1022 (an example of a first display area). The text data field 1022 includes text data 1001 to 1007 divided for each uttered sentence (for each square), a text being recognized 1008, and a chat input field 1009. The text data 1001 to 1007 in FIG. 28 scroll downward when new text data is added, and the latest text data converted from speech is displayed. Note that, as shown in the content management table of FIG. 7, the start time and end time of speech are recorded for each text data.

The text being recognized 1008 displays a portion of the text data uttered by the participant and currently being voice recognized. The text under recognition 1008 is a character string that has not yet been determined as an utterance.

When a participant clicks or taps text data or hovers over the text data, the display control unit 54 displays a list 1011 of reaction buttons. As will be described later, participants can register reaction buttons in the reaction type management table. Each reaction button is a like button, a super like button, a good button, a great button, a sad button, a terrible button, a question button 1013, etc., but the participants are not limited to these and can arbitrarily register them.

Conference participants can press reaction buttons as appropriate for text data, and can indicate their reactions with a simple button operation. One of the reaction buttons is a question button 1013, and when a participant who has a question about text data presses the question button 1013, the question button 1013 is displayed in the bibliographic information column 1021 of the conference screen 1100 together with the text data that was pressed. A message indicating that the question button 1013 has been pressed is displayed. Participants can ask questions verbally during the meeting without pressing the question button 1013, but participants who have difficulty asking questions verbally, such as those who are hard of hearing or deaf, should press the question button 1013. If so, you will be able to get an answer later.

In FIG. 28, it is assumed that the participant clicks or taps a question button 1013 in response to text data 1003 that says "I'm Mr. Tanuki."

Note that the reaction button list 1011 displays reaction buttons that are registered in the reaction type management table and that match the targets of the reaction type management table. In other words, a real-time button whose target in the reaction type management table is "Organization" can be displayed by a participant with the same organization ID in the reaction type management table, and a real-time button whose target is "event" in the reaction type management table can be displayed by a participant whose target is "Organization" in the reaction type management table. Participants participating in the same event as the event ID on the table can be displayed.

FIG. 29 is an example of a conference screen 1100 displayed by the PC 5 when a participant clicks or taps the question button 1013. In the bibliographic information column 1021, the fact that the question button 1013 was pressed and the text data 1016 for which the question button 1013 was pressed are displayed. An icon 1015 similar to the question button 1013 and a message 1014 saying "I don't understand" are displayed to indicate that the question button 1013 has been pressed.

Even if the text data 1003 is scrolled out from the text data field 1022, the fact that the question button 1013 in the bibliographic information field 1021 was pressed and the text data 1016 for which the question button 1013 was pressed will not be scrolled out, so the text data Any participant, such as the speaker 1016, can respond to the question taking into consideration the timing of the discussion. Note that the text data 1016 may be at least a portion of the text data 1003. Since the time 1018 when the question was uttered is also displayed in the text data 1016, the user can know how long the question has been left unanswered since the question button 1013 was pressed. This time 1018 may be the time when the question button 1013 was pressed.

Further, an icon 1017 similar to the question button 1013 is displayed in association with the text data 1003 in the text data column 1022. This icon 1017 also corresponds to the fact that the question button 1013 has been pressed. To indicate that the question button 1013 in the text data field 1022 has been pressed, the text data 1003 or the frame may be highlighted.

Furthermore, as additional text data is displayed in the text data column 1022, the text data 1003 in the text data column 1022 scrolls downward, but the text associated with the icon 1017 indicating that the question button 1013 has been pressed is A display format in which only the data 1003 is not scrolled out is also possible. Such control is performed by the display control unit 54 of the PC 5 determining the presence or absence of an icon 1017 indicating that the question button 1013 has been pressed for each of the text data 1001 to 1007 in the text data column 1022, and determining whether the question button 1013 has been pressed. This is achieved by fixing the text data 1003 associated with the icon 1017 to the bottom line.

The message indicating that the question button 1013 in the bibliographic information field 1021 and the text data field 1022 has been pressed (icon 1015, message 1014, icon 1017) can be deleted by the questioner performing a deletion operation. For example, the questioner displays a delete button by mousing over the mouse.

<Display that the reaction button is pressed and the question button is pressed>
FIG. 30 is a sequence diagram in which the PC 5 receives the press of the question button 1013, and the communication terminal 2 and the display terminal 10 share the fact that the question button 1013 has been pressed. Here, it is assumed that the questioner is operating the PC 5 (an example of the first terminal), and the users of the other PC 5, the communication terminal 2, and the display terminal 10 (an example of the second terminal) answer the question. .

S101: The questioner has already signed in to the shared support server 6 as in steps S32 and S33 of FIG. Therefore, the user ID, organization ID, and event ID of the questioner are specified. The PC 5 displays the conference screen 1100 when a participant signs in or explicitly requests to view the conference screen.

S102: In order to display the conference screen 1100, the communication unit 51 of the PC 5 specifies the organization ID and user ID and requests the shared support server 6 to set a reaction. Note that the organization ID and user ID can be replaced with an access token. The reaction setting is a request for a list of real-time buttons that can be displayed by this questioner.

S103: The communication unit 61 of the shared support server 6 receives the request for reaction settings, and the storage/reading processing unit 69 acquires the reaction button list 1011 from the reaction type management table. The processing unit 66 extracts reaction buttons whose target item in the reaction type management table is "organization" that match the organization ID to which the signed-in questioner belongs. The processing unit 66 extracts reaction buttons whose target item in the reaction type management table is "event" that matches the event ID in which the questioner is participating. The communication unit 61 of the shared support server 6 transmits a list 1011 of reaction buttons (reaction type of each reaction button, button image, etc.) to the PC 5.

S104: The communication unit 51 of the PC 5 receives a list 1011 of reaction buttons (reaction type of each reaction button, button image, etc.). Then, when the questioner performs an operation to display the reaction button list 1011 for arbitrary text data, the reception unit 52 accepts the operation, and the display control unit 54 displays the reaction button list 1011 using the button image.

S105: When the questioner presses the question button 1013 from the reaction button list 1011, the reception unit 52 accepts the question.

S106: The communication unit 51 of the PC 5 transmits the reaction type of the pressed reaction button, the content processing ID of the text data for which the reaction button was pressed, and the user ID to the shared support server 6.

S107: The communication unit 61 of the shared support server 6 receives the reaction type, content processing ID, and user ID, and the processing unit 66 determines whether the reaction type is a predetermined reaction. The predetermined reaction is a reaction that causes trouble for the person who presses the reaction button when scrolling out, and in this embodiment is, for example, the question button 1013 (“?” (question)).

If the reaction type is a predetermined reaction, the processing unit 66 of the shared support server 6, via the communication unit 61, displays the text data for which the question button 1013 was pressed in the bibliographic information field 1021 (an example of display instruction information). ), the reaction press ID (required at the time of deletion), the fact that the question button 1013 was pressed, and the content processing ID are transmitted to the PC 5. The communication unit 61 may transmit the text data itself instead of the content processing ID.

S108: The communication unit 51 of the PC 5 receives these, and the display control unit 54 displays that the question button 1013 has been pressed in association with the text data specified by the content processing ID. That is, the display control unit 54 displays an icon 1015 indicating that the question button 1013 was pressed in the bibliographic information field 1021, a message 1014, and text data 1016, and displays an icon 1015 indicating that the question button 1013 in the text data field 1022 was pressed. An icon 1017 indicating that the question button 1013 has been pressed is displayed in association.

In addition, in steps S106 to S108, the PC 5 notifies the shared support server 6 that the question button 1013 has been pressed, and sends text data 1016 from the shared support server 6 to the bibliographic information column 1021 and the question button 1013 has been pressed. A request to display the information has been received. However, after the question button 1013 is pressed and before communicating with the shared support server 6, the PC 5 may display text data 1016 indicating that the question button 1013 has been pressed in the bibliographic information field 1021. Further, the PC 5 may display an icon 1017 in the text data column 1022.

S109: The storage/reading processing unit 69 registers the reaction press ID (numbered to avoid duplication), content processing ID, user ID, and reaction type in the reaction press management table.

S110, S111: In order to share that the reaction button has been pressed, the processing unit 66 of the shared support server 6 displays the text data 1016 in the bibliographic information field 1021 via the communication unit 61, and also displays the reaction press ID. , the fact that the question button 1013 was pressed and the content processing ID are transmitted to the communication terminal 2 and display terminal 10 participating in the conference.

S112, S113: The communication unit 21 of the communication terminal 2 and the communication unit 11 of the display terminal 10 each receive the reaction press ID, the fact that the question button 1013 has been pressed, and the content processing ID. The display control unit 24 of the communication terminal 2 and the display control unit 13 of the display terminal 10 each display text data 1016 using the text data 1003 specified by the content processing ID, and also display information that the question button 1013 has been pressed. Display. The display control units 24 and 13 each display an icon 1017 around the text data 1003 specified by the content processing ID. These display methods may be the same as those of the PC5.

<Registration of reaction button>
Next, registration of reaction buttons will be explained with reference to FIGS. 31 and 32. FIG. 31 is a reaction button registration screen 1200 displayed by the PC 5. Note that the reaction button registration screen 1200 can also be displayed by the communication terminal 2 or the display terminal 10. When a participant presses the settings button 1012 displayed on the conference screen 1100, the reaction button registration screen 1200 shown in FIG. 31 is displayed.

The reaction button registration screen 1200 includes a button image list 1201, radio buttons 1202 and 1203 associated with "all tenants" and "only this meeting", a back button 1204, and an apply button 1205. The button image list 1201 is images of reaction buttons that are registered in advance in the shared support server 6 to be displayed on the reaction button registration screen 1200. Each reaction button corresponds to a reaction type. Participants can select the reaction button they wish to use using a mouse or the like. As shown in FIG. 31, the reaction button selected by the participant is surrounded by a frame 1206.

“Entire tenant” means that reaction buttons surrounded by a frame 1206 are displayed for all events held in the organization to which the participant belongs. "Only for this meeting" means that the reaction button surrounded by the frame 1206 is displayed only for the event currently being held (for example, a meeting). The settings of "Entire tenant" and "Only this meeting" are set in the target item of the reaction type management table. When the apply button 1205 is pressed, the reaction type selected in the frame 1206 of the reaction button registration screen 1200 and the targets of "all tenants" and "only this meeting" are sent to the shared support server 6.

FIG. 32 is a sequence diagram illustrating a process in which a participant registers a reaction button in the shared support server 6.

S121: A participant presses the settings button 1012 on the conference screen 1100. The reception unit 52 accepts the press, and the communication unit 51 requests the shared support server 6 for a reaction button registration screen 1200. The communication unit 61 of the shared support server 6 receives the request, and the creation unit 63 creates a reaction button registration screen 1200. The communication unit 61 transmits screen information of the reaction button registration screen 1200 to the PC 5. Note that the reaction button registration screen 1200 may be displayed by processing within the PC 5.

S122: The display control unit 54 of the PC 5 displays the reaction button registration screen 1200, and the participant makes settings regarding the reaction button as described with reference to FIG. The reception unit 52 accepts the settings.

S123: When the participant presses the apply button 1205, the communication unit 51 transmits the participant's organization ID, the participant's user ID, and the reaction type (organization or event) to the shared support server 6. Information regarding participants who have already signed in may be used as the organization ID and user ID.

S124: The communication unit 61 of the shared support server 6 receives the organization ID, user ID, and reaction type (organization or event), and the processing unit 66 registers it in the reaction type management table via the storage/reading processing unit 69. do.

In this way, participants can register reaction buttons that they want to use even during the meeting. Since it is possible to set "for the entire tenant" and "only for this meeting," there is no need for participants to register general reaction buttons for each meeting.

<Main effects>
As explained above, the shared support server 6 of the present embodiment displays the question button 1013 regarding the text data of the part of the statement that is the target of the question, and when the question button 1013 is pressed, the shared support server 6 displays the text data of the voice-recognized text data. displays the fact that the question button 1013 has been pressed and the text data 1016 that is the subject of the question in another area. Since the information that the question button 1013 has been pressed and the text data 1016 are not scrolled out, it is possible to reduce the likelihood that the answerer will overlook the question. Furthermore, it is possible to reduce situations in which a hearing-impaired person who has difficulty following the flow of a conversation hesitates even when asked a question, or a situation in which a hearing-impaired person is unable to express his or her own intentions.

<Other application examples>
Although the best mode for carrying out the present invention has been described above using examples, the present invention is not limited to these examples in any way, and various modifications can be made without departing from the gist of the present invention. and substitutions can be added.

In addition, the configuration example shown in FIG. 5 is designed to facilitate understanding of the processing by the communication terminal 2, PC 5, shared support server 6, schedule management server 8, speech-to-text conversion server 9, and display terminal 10. It is divided according to function. The present invention is not limited by the method of dividing the processing units or the names thereof. The processing of the communication terminal 2, PC 5, shared support server 6, schedule management server 8, speech-to-text conversion server 9, and display terminal 10 can be divided into more processing units depending on the processing content. Furthermore, one processing unit can be divided to include more processing.

Additionally, the devices described in the Examples are merely illustrative of one of multiple computing environments for implementing the embodiments disclosed herein. In some embodiments, shared support server 6 includes multiple computing devices, such as a server cluster. The plurality of computing devices are configured to communicate with each other via any type of communication link, including a network, shared memory, etc., to perform the processes disclosed herein.

Furthermore, the shared support server 6 can be configured to share the processing steps disclosed in this embodiment, such as in FIG. 28, in various combinations. For example, a process executed by a predetermined unit may be executed by a plurality of information processing devices included in the shared support server 6. Furthermore, the shared support server 6 may be integrated into one server device, or may be divided into a plurality of devices.

Each function of the embodiments described above can be realized by one or more processing circuits. Here, the term "processing circuit" as used herein refers to a processor programmed to execute each function by software, such as a processor implemented by an electronic circuit, or a processor designed to execute each function explained above. This includes devices such as ASICs (Application Specific Integrated Circuits), DSPs (Digital Signal Processors), FPGAs (Field Programmable Gate Arrays), and conventional circuit modules.

<Additional notes regarding multi-multi claims>
[Claim 1]
An information processing device that communicates with multiple terminals via a network,
a communication unit that transmits text data converted from sound data transmitted by the terminal to the first terminal and the second terminal together with display instruction information in the first display area;
a processing unit that associates operation information regarding the text data received from the terminal with the text data and stores it in a storage unit;
The communication unit transmits, to the plurality of terminals, display instruction information for displaying at least a portion of the text data on which the operation information has been manipulated in a second display area different from the first display area. An information processing device characterized by:
[Claim 2]
When the processing unit detects that the operation information stored in the storage unit is predetermined operation information,
The communication unit transmits to the plurality of terminals display instruction information for displaying at least a portion of the text data on which the operation of the predetermined operation information has been performed in a second display area different from the first display area. The information processing apparatus according to claim 1, wherein the information processing apparatus transmits the information to the information processing apparatus.
[Claim 3]
The communication unit transmits to the plurality of terminals, in addition to display instruction information for at least a portion of the text data, display instruction information in the second display area indicating that the predetermined operation information has been operated. The information processing device according to claim 2, characterized in that:
[Claim 4]
The plurality of terminals are participating in an event to share text data converted from sound data,
a creation unit that creates a screen that accepts settings for operations that the terminal can accept with respect to the text data;
The communication unit transmits screen information of the screen created by the creation unit to a terminal,
4. The processing unit stores, in a storage unit, operation information transmitted from the terminal participating in the event and indicating operations that can be accepted by the terminal. The information processing device described in .
[Claim 5]
On the screen, it is possible to select either the organization to which the user belongs or the event in which the user is participating as a target for which operations that can be accepted by the terminal are provided;
The processing unit associates operation information of an operation that the terminal can accept, the target, and identification information of the target organization or identification information of the event transmitted from the terminal and stores it in the storage unit. The information processing apparatus according to claim 4, wherein the information processing apparatus stores the information.
[Claim 6]
The communication department sends information to terminals participating in the event,
When the target of the operation information stored in the storage unit is an organization, the operation information is associated with identification information of the same organization as the identification information of the organization to which the user of the terminal belongs, and
When the target of the operation information stored in the storage unit is the event, the operation is associated with identification information of the same event as the identification information of the event in which the user of the terminal is participating. The information processing device according to claim 5, wherein the information processing device transmits information.
[Claim 7]
The first display area scrolls out oldest text data converted from sound data, and the second display area prevents at least a part of the text data from scrolling out. 7. The information processing device according to any one of items 1 to 6.
[Claim 8]
3. The information processing apparatus according to claim 2, wherein the predetermined operation information is operation information indicating that a question regarding the text data has been received.
[Claim 9]
4. The second display area displays at least a part of the text data on which the operation information has been operated, as well as information indicating that a question has been asked regarding the text data. The information processing device described.

1 Communication system 2 Communication terminal 3 Video conference terminal 5 PC
6 Shared support server 8 Schedule management server 9 Speech-to-text conversion server

Patent No. 6746923

Claims (20)

An information processing device that communicates with multiple terminals via a network,
a communication unit that transmits text data converted from sound data transmitted by the terminal to the first terminal and the second terminal together with display instruction information in the first display area;
a processing unit that associates operation information regarding the text data received from the terminal with the text data and stores it in a storage unit;
The communication unit transmits, to the plurality of terminals, display instruction information for displaying at least a portion of the text data on which the operation information has been manipulated in a second display area different from the first display area. An information processing device characterized by: When the processing unit detects that the operation information stored in the storage unit is predetermined operation information,
The communication unit transmits to the plurality of terminals display instruction information for displaying at least a portion of the text data on which the operation of the predetermined operation information has been performed in a second display area different from the first display area. The information processing apparatus according to claim 1, wherein the information processing apparatus transmits the information to the information processing apparatus. The communication unit transmits to the plurality of terminals, in addition to display instruction information for at least a portion of the text data, display instruction information in the second display area indicating that the predetermined operation information has been operated. The information processing device according to claim 2, characterized in that: The plurality of terminals are participating in an event to share text data converted from sound data,
a creation unit that creates a screen that accepts settings for operations that the terminal can accept with respect to the text data;
The communication unit transmits screen information of the screen created by the creation unit to a terminal,
4. The processing unit stores, in a storage unit, operation information transmitted from the terminal participating in the event and indicating operations that can be accepted by the terminal. The information processing device described in . On the screen, it is possible to select either the organization to which the user belongs or the event in which the user is participating as a target for which operations that can be accepted by the terminal are provided;
The processing unit associates operation information of an operation that the terminal can accept, the target, and identification information of the target organization or identification information of the event transmitted from the terminal and stores it in the storage unit. The information processing apparatus according to claim 4, wherein the information processing apparatus stores the information. The communication department sends information to terminals participating in the event,
When the target of the operation information stored in the storage unit is an organization, the operation information is associated with identification information of the same organization as the identification information of the organization to which the user of the terminal belongs, and
When the target of the operation information stored in the storage unit is the event, the operation is associated with identification information of the same event as the identification information of the event in which the user of the terminal is participating. The information processing device according to claim 5, wherein the information processing device transmits information. The first display area scrolls out oldest text data converted from sound data, and the second display area prevents at least a part of the text data from scrolling out. 1. The information processing device according to 1. 3. The information processing apparatus according to claim 2, wherein the predetermined operation information is operation information indicating that a question regarding the text data has been received. 4. The second display area displays at least a part of the text data on which the operation information has been operated, as well as information indicating that a question has been asked regarding the text data. The information processing device described. A display method performed by an information processing device that communicates with multiple terminals via a network, the display method comprising:
a step in which the communication unit transmits the text data converted from the sound data transmitted by the terminal to the first terminal and the second terminal together with display instruction information in the first display area;
a step in which the processing unit associates the operation information regarding the text data received from the terminal with the text data and stores it in the storage unit;
The communication unit transmits to the plurality of terminals display instruction information for displaying at least a portion of the text data on which the operation information has been manipulated in a second display area different from the first display area. A display method comprising the steps of: When the processing unit detects that the operation information stored in the storage unit is predetermined operation information,
The communication unit transmits to the plurality of terminals display instruction information for displaying at least a portion of the text data on which the operation of the predetermined operation information has been performed in a second display area different from the first display area. 11. The display method according to claim 10, wherein the display method is transmitted to. The communication unit transmits to the plurality of terminals, in addition to display instruction information for at least a portion of the text data, display instruction information in the second display area indicating that the predetermined operation information has been operated. The display method according to claim 11, characterized in that: The plurality of terminals are participating in an event to share text data converted from sound data,
a step in which the creation unit creates a screen that accepts settings for operations that the terminal can accept with respect to the text data;
a step in which the communication unit transmits screen information of the screen created by the creation unit to a terminal;
Claims 10 to 12, characterized in that the processing unit has a step of storing, in a storage unit, operation information transmitted from the terminal participating in the event and indicating operations that can be accepted by the terminal. The display method described in any one of the following. On the screen, it is possible to select either the organization to which the user belongs or the event in which the user is participating as a target for which operations that can be accepted by the terminal are provided;
The processing unit associates operation information of an operation that the terminal can accept, the target, and identification information of the target organization or identification information of the event transmitted from the terminal and stores it in the storage unit. 14. The display method according to claim 13, further comprising storing the information. The communication department sends information to terminals participating in the event,
When the target of the operation information stored in the storage unit is an organization, the operation information is associated with identification information of the same organization as the identification information of the organization to which the user of the terminal belongs, and
When the target of the operation information stored in the storage unit is the event, the operation is associated with identification information of the same event as the identification information of the event in which the user of the terminal is participating. 15. The display method according to claim 14, further comprising transmitting information. The first display area scrolls out oldest text data converted from sound data, and the second display area prevents at least a part of the text data from scrolling out. 10. The display method described in 10. 12. The display method according to claim 11, wherein the predetermined operation information is operation information indicating that a question regarding the text data has been received. 13. The second display area displays at least a part of the text data on which the operation information has been operated, as well as information indicating that a question has been asked regarding the text data. How to display the description. A communication system in which a plurality of terminals and an information processing device communicate via a network,
The information processing device includes a communication unit that transmits to the first terminal and the second terminal together with display instruction information on the first display area of the text data converted from the sound data transmitted by the terminal;
a processing unit that associates operation information regarding the text data received from the terminal with the text data and stores it in a storage unit;
The communication unit transmits, to the plurality of terminals, display instruction information for displaying at least a portion of the text data on which the operation information has been manipulated in a second display area different from the first display area. death,
The plurality of terminals are
Text data received from the information processing device is displayed in the first display area, and at least a part of the text data on which the operation information has been manipulated is displayed in the second display area. communication system. An information processing device that communicates with multiple terminals via a network,
a communication unit that transmits text data converted from sound data transmitted by the terminal to the first terminal and the second terminal together with display instruction information in the first display area;
functioning as a processing unit that associates operation information regarding text data received from a terminal with the text data and stores it in a storage unit;
The communication unit transmits, to the plurality of terminals, display instruction information for displaying at least a portion of the text data on which the operation information has been manipulated in a second display area different from the first display area. A program that is characterized by:

Images