JP2024008632A - Information processing system, display method, program, and recording information creation system - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide importance information regarding recorded information recorded at meetings, etc.

SOLUTION: An information processing system according to the present invention that provides information recorded by a device capable of recording audio and video to a terminal device includes an importance determining portion that determines importance information regarding recorded information for each time shorter than a recording time of the recorded information on the basis of at least one of recorded information including the voice or handwritten data associated with the recorded information, and a display information providing portion that provides the terminal device with information for displaying the importance information according to a recording time of the recorded information.

SELECTED DRAWING: Figure 1

COPYRIGHT: (C)2024,JPO&INPIT

Description

The present invention relates to an information processing system, a display method, a program, and a recorded information creation system.

2. Description of the Related Art As a technology for supporting the creation of minutes of a meeting, there is a known technology for recording the statements of participants during the meeting and the video recording of the meeting. Participants can playback comments and videos by operating playback software.

2. Description of the Related Art A technique is known in which a message and a video are linked and displayed (see, for example, Patent Document 1). Patent Document 1 discloses that the input order or input time of each stroke is associated with each audio information representing each audio recorded by the audio recording means during the input period of each stroke, and the stroke and audio information are A technique has been disclosed for displaying strokes and emitting each sound based on audio information while taking correspondence.

However, the conventional technology has a problem in that it is difficult to determine which parts of recorded video and audio are important, for example. In other words, when a viewer views time-series information that spans a long period of time, such as a meeting, if there is no suggestion as to which period of information to view, the viewer will have to view the entire page, which is inefficient. .

In view of the above-mentioned problems, the present invention aims to provide importance information regarding recorded information recorded at a meeting or the like.

In view of the above problems, the present invention provides an information processing system that provides information recorded by a device capable of recording audio and video to a terminal device, the system being compatible with at least the recorded information including the audio or the recorded information. an importance determining unit that determines importance information regarding the recorded information at intervals shorter than a recording time of the recorded information based on at least one of the attached handwritten data; and a display information providing unit that provides the terminal device with information to be displayed along the recording time.

Importance information can be provided for recorded information recorded at meetings and the like.

It is a figure which shows the activity level expressed by the heat map. FIG. 2 is a diagram illustrating the outline of creating recording information that saves the screen of an application executed during a remote conference together with a surrounding panoramic image. It is a diagram showing an example of the configuration of a recorded information creation system. 1 is a diagram illustrating an example of a hardware configuration of an information processing system and a terminal device. FIG. 2 is a diagram illustrating an example of a hardware configuration of a meeting device. FIG. 3 is a diagram illustrating an imaging range of a meeting device. FIG. 3 is a diagram illustrating cutting out of a panoramic image and a speaker image. FIG. 2 is a hardware configuration diagram of an example of an electronic whiteboard. 1 is an example of a functional block diagram illustrating functions of a terminal device, a meeting device, and an information processing system in the recorded information creation system divided into blocks. It is a figure which shows an example of the moving image recording information which the information storage part memorize|stores. FIG. 3 is a diagram illustrating an example of conference information managed by a communication management section. FIG. 3 is a diagram showing an example of association information in which a conference ID and device identification information are associated with each other, which is stored in an association information storage unit. FIG. 3 is a diagram showing an example of recorded information and object information stored in a recorded information storage unit. FIG. 2 is a diagram illustrating the structure of text data that is part of recorded information. FIG. 7 is a diagram illustrating an example of speaker switching data that is saved together with a combined image video. 1 is an example of a functional block diagram illustrating functions of an electronic whiteboard divided into blocks. FIG. 3 is a diagram illustrating an example of device identification information etc. stored in a device information storage unit. FIG. 3 is a diagram illustrating object information stored in an object information storage unit. FIG. 2 is a diagram illustrating the structure of stroke data that constitutes handwritten data. FIG. 2 is an example of a sequence diagram showing a procedure for an information recording application to record a panoramic image, a speaker image, and an application screen; FIG. It is an example of a sequence diagram showing a procedure in which a viewer operates an information recording application to display a combined image video. FIG. 3 is a diagram showing an example of a combined image moving image display screen displayed by the information recording application. FIG. 7 is an example of a flowchart illustrating a procedure in which the importance determining unit determines the degree of activity; FIG. It is a figure which shows an example of the heat map which expresses the activity degree by brightness|luminance. It is a figure which shows an example of the heat map which expresses the degree of activity by color gradation. It is a figure showing an example of a plurality of heat maps each expressed by a plurality of degrees of activity. FIG. 3 is a diagram showing an example of a heat map in which the frequency of appearance of keywords is used as the degree of activity. FIG. 3 is a diagram showing an example of a heat map displayed for each activity determining factor. FIG. 3 is a diagram showing an example of an enlarged heat map. It is a figure which shows an example of the heat map which uses the activity level of a participant as an activity level. FIG. 3 is a diagram illustrating a method of displaying activity levels that displays time periods with high activity levels without using colors.

Hereinafter, as an example of a mode for implementing the present invention, a recorded information creation system and a display method performed by the recorded information creation system will be described.

<Outline of display of recorded information>
The recorded information creation system of this embodiment calculates the level of activity of a meeting for each time period from video (combined image video to be described later), audio, and object information (information input to the electronic blackboard) recorded during the meeting. calculate. As for the activity level,
- Frequency of written strokes - Number of utterances - Number of speakers - Frequency of switching speakers - Frequency of keyword appearance (spoken or handwritten)
It is possible to use activity determining factors such as: By displaying the degree of activity in chronological order, the record information creation system makes it easier for viewers to understand which part of the meeting record information they should view.

FIG. 1 is a diagram showing the degree of activity represented by a heat map. A heat map 246 is displayed along the time indicator 241d at the bottom of the playback screen. The heat map 246 in FIG. 1 shows the degree of activity in which the degrees of activity based on the five activity determining factors described above are integrated. The time indicator 241d is one of the functions provided in music or video playback software, and is a bar-shaped display component that displays the playback location of music or video. The heat map 246 visually expresses the level of activity using color density and brightness, making it possible to visualize at what time the meeting was most exciting.

By checking the heat map 246, the viewer can more quickly find the part of the recorded information that he or she should or wants to view.

<About terms>
A heat map is a type of visualization graph that expresses a series of numerical values as colors and shades.

A meeting means to meet and discuss matters. A conference may also be called a convention, gathering, gathering, meeting, council, recital, presentation, conference, collusion, meeting, discussion, discussion, seminar, etc.

Importance information is information that indicates which time of recorded information should be viewed. Whether or not the content should be viewed may vary depending on the viewer and the viewer's situation, and the importance information only needs to be that there is a possibility that the content is important. For example, what is important may mean that a discussion is active, that an important conversation is held, or that a particular participant makes a statement. In this embodiment, the importance level information will be explained using the term "activity level".

The recorded information includes, for example, audio recorded during the meeting. The recorded information may include video. The audio may or may not be converted to text data.

The recording time is the time (total time) during which the recording information was recorded. A time indicator 241d indicates this recording time. Therefore, the heat map 246 is displayed along the recording time.

A series of operations in which the user presses an input device against the display, continuously moves it, and then releases it from the display is called a stroke. Stroking involves tracking the user's movements without touching the display. In this case, the electronic whiteboard may initiate a stroke by a user's gesture, by pressing a button with the user's hand or foot, or in other ways, for example using a mouse or pointing device. Additionally, the user may terminate the stroke using the same or a different gesture, releasing a button, or using a mouse or pointing device.

Stroke data is information displayed on a display based on a trajectory of coordinates input by an input means. Stroke data may be interpolated as appropriate. Handwritten data is data that includes one or more stroke data. Handwritten input indicates that handwritten data is input by the user. Handwriting input may be performed using a touch interface, a tactile object such as a pen or stylus, or the user's body. Handwriting input may also be performed via other types of input, such as gesture-based input, hand movement tracking input, or other touch-free input by the user. Although embodiments of the invention refer to handwritten input and handwritten input data, other forms of handwritten input may be utilized.

An object displayed on a display based on stroke data is called an object. The term "object" means a target, and in this embodiment, it means a display target. Objects converted by handwriting recognition of stroke data may include, in addition to text, fixed characters such as "done", stamps displayed as marks, figures such as circles and stars, straight lines, and the like. Text is a character string (character code) that mainly includes one or more characters, and also includes numbers, symbols, etc. Text is sometimes called a string.

<An example of how to create minutes for a remote meeting>
With reference to FIG. 2, an outline of a method for creating minutes using a panoramic image and an application screen will be described. FIG. 2 is a diagram illustrating the outline of creation of recording information that saves the screen of an application executed during a remote conference together with a surrounding panoramic image. As shown in FIG. 2, a user at the illustrated base 102 is holding a remote conference with another base 101 using the remote conference service system 90.

The recorded information creation system 100 of this embodiment is created by a horizontal panoramic image (hereinafter referred to as a panoramic image) captured by a meeting device 60 equipped with a microphone and a speaker, and an application executed by a terminal device 10 (hereinafter referred to as an application). Create recorded information (minutes) using the screen and . Regarding audio, the recorded information creation system 100 synthesizes the audio received by the remote conference application 42 and the audio acquired by the meeting device 60, and includes the synthesized audio in the recorded information. The outline will be explained below.

(1) In the terminal device 10, an information recording application 41 and a remote conference application 42, which will be described later, are operating. In addition, an application for displaying materials may also be running. The information recording application 41 transmits the audio output by the terminal device 10 (including the audio received by the remote conference application 42 from another base) to the meeting device 60 (a second device, an example of a device). The meeting device 60 mixes (synthesizes) the audio that it has acquired and the audio of the remote conference application 42.

(2) The meeting device 60 is equipped with a microphone, and creates a speaker image by cutting out the speaker from the panoramic image based on the direction in which the voice was acquired. The meeting device 60 transmits both the panoramic image and the speaker image to the terminal device 10.

(3) The information recording application 41 running on the terminal device 10 can display a panoramic image 203 and a speaker image 204. The information recording application 41 combines any application screen selected by the user (for example, the screen 103 of the remote conference application 42), the panoramic image 203, and the speaker image 204. For example, the panoramic image 203, the speaker image 204, and the app screen 103 are combined so that the panoramic image 203 and the speaker image 204 are placed on the left side, and the screen 103 of the remote conference app 42 is placed on the right side (hereinafter, the combined image 105). Since the process (3) is repeatedly executed, the combined image 105 becomes a moving image (hereinafter referred to as a combined image moving image). Furthermore, the information recording application 41 combines the synthesized audio with the combined image video to create a video with audio.

Note that in this embodiment, an example will be described in which the panoramic image 203, the speaker image 204, and the application screen 103 are combined, but the information recording application 41 may store these separately and arrange them on the screen during playback. .

(4) The information recording application 41 accepts editing work (cutting of unnecessary parts by the user) and completes the combined image video. The combined image video constitutes part of the recorded information.

(5) The information recording application 41 transmits the created combined image video (with audio) to the storage service system 70 and stores it.

(6) Further, the information recording application 41 extracts only the audio from the combined image video (the audio before combining may be saved), and transmits the extracted audio to the information processing system 50. The information processing system 50 transmits the speech to a speech recognition service system 80 that converts the speech into text data, and converts the speech into text. The text data also includes data on how many minutes after the start of recording, the person spoke.

Note that in the case of real-time text conversion, the meeting device 60 directly transmits audio to the information processing system 50. The information processing system 50 transmits text data obtained through voice recognition to the information recording application 41 in real time.

(7) The information processing system 50 additionally stores text data in the storage service system 70 that stores the combined image video. Text data forms part of the recorded information.

Note that the information processing system 50 can perform billing processing on the user according to the service used. For example, charges are calculated based on the amount of text data, the file size of the combined image video, processing time, and the like.

In this way, the combined image video displays a panoramic image of the surroundings including the user and an image of the speaker, and further displays the screen of an application displayed during the remote conference, such as the remote conference application 42. When a participant or a non-participant in a remote conference views the combined image video as minutes, the situation during the remote conference is reproduced with a sense of realism.

<System configuration example>
Next, with reference to FIG. 3, the system configuration of the recorded information creation system 100 will be described. FIG. 3 shows an example of the configuration of the recorded information creation system 100. FIG. 3 shows one base (self-base 102) among a plurality of bases where a remote conference is held, and the terminal device 10 at the own base 102 connects to the information processing system 50, the storage service system 70, and the remote conference via the network. Communicate with service system 90. Further, a meeting device 60 and an electronic blackboard 2 are arranged at the own base 102, and the terminal device 10 is communicably connected to the meeting device 60 via a USB cable or the like. Note that the meeting device 60, electronic whiteboard 2, and information processing system 50 operate as a device management system.

In the terminal device 10, at least an information recording application 41 and a remote conference application 42 operate. The remote conference application 42 can communicate with the terminal devices 10 of other bases 101 via the remote conference service system 90 on the network, allowing users at each base to have a conference with each other from a remote location. The information recording application 41 uses the functions of the information processing system 50 and the meeting device 60 to create recorded information in the remote conference conducted by the remote conference application 42 .

Note that although this embodiment describes an example of creating recorded information during a remote conference, the conference does not have to be a conference in which communication is performed with a remote base. In other words, the conference may be a conference in which only participants within one location participate. In this case, the image captured by the meeting device 60 and the collected audio are each saved without being combined, and there is no change in the processing of the information recording application 41.

The terminal device 10 has a built-in camera with a normal angle of view (it may be attached externally), and captures an image of the front including the user 107 operating the terminal device 10. A normal angle of view is an image that is not a panoramic image, but in this embodiment, it is mainly a planar image that is not a curved surface, such as a spherical image. Therefore, the user can have a conventional remote conference using the remote conference application 42 without being aware of the information recording application 41. The information recording application 41 and the meeting device 60 do not affect the remote conference application 42 except for an increase in the processing load on the terminal device 10. Note that the remote conference application 42 can also transmit a panoramic image and speaker images captured by the meeting device 60 to the remote conference service system 90.

The information recording application 41 communicates with the meeting device 60 to create recorded information. The information recording application 41 also performs synthesis of the audio acquired by the meeting device 60 and the audio received from another base by the remote conference application 42. This synthesis may be performed by the meeting device 60. The meeting device 60 is a meeting device that includes a panoramic image capturing device, a microphone, and a speaker. Although the camera included in the terminal device 10 can only image a limited area in front, the meeting device 60 can image the entire surrounding area (not necessarily the entire area) surrounding the meeting device 60. The meeting device 60 can always keep the plurality of participants 106 shown in FIG. 3 within the viewing angle.

In addition, the meeting device 60 performs operations such as cutting out a speaker image from a panoramic image. Note that the meeting device 60 may be placed anywhere in the home base 102, not only on the desk. Since the meeting device 60 can capture a spherical image, it may be placed on the ceiling, for example.

The information recording application 41 displays a list of applications being executed on the terminal device 10, synthesizes images for the above recorded information (creates a combined image video), plays the combined image video, accepts editing, and the like. The information recording application 41 also displays a list of remote conferences that have been held or are scheduled to be held in the future. The list of teleconferences is used for information about recorded information and allows users to associate teleconferences with recorded information.

The remote conference application 42 performs communication connections with other bases 101, transmission and reception of images and audio with other bases 101, display of images, output of audio, and the like.

Note that the information recording application 41 and the remote conference application 42 may be web applications or native applications. A web application is an application in which a program on a web server and a program on a web browser cooperate to perform processing, and does not require installation on the terminal device 10. A native application is an application installed and used on the terminal device 10. In this embodiment, the explanation will be given assuming that both are native applications.

The terminal device 10 may be a general-purpose information processing device with a communication function, such as a PC (Personal Computer), a smartphone, or a tablet terminal. In addition, the terminal device 10 may be an electronic whiteboard 2, a game machine, a PDA (Personal Digital Assistant), a wearable PC, a car navigation system, an industrial machine, a medical device, a network home appliance, or the like. The terminal device 10 may be any device on which the information recording application 41 and the remote conference application 42 operate.

The electronic blackboard 2 displays data handwritten on a touch panel using an input means such as a pen or a finger on a display. The electronic whiteboard 2 can communicate with the terminal device 10 or the like by wire or wirelessly, and can capture the screen displayed by the terminal device 10 and display it on the display. The electronic blackboard 2 can convert handwritten data into text data, and can share information displayed on a display with electronic blackboards 2 at other locations. The electronic whiteboard 2 may be of a form in which a projector projects an image onto a simple white board without a touch panel. Furthermore, the electronic whiteboard 2 may be a tablet terminal equipped with a touch panel, a notebook PC, a PDA, a game console, or the like.

The electronic whiteboard 2 can communicate with the information processing system 50. The electronic blackboard 2 can receive information from the information processing system 50 by, for example, polling the information processing system 50 after being powered on.

The information processing system 50 is one or more information processing devices arranged on a network. The information processing system 50 includes one or more server applications that perform processing in cooperation with the information recording application 41, and infrastructure services. This server app manages the list of remote conferences, the recorded information recorded in remote conferences, and manages various settings and storage paths. Basic services perform user authentication, contracts, billing processing, etc.

Note that all or part of the functions of the information processing system 50 may exist in a cloud environment or may exist in an on-premises environment. The information processing system 50 may be configured with a plurality of server devices, or may be configured with one information processing device. For example, the server application and the infrastructure service may be provided by separate information processing devices, or there may be an information processing device for each function within the server application. The information processing system 50, a storage service system 70, and a voice recognition service system 80, which will be described below, may be integrated.

The storage service system 70 is a storage device on a network, and provides a storage service that accepts storage of files and the like. As the storage service system 70, One Drive (registered trademark), Google Workspace (registered trademark), DropBox (registered trademark), etc. are known. The storage service system 70 may be an on-premises NAS (Network Attached Storage) or the like.

The speech recognition service system 80 provides a service that performs speech recognition on speech data and converts it into text data. The speech recognition service system 80 may be a general-purpose commercial service or may be part of the functions of the information processing system 50.

<Hardware configuration example>
With reference to FIG. 4, the hardware configurations of the information processing system 50 and the terminal device 10 according to this embodiment will be described.

<<Information processing system and terminal device>>
FIG. 4 is a diagram showing an example of the hardware configuration of the information processing system 50 and the terminal device 10 according to the present embodiment. As shown in FIG. 4, the information processing system 50 and the terminal device 10 are constructed by a computer, including a CPU 501, ROM 502, RAM 503, HD (Hard Disk) 504, HDD (Hard Disk Drive) controller 505, and display 506. , an external device connection I/F (Interface) 508, a network I/F 509, a bus line 510, a keyboard 511, a pointing device 512, an optical drive 514, and a media I/F 516.

Among these, the CPU 501 controls the entire operation of the information processing system 50 and the terminal device 10 . The ROM 502 stores programs used to drive the CPU 501 such as IPL. RAM 503 is used as a work area for CPU 501. The HD 504 stores various data such as programs. The HDD controller 505 controls reading and writing of various data to the HD 504 under the control of the CPU 501. The display 506 displays various information such as a cursor, menu, window, characters, or images. External device connection I/F 508 is an interface for connecting various external devices. The external device in this case is, for example, a USB (Universal Serial Bus) memory, a printer, or the like. The network I/F 509 is an interface for data communication using a network. The bus line 510 is an address bus, a data bus, etc. for electrically connecting each component such as the CPU 501 shown in FIG. 4.

Further, the keyboard 511 is a type of input means that includes a plurality of keys used for inputting characters, numbers, various instructions, and the like. The pointing device 512 is a type of input means for selecting and executing various instructions, selecting a processing target, moving a cursor, and the like. The optical drive 514 controls reading and writing of various data to and from an optical storage medium 513, which is an example of a removable recording medium. Note that the optical storage medium 513 may be a CD, DVD, Blu-ray (registered trademark), or the like. The media I/F 516 controls reading or writing (storage) of data to a recording medium 515 such as a flash memory.

＜＜Meeting device＞＞
The hardware configuration of the meeting device 60 will be explained using FIG. 5. FIG. 5 is an example of a hardware configuration diagram of a meeting device 60 capable of capturing 360° moving images. In the following, the meeting device 60 is assumed to be a device that uses an image sensor to capture a 360° moving image around the device at a predetermined height, but the number of image sensors may be one, two or more. Further, it does not necessarily have to be a dedicated device, and a 360° video imaging unit may be attached to a PC, digital camera, smartphone, etc. afterward to have substantially the same functions.

As shown in FIG. 5, the meeting device 60 includes an imaging unit 601, an image processing unit 604, an imaging control unit 605, a microphone 608, a sound processing unit 609, a CPU (Central Processing Unit) 611, and a ROM (Read Only Memory). ) 612, SRAM (Static Random Access Memory) 613, DRAM (Dynamic Random Access Memory) 614, operation unit 615, external device connection I/F 616, communication unit 617, antenna 617a, audio sensor 618, and concave shape for Micro USB. It is configured by a terminal 621.

Among these, the imaging unit 601 includes a wide-angle lens (so-called fisheye lens) 602 having a 360° angle of view for forming a hemispherical image, and an imaging element 603 (image sensor) provided corresponding to each wide-angle lens. It is equipped with The image sensor 603 is an image sensor such as a CMOS (Complementary Metal Oxide Semiconductor) sensor or a CCD (Charge Coupled Device) sensor that converts an optical image formed by the fisheye lens 602 into electrical signal image data and outputs the image data. It includes a timing generation circuit that generates synchronization signals and pixel clocks, and a group of registers in which various commands and parameters necessary for the operation of this image sensor are set.

An image sensor 603 (image sensor) of the imaging unit 601 is connected to an image processing unit 604 via a parallel I/F bus. On the other hand, the imaging element 603 of the imaging unit 601 is connected to the imaging control unit 605 via a serial I/F bus (such as an I2C bus). The image processing unit 604, the imaging control unit 605, and the sound processing unit 609 are connected to the CPU 611 via a bus 610. Furthermore, a ROM 612, an SRAM 613, a DRAM 614, an operation section 615, an external device connection I/F 616, a communication section 617, an audio sensor 618, and the like are also connected to the bus 610.

The image processing unit 604 takes in the image data output from the image sensor 603 through the parallel I/F bus, performs predetermined processing on the image data, and creates panoramic image and speaker image data from the fisheye video. do. Further, the image processing unit 604 synthesizes the panoramic image, the speaker image, etc., and outputs one moving image.

Generally, the imaging control unit 605 sets commands and the like in a register group of the imaging device 603 using the I2C bus, with the imaging control unit 605 as a master device and the imaging device 603 as a slave device. Necessary commands and the like are received from the CPU 611. The imaging control unit 605 also uses the I2C bus to take in status data and the like of the register group of the imaging device 603 and sends it to the CPU 611.

Further, the imaging control unit 605 instructs the imaging device 603 to output image data at the timing when the imaging start button on the operation unit 615 is pressed or at the timing when an imaging start instruction is received from the PC. Depending on the meeting device 60, it may have a preview display function or a function that supports video display on a display (for example, a display of a PC or a smartphone). In this case, image data is continuously output from the image sensor 603 at a predetermined frame rate (frames/minute).

The imaging control unit 605 also functions as a synchronization control unit that synchronizes the output timing of image data from the imaging device 603 in cooperation with the CPU 611, as will be described later. Note that in this embodiment, the meeting device 60 is not provided with a display, but may be provided with a display section.

Microphone 608 converts sound into sound (signal) data. The sound processing unit 609 takes in audio data output from the microphones 608a, 608b, and 608c through the I/F bus, mixes these audio data, and performs predetermined processing. The sound processing unit 609 also determines the direction of the sound source (speaker) from the sound level (volume) input from the microphones 608a, 608b, and 608c.

The CPU 611 controls the overall operation of the meeting device 60 and executes necessary processing. ROM 612 stores various programs for operating meeting device 60. The SRAM 613 and DRAM 614 are work memories that store programs executed by the CPU 611, data being processed, and the like. In particular, the DRAM 614 stores image data that is currently being processed by the image processing unit 604 and data of processed equirectangular projection images.

The operation unit 615 is a general term for operation buttons such as the imaging start button 615a. By operating the operation unit 615, the user not only starts imaging and recording, but also turns the power ON/OFF, establishes a communication connection, and inputs settings such as various imaging modes and imaging conditions.

The external device connection I/F 616 is an interface for connecting various external devices. The external device in this case is, for example, a PC. The video data and image data stored in the DRAM 614 are transmitted to an external terminal via this external device connection I/F 616 or recorded on external media.

The communication unit 617 communicates with a cloud server via the Internet using wireless communication technology such as Wi-Fi (registered trademark) via an antenna 617a provided in the meeting device 60, and transfers the stored video data and image data to the cloud. It may also be sent to the server. Further, the communication unit 617 may be able to communicate with nearby devices using short-range wireless communication technology such as BLE (Bluetooth Low Energy.registered trademark) and NFC.

The audio sensor 618 is a sensor that acquires 360° audio information in order to identify from which direction a loud audio is input in 360° around the meeting device 60 (horizontal plane). The sound processing unit 609 identifies the strongest direction based on the input 360° audio parameters and outputs the 360° audio input direction.

Note that other sensors (azimuth/acceleration sensor, GPS, etc.) may calculate the azimuth, position, angle, acceleration, etc., and may be used for image correction or addition of location information.

Further, the image processing unit 604 creates a panoramic image using the following method. The CPU 611 performs predetermined camera image processing such as Bayer conversion (RGB complementation processing) on RAW data input from an image sensor that inputs spherical images to create a fisheye image (a curved image). Furthermore, DeWarp processing (distortion correction processing) is performed on the created fisheye video (curved surface video) to create a panoramic image (flat video) showing 360° of the surroundings of the meeting device 60.

The CPU 611 creates a speaker image using the following method. The CPU 611 creates a speaker image by cutting out the speaker from a panoramic image (flat image) showing 360° of the surrounding area. The CPU 611 cuts out a speaker image from the panoramic image, using the audio input direction specified from the 360° output using the audio sensor 618 and the sound processing unit 609 as the speaker's direction. At this time, the method of cutting out a human image from the audio input direction is to cut out a 30° area centered on the specified audio direction from 360°, perform face detection within that area, and then cut out the image. Among the cut out speaker images, the CPU 611 further identifies speaker images for a specific number of people (such as 3 speakers) who have recently spoken.

The panoramic image and one or more speaker images may be sent individually to the information recording application 41, or the meeting device 60 may create one image from these and send it to the information recording application 41. . In this embodiment, it is assumed that the panoramic image and one or more speaker images are individually transmitted from the meeting device 60 to the information recording application 41.

FIG. 6 is a diagram illustrating the imaging range of the meeting device 60. As shown in FIG. 6(a), the meeting device 60 captures an image in a 360° range in the horizontal direction. As shown in FIG. 6(b), the meeting device 60 has an imaging range of 0° in a direction horizontal to the height of the meeting device 60, and a predetermined angle above and below.

FIG. 7 is a diagram illustrating cutting out of a panoramic image and a speaker image. As shown in FIG. 7, the image captured by the meeting device 60 forms part of a sphere 110 and therefore has a three-dimensional shape. As shown in FIG. 6B, the meeting device 60 performs perspective projection transformation by dividing the angle of view into vertical predetermined angles and left and right predetermined angles. A predetermined number of planar images can be obtained by performing perspective projection transformation over the entire 360 degrees in the horizontal direction without any gaps, so a panoramic image 111 can be obtained by connecting the predetermined number of planar images from side to side. Furthermore, the meeting device 60 creates a speaker image 112 by performing face detection from the panoramic image in a predetermined range centered on the audio direction and cutting out 15° left and right (30° in total) from the center of the face. do.

＜＜Electronic blackboard＞＞
FIG. 8 is a hardware configuration diagram of the electronic whiteboard 2. As shown in FIG. As shown in FIG. 8, the electronic whiteboard 2 includes a CPU (Central Processing Unit) 401, a ROM (Read Only Memory) 402, a RAM (Random Access Memory) 403, an SSD (Solid State Drive) 404, a network I/ F405 and an external device connection I/F (Interface) 406.

Among these, the CPU 401 controls the entire operation of the electronic whiteboard 2 . The ROM 402 stores programs used for starting the OS, such as IPL (Initial Program Loader). RAM 403 is used as a work area for CPU 401. The SSD 404 stores various data such as programs for the electronic whiteboard 2. Network I/F 405 controls communication with a communication network. External device connection I/F 406 is an interface for connecting various external devices. The external devices in this case are, for example, a USB (Universal Serial Bus) memory 430 and external devices (microphone 440, speaker 450, camera 460).

The electronic whiteboard 2 also includes a capture device 411, a GPU 412, a display controller 413, a contact sensor 414, a sensor controller 415, an electronic pen controller 416, a short-range communication circuit 419, an antenna 419a of the short-range communication circuit 419, a power switch 422, and Selection switches 423 are provided.

Among these, the capture device 411 displays display information on the display of the external PC 470 as a still image or a moving image. A GPU (Graphics Processing Unit) 412 is a semiconductor chip that specializes in graphics. The display controller 413 controls and manages screen display in order to output the output image from the GPU 412 to the display 480 or the like. The contact sensor 414 detects contact of the electronic pen 490, the user's hand H, etc. with the display 480. Sensor controller 415 controls processing of contact sensor 414. The contact sensor 414 performs coordinate input and coordinate detection using an infrared blocking method. This method of inputting and detecting coordinates is such that two light receiving and emitting devices installed at both ends of the upper side of the display 480 emit a plurality of infrared rays in parallel to the display 480. This is a method of receiving light that is reflected by a reflecting member and returns along the same optical path as the light emitted by the light receiving element. The light receiving and emitting device, which is the contact sensor 414, outputs the position where the infrared rays are blocked by the object (the position on the light receiving element) to the sensor controller 415, and the sensor controller 415 determines the contact position of the object from these two positional information. Identify a certain coordinate position. The electronic pen controller 416 communicates with the electronic pen 490 via Bluetooth to determine whether or not the display 480 has been touched with the tip of the pen or the butt of the pen. The near field communication circuit 419 is a communication circuit such as NFC (Near Field Communication) or Bluetooth (registered trademark). The power switch 422 is a switch for switching the power of the electronic blackboard 2 ON/OFF. The selection switches 423 are, for example, a group of switches for adjusting the brightness, shade, etc. of the display on the display 480.

Further, the electronic whiteboard 2 includes a bus line 410. The bus line 410 is an address bus, a data bus, etc. for electrically connecting each component such as the CPU 401 shown in FIG. 8.

Note that the contact sensor 414 is not limited to an infrared shielding type touch panel, and may be a capacitive type touch panel that specifies a touch position by detecting a change in capacitance. The contact sensor 414 may be a resistive touch panel that specifies the contact position based on voltage changes between two resistive films facing each other. The contact sensor 414 may be an electromagnetic induction type touch panel that detects electromagnetic induction generated when a touch object contacts the display unit and specifies the contact position. In addition, the contact sensor 414 may use various detection means. Furthermore, the electronic pen controller 416 may determine whether or not a portion of the electronic pen 490 that the user grasps or other portions of the electronic pen is touched, in addition to the pen tip and pen tail of the electronic pen 490.

<About functions>
Next, with reference to FIG. 9, the functional configuration of the recorded information creation system 100 will be described. FIG. 9 is an example of a functional block diagram illustrating functions of the terminal device 10, meeting device 60, and information processing system 50 in the recorded information creation system 100 divided into blocks.

<<Terminal device>>
The information recording application 41 that operates on the terminal device 10 includes a communication section 11, an operation reception section 12, a display control section 13, an application screen acquisition section 14, an audio acquisition section 15, a device communication section 16, a recording control section 17, and an audio data processing section. 18, a recording/playback section 19, an upload section 20, an editing section 21, and a code analysis section 22. Each of these units included in the terminal device 10 is realized by one of the components shown in FIG. 4 operating in response to an instruction from the CPU 501 according to the information recording application 41 expanded from the HD 504 to the RAM 503. It is a function or a means of functioning. Furthermore, the terminal device 10 has a storage unit 1000 constructed by the HD 504 shown in FIG. 4 and the like. An information storage unit 1001 is constructed in the storage unit 1000 .

The communication unit 11 communicates various information with the information processing system 50 via the network. The communication unit 11 receives, for example, a list of remote conferences from the information processing system 50, and transmits a voice data recognition request to the information processing system 50.

The display control unit 13 displays various screens serving as a user interface in the information recording application 41 according to screen transitions set in the information recording application 41. The operation accepting unit 12 accepts various operations on the information recording application 41.

The application screen acquisition unit 14 acquires a desktop screen or a screen of an application selected by the user from an OS (Operating System) or the like. If the application selected by the user is the remote conference application 42, a screen (images of each base, images of materials, etc.) generated by the remote conference application 42 is obtained.

The audio acquisition unit 15 acquires audio data received by the remote conference application 42 in a remote conference from the remote conference application 42. Note that the audio acquired by the audio acquisition unit 15 does not include the audio collected by the terminal device 10, but only the audio data received in the remote conference. This is because the meeting device 60 collects the audio separately.

The device communication unit 16 communicates with the meeting device 60 using a USB cable or the like. The device communication unit 16 may communicate with the meeting device 60 via wireless LAN, Bluetooth (registered trademark), or the like. The device communication unit 16 receives the panoramic image and the speaker image from the meeting device 60, and transmits the audio data acquired by the audio acquisition unit 15 to the meeting device 60. The device communication unit 16 receives audio data synthesized by the meeting device 60.

The recording control unit 17 combines the panoramic image and speaker image received by the device communication unit 16 and the application screen acquired by the application screen acquisition unit 14 to create a combined image. Furthermore, the recording control unit 17 connects the repeatedly created combined images in chronological order to create a combined image video, and combines the audio data synthesized by the meeting device 60 with the combined image video to create a combined image video with audio. create.

The audio data processing unit 18 causes the information processing system 50 to convert the audio data combined into the combined image video extracted by the recording control unit 17 or the synthesized audio data received from the meeting device 60 into text data. request.

The recording and playback unit 19 plays back the combined image video. The combined image video is stored in the terminal device 10 during recording, and then uploaded to the information processing system 50.

When the remote conference ends, the upload unit 20 transmits the combined image video to the information processing system 50.

The editing processing unit 21 executes editing (partial deletion, splicing, etc.) of the combined image video according to the user's operation.

The code analysis unit 22 detects a two-dimensional code included in the panoramic image, and also analyzes the two-dimensional code to obtain device identification information.

FIG. 10 shows moving image recording information stored in the information storage unit 1001. The video recording information includes items such as a conference ID, recording ID, update date and time, title, upload, and storage location. When the user logs in to the information processing system 50, the information recording application 41 downloads conference information from the conference information storage unit 5001 of the information processing system 50. The conference ID and the like included in the conference information are reflected in the video recording information. The video recording information in FIG. 10 is held by the terminal device 10 operated by a certain user.

- The conference ID is identification information that identifies the remote conference that was held. The conference ID is assigned when a remote conference schedule is registered in the conference management system 9, or assigned by the information processing system 50 in response to a request from the information recording application 41.

-Recording ID is identification information that identifies a combined image video recorded in a remote conference. The recording ID is numbered by the meeting device 60, but may also be numbered by the information recording application 41 or the information processing system 50. Different recording IDs are assigned to the same conference ID when recording ends during the remote conference but is restarted for some reason.

- The update date and time is the date and time when the combined image video was updated (recording ended). If the combined image video was edited, this is the date and time of editing.

・The title is the name of the conference. It may be set at the time of registering the conference to the conference management system 9, or may be set arbitrarily by the user.

- Upload indicates whether the combined image video has been uploaded to the information processing system 50.

- The save destination indicates the location (URL or file path) where the combined image video, text data, and object information are saved in the storage service system 70. Therefore, the user can view the uploaded combined image video at will. Note that the combined image video and text data are saved with separate file names following the URL, for example.

＜＜Meeting device＞＞
This will be explained by returning to FIG. 9. The meeting device 60 includes a terminal communication section 61 , a panoramic image creation section 62 , a speaker image creation section 63 , a sound collection section 64 , a voice synthesis section 65 , and a speaker change detection section 66 . Each of these parts of the meeting device 60 has a function that is realized when any of the components shown in FIG. It is a means of functioning.

The terminal communication unit 61 communicates with the terminal device 10 using a USB cable or the like. The terminal communication unit 61 need only be able to communicate with the terminal device 10 not only by a wired cable but also by wireless LAN, Bluetooth (registered trademark), or the like.

The panoramic image creation unit 62 creates a panoramic image. The speaker image creation unit 63 creates a speaker image. The method for creating these has been explained with reference to FIGS. 6 and 7.

The sound collection unit 64 converts the audio acquired by the microphone of the meeting device 60 into audio data (digital). As a result, the contents uttered by the users and participants are collected at the base on the terminal device 10 side.

The voice synthesis section 65 synthesizes the voice transmitted from the terminal device 10 and the voice collected by the sound collection section 64. As a result, the voices uttered at other bases 101 and the utterances from the own base 102 are combined.

The speaker change detection unit 66 records that the speaker has changed in association with the time. The speaker change detection unit 66 records that the speaker has been changed in association with the time when the microphone sound collection direction changes or the direction of a panoramic image cut out as a speaker image changes. The speaker change detection unit 66 identifies the speaker by voiceprint registration, face recognition, etc., or identifies the scribe by the user ID associated with the electronic pen 490. Therefore, even the identity of the speaker is specified.

<<Information processing system>>
The information processing system 50 includes a communication section 51 , an authentication section 52 , a display information providing section 53 , a communication management section 54 , a device management section 55 , a text conversion section 56 , and an importance determining section 57 . Each of these units included in the information processing system 50 has a function that is realized when any of the components shown in FIG. or a means of functioning. The information processing system 50 also includes a storage unit 5000 constructed from the HD 504 shown in FIG. 4 and the like. The storage unit 5000 includes a conference information storage unit 5001, a recording information storage unit 5002, and an association information storage unit 5003.

The communication unit 51 transmits and receives various information to and from the terminal device 10 . For example, the communication unit 51 transmits a list of remote conferences to the terminal device 10 or receives a voice data recognition request from the terminal device 10.

The authentication unit 52 authenticates the user who operates the terminal device 10. The authentication unit 52 authenticates the user, for example, depending on whether the authentication information (user ID and password) included in the authentication request received by the communication unit 51 matches authentication information held in advance. Note that the authentication information may be a card number of an IC card, biometric information such as a face or a fingerprint, or the like. Further, the authentication unit 52 may perform authentication using an external authentication system or an authentication method such as OAUTH.

The display information providing unit 53 provides information displayed by the information recording application 41. Since the screen configuration is owned by the information recording application 41, the display information providing unit 53 provides the heat map, activity level, etc. to the terminal device 10 in XML or the like. When the terminal device 10 executes a web application, the display information providing unit 53 generates screen information displayed by the web application. The screen information is created using HTML, XML, CSS (Cascade Style Sheet), JavaScript (registered trademark), and the like.

The communication management unit 54 acquires information regarding the remote conference from the conference management system 9 using each user's account or a system account assigned to the information processing system 50 . The communication management unit 54 stores the conference information of the reserved conference in the conference information storage unit 5001 in association with the conference ID. Further, the communication management unit 54 acquires conference information that users belonging to the tenant have viewing authority. Since a conference ID is set for a conference, a remote conference and recorded information are associated with each other by the conference ID.

When the device management section 55 receives the device identification information of the electronic whiteboard 2 and the meeting device 60 used in the conference, the device management section 55 associates them and stores them in the association information storage section 5003. Therefore, the conference ID, the device identification information of the electronic whiteboard 2, and the device identification information of the meeting device 60 are associated. Since the combined image video is also associated with the conference ID, the handwritten data input on the electronic blackboard 2 and the combined image video are also associated. Further, the device management section 55 deletes the association from the association information storage section 5003 when the recording is finished (when the meeting is finished).

The text conversion unit 56 converts voice data requested to be converted into text data from the terminal device 10 into text data using an external voice recognition service system 80 . The text conversion unit 56 itself may perform the conversion.

The importance level determining unit 57 determines the level of activity using the following level of activity determining elements included in the record information or object information.
・Number of strokes written in a certain period of time (different from the certain period of time for detecting silence) ・Number of utterances in a certain period of time ・Number of speakers in a certain period of time ・Keyword appearance in a certain period of time (writing + speaking) Number of Times FIG. 11 is an example of conference information stored in the conference information storage unit 5001 managed by the communication management unit 54. The communication management unit 54 can use the above account to obtain a list of remote conferences that the user belonging to the tenant is authorized to view. In this embodiment, a remote conference is taken as an example, but the list of remote conferences also includes conferences held in only one conference room.

Conference information is managed using a conference ID, and is associated with participants, title (meeting name), start date and time, end date and time, location, and the like. These are examples of conference information, and conference information may include other information.

- The participant item is the participant of the meeting.

- The title item represents the content of the meeting, such as the meeting name and meeting agenda.

- The start date and time item is the date and time when the meeting is scheduled to start.

- The end date and time item is the date and time when the meeting is scheduled to end.

- The location item is the location where the meeting will be held, such as a conference room, branch name, building, etc.

- The electronic blackboard information item is identification information of the electronic blackboard 2 used in the meeting.

- The item of meeting device is identification information of the meeting device 60 used in the meeting.

- The item of viewing authority is a user ID registered by the conference organizer as a user with viewing authority when registering conference information in advance or after the conference is held. For example, for each conference, only participants, participants + arbitrary user names, or arbitrary user names are registered in the conference information.

As shown in FIGS. 10 and 11, the combined image video recorded at the conference is specified by the conference ID.

The recording information stored in the recording information storage unit 5002 may be the same as that shown in FIG. However, the information processing system 50 has a list of combined image videos recorded by all users belonging to the tenant.

FIG. 12 shows association information stored in the association information storage unit 5003 in which the device identification information of the electronic blackboard 2 and the device identification information of the meeting device 60 are associated with the conference ID. This association information is held after the information recording application 41 transmits the device identification information to the information processing system 50 until the recording ends.

<<Storage service system>>
The storage service system 70 may be any service system that stores record information and object information. Recorded information storage unit 7001 stores recorded information (combined image moving image, text data) and object information.

FIG. 13 shows recorded information and object information stored in the recorded information storage unit 7001. As shown in FIG. 13, recorded information such as a combined image moving image, text data, and object information are stored in association with a conference ID. The combined image video includes synthesized audio, and the text data is the synthesized audio data converted by speech recognition. The object information is information regarding objects such as handwritten data input to the electronic blackboard 2, which will be described later. Speaker switching data is attached to the combined image video. Since the record information and object information are associated with the conference ID, they are also associated with the conference information.

FIG. 14 is a diagram illustrating the structure of text data that is part of recorded information. As shown in FIG. 14, the following items are associated with the text data: ID, time, recognition result character string, voice data, base identification information, and speaker ID.

・ID is identification information that is assigned a number when the local site voice and the other site voice are divided according to a predetermined rule. The predetermined rules are set in the meeting device 60 (and at least one of the voice recognition service system 80), and include, for example, dividing the meeting when silence continues for a certain period of time, or forcibly dividing the meeting after a certain period of time even if there is no silence. , dividing each sentence detected by morphological analysis, etc.

-time is the speaking time based on the duration from the start of recording. Since the so-called time is also saved at the start of recording, you can also know the time (absolute time) when the text was uttered.

・The recognition result string is part of the text data obtained by converting the divided synthesized speech using speech recognition. The synthesized speech is the speech data that is the source of the conversion of the recognition result character string.

- The audio data is a synthesized voice (already divided) in which the own site's voice and the other site's voice are synthesized after the base has been determined.

- The base identification information is the identification information of the base where the voice data was uttered, which is determined based on the sound pressure of the own base's voice and the sound pressure of the other base's voice. In the base identification information, for example, 1 represents the own base and 2 represents the other base.

-Speaker ID is a user ID indicating the speaker who uttered the recognition result character string. It is also possible to identify which participant made a comment based on the user ID. Several methods are known for identifying speakers during a conference. The identification method may be any method, such as a method in which each employee registers a voiceprint in advance, or a method in which the meeting device 60 detects the direction of the speaker and recognizes the faces of participants in that direction. But that's fine. In venues where microphones are provided for each speaker, the speaker is identified by which microphone collects the sound.

FIG. 15 is an example of speaker switching data that is saved together with the combined image video. This speaker switching data indicates the time when the speaker was switched by the camera. The meeting device 60 is capturing a panoramic image, and detects the horizontal angle of view where there is movement, and uses this and face detection (image processing to detect the presence of a human face) to detect when the participants are gesturing. can be detected. The switching of the speaker by the camera means that the participant who is gesturing has changed. In the example of FIG. 15, it is recorded that the speaker changed three times in the first minute immediately after the start, and six times in the minute around 14 minutes. Note that it is preferable that the user ID of the speaker is associated with the switching time using personal identification technology using face recognition. By using body language as an indicator of speaker switching, it becomes easier to identify times when discussions are active, rather than simply determining that the speaker has switched based on utterances.

＜＜Electronic blackboard＞＞
FIG. 16 is a functional block diagram illustrating the functions of the electronic whiteboard 2 divided into blocks. The electronic blackboard 2 includes a contact position detection section 31 , a drawing data generation section 32 , a data recording section 33 , a display control section 34 , a code generation section 35 , a communication section 36 , and an authentication section 37 . Each function that the electronic whiteboard 2 has is a function that is realized when any of the components shown in FIG. It is a means.

The contact position detection unit 31 detects the coordinates of the position where the electronic pen 490 contacts the contact sensor 414 . The drawing data generation unit 32 acquires the coordinates that the pen tip of the electronic pen 490 has touched from the contact position detection unit 31 . The drawing data generation unit 32 interpolates and connects this coordinate point sequence to generate stroke data.

The display control unit 34 displays handwritten data, menus for user operations, etc. on the display.

The data recording unit 33 stores handwritten data handwritten on the electronic blackboard 2, figures such as circles and triangles, stamps such as “done”, PC screens, files, etc. in the object information storage unit 3002. Handwritten data, graphics, images such as PC screens, files, etc. are treated as objects.

The communication unit 36 connects to WI-Fi or LAN and communicates with the information processing system 50. The communication unit 36 transmits object information to the information processing system 50, receives object information stored in the information processing system 50 from the information processing system 50, and causes the display 480 to display the object information.

The code generation unit 35 encodes the device identification information of the electronic whiteboard 2 stored in the device information storage unit 3001 and information indicating that the device is usable in a conference into a two-dimensional pattern to generate a two-dimensional code. . Further, the code generation unit 35 can encode the device identification information of the electronic whiteboard 2 and information indicating that the device is usable in a conference into a barcode. The device identification information may be a serial number, a UUID (Universally Unique Identifier), or the like. It may be set by the user.

The authentication unit 37 authenticates the user of the electronic whiteboard 2. The authentication method may be the same as that of the authentication section 52. Further, the authentication unit 37 may request the authentication unit 52 to perform authentication.

Furthermore, the electronic whiteboard 2 has a storage unit 3000 constructed by the SSD 404 shown in FIG. 8 or the like. The storage unit 3000 includes a device information storage unit 3001 and an object information storage unit 3002.

FIG. 17 shows device identification information and the like stored in the device information storage unit 3001.
- The device identification information is identification information of the electronic whiteboard 2.
- The IP address is an IP address for other devices to connect to the electronic whiteboard 2 via the network.
- The password is used for authentication when another device connects to the electronic whiteboard 2.

FIG. 18 is a diagram illustrating object information stored in the object information storage unit 3002. The object information is information for managing objects displayed on the electronic whiteboard 2. The object information is transmitted to the information processing system 50 and used as minutes. If the electronic blackboard 2 is placed at another base when a remote conference is held, object information is shared.

- In the item of conference ID, the identification information of the conference notified from the information processing system 50 is set.

- In the object ID field, identification information for identifying the object is set.

- In the type item, the type of object is set, and includes character strings, handwritten data, characters, figures, images, etc. The character string is one or more characters (the substance is a character code) obtained by handwriting recognition of handwritten data. Character strings are sometimes referred to as text data. Handwritten data is one or more stroke data (coordinate point sequence). Handwritten data is divided when the time from pen-up of a stroke to pen-down of the next stroke exceeds a threshold value. The figure is a geometric shape such as a triangle or a square. The image is image data such as Jpeg, Png, TI/FF, etc. imported from a PC or the Internet. For each object, the data body is stored in association with the object ID.

- One screen of the electronic blackboard 2 is called a page. The page item is the page number.

- In the coordinate item, the position of the object with respect to a predetermined origin of the electronic whiteboard 2 is set. The position of the object is, for example, the upper left vertex of the circumscribed rectangle of the object. For example, the coordinates are expressed in display pixel units.

・For the size item, the width and height of the circumscribed rectangle of the object are set.

- The scribe ID item is the user ID of the user who input the object. The user logs into the electronic blackboard 2 before starting to use the electronic blackboard 2. A user ID is identified by logging in. For example, when only one user inputs information to the electronic whiteboard 2 at the same time, the user ID of the user who logged in last is associated with the object. When a plurality of users input information to the electronic whiteboard 2 at the same time, it is preferable that the electronic pens and user IDs be associated with each other. For example, the ID of the electronic pen and the user ID are associated with each other in the order in which the user logs in, and the user ID that inputs the object is identified by the electronic pen that the user used for input. The reason why a plurality of user IDs are registered in the scribe ID field is because one piece of handwritten data has a plurality of strokes, each of which was handwritten by a different user. Note that in this embodiment, the scribe ID is assigned on the information processing system 50 side, but is illustrated in FIG. 18 for convenience of explanation. The electronic blackboard 2 may provide the scribe ID.

- In the time stamp item, the time when inputting the object started (an example of time information) is set. The timestamp may be the time when inputting the object ended.

FIG. 19 is a diagram illustrating the structure of stroke data that constitutes handwritten data. FIG. 19(a) is a conceptual diagram showing page data including strokes included in each page. The page data is stroke array data for one page displayed on the display 480. As shown in FIG. 19(a), the page data includes a page data ID for identifying any one page, a start time indicating the time when this page started to be displayed, and a page information such as a stroke or gesture. An end time indicating the time when the content is no longer rewritten, and a stroke array data ID for identifying stroke array data generated by strokes by the electronic pen 490 or the user's hand H are stored in association with each other.

With such page data, for example, when the user draws the alphabet "S" with the electronic pen 490, it is written in one stroke, so one stroke data ID indicates one letter of the alphabet [S]. However, when the user draws the alphabet "T" with the electronic pen 490, it is written in two strokes, so one letter of the alphabet "T" is indicated by two stroke data IDs.

The stroke array data shows detailed information as shown in FIG. 19(b). FIG. 19(b) is a conceptual diagram showing stroke array data. One stroke array data is represented by a plurality of stroke data. One piece of stroke data includes an object ID for associating with handwritten data, a stroke data ID for identifying stroke data, a start time indicating the time at which one stroke starts, and a time at which one stroke ends. , the color of the stroke, the width of the stroke, the coordinate array data ID for identifying the arrangement of passing points of the stroke, and the user ID of the scribe who wrote the stroke.

This coordinate array data shows detailed information as shown in FIG. 19(c). FIG. 19(c) is a conceptual diagram showing coordinate array data. As shown in FIG. 19(c), the coordinate array data includes one point (X coordinate value, Y coordinate value) on the display 480, and the difference from the stroke start time when passing this one point. The time (ms) and information on the writing pressure of the electronic pen 490 at this one point are shown. That is, the collection of one point shown in FIG. 19(c) is represented by one coordinate array data shown in FIG. 19(b). For example, when a user draws the alphabet "S" with the electronic pen 490, it is a single stroke, but the user passes through multiple passing points before finishing drawing "S", so the coordinate array data is information is shown.

<Processing or operation>
Next, the operation and processing performed by the recorded information creation system 100 will be explained based on the above configuration.

<<Save combined image video>>
With reference to FIG. 20, a process for saving a combined image moving image will be described. FIG. 20 is an example of a sequence diagram showing a procedure in which the information recording application 41 records a panoramic image, a speaker image, and an application screen. Although FIG. 20 describes a case where speech is recognized in real time, speech recognition may be performed at regular intervals or all at once after the end of a meeting.

S21: The user operates the remote conference application 42 to start a remote conference. Here, it is assumed that the remote conference application 42 of the local base 102 and the other base 101 have started a remote conference. The remote conference application 42 of the own base 102 transmits the image captured by the camera of the terminal device 10 and the sound collected by the microphone to the remote conference application 42 of the other base 101. The remote conference application 42 of the other base 101 displays the received image on the display and outputs the received audio from the speaker. Similarly, the remote conference application 42 of the other base 101 transmits the image captured by the camera of the terminal device 10 and the audio collected by the microphone to the remote conference application 42 of the own base 102 . The remote conference application 42 of the own base 102 displays the received image on the display and outputs the received audio from the speaker. Each remote conference application 42 repeats this process to realize a remote conference.

S22: The user makes recording settings such as recording items (whether to record the application screen of the remote conference application 42, audio, etc.). The operation reception unit 12 of the information recording application 41 accepts the settings.

If the user has reserved a remote conference in advance, the user can display a list of remote conferences and select the remote conference to which the combined image video will be associated. Since the user has already logged into the information processing system 50, the information processing system 50 identifies remote conferences that the logged-in user has viewing authority. The information processing system 50 transmits the list of identified remote conferences to the terminal device 10, so the user selects a remote conference that is currently being held or will be held in the future. As a result, information regarding the remote conference, such as the conference ID, is determined.

In addition, users can create a conference when creating a combined image video even if they have not reserved a remote conference in advance. In the following, a case will be described in which the information recording application 41 creates a conference when creating a combined image video and acquires the conference ID from the information processing system 50.

S23: The user instructs the information recording application 41 to start recording. The operation reception unit 12 of the information recording application 41 receives the instruction. The display control unit 13 displays a recording screen.

S24: Since the remote conference has not been selected (because the conference ID has not been determined), the communication unit 11 of the information recording application 41 transmits a remote conference creation request to the information processing system 50.

S25: The communication unit 51 of the information processing system 50 receives the remote conference creation request, the communication management unit 54 obtains a unique conference ID numbered by the conference management system 9, and the communication unit 51 records the conference ID as information. Send to application 41.

S26: Furthermore, the communication management unit 54 transmits the storage destination of the combined image video (URL of the storage service system 70) to the information recording application 41 via the communication unit 51.

S27: When the communication unit 11 of the information recording application 41 receives the conference ID and the storage location of the recorded file, the recording control unit 17 determines that preparations for recording are complete and starts recording.

S28: The application screen acquisition unit 14 of the information recording application 41 requests the application to display the screen of the application selected by the user (more specifically, the application screen acquisition unit 14 acquires the screen of the application via the OS). ). In FIG. 20, the application selected by the user is the remote conference application 42.

S29: The recording control unit 17 of the information recording application 41 notifies the meeting device 60 via the device communication unit 16 of the start of recording. The meeting device 60 transmits the panoramic image and the speaker image to the information recording application 41.

S30: When the terminal communication unit 61 of the meeting device 60 receives the start of recording, it assigns a unique recording ID number and returns the recording ID to the information recording application 41. Note that the recording ID may be assigned by the information recording application 41 or may be acquired from the information processing system 50.

S31: The remote conference service system 90 repeatedly transmits audio data and image data transmitted from other bases to the remote conference application.

S32: The audio acquisition unit 15 of the information recording application 41 acquires the audio data output by the terminal device 10 (the audio data received by the remote conference application 42).

S33: The device communication unit 16 transmits the audio data acquired by the audio acquisition unit 15 and the synthesis request to the meeting device 60.

S34: The terminal communication unit 61 of the meeting device 60 receives the audio data and the synthesis request. The sound collection unit 64 always collects surrounding sounds. The meeting device 60 divides the sound of other bases received by the terminal communication unit 61 and the sound of the own base collected by the sound collection unit 64 according to a predetermined rule, and determines the base based on the sound pressure of each.

S35: Next, the voice synthesis unit 65 synthesizes the local site voice (surrounding voice data) collected by the sound collection unit 64 and the other site voice received by the terminal communication unit 61. Therefore, synthesized speech is generated in a divided state. For example, the speech synthesis unit 65 adds the own site's voice and the other site's voice. Since the clear audio around the meeting device 60 is recorded, the accuracy of converting the audio around the meeting device 60 (on the conference room side) into text is particularly improved.

This voice synthesis is also possible in the terminal device 10. However, by distributing the recording function to the terminal device 10 and distributing the audio processing to the meeting device 60, the load on the terminal device 10 and the meeting device 60 can be reduced. The recording function may be distributed to the meeting device 60, and the audio processing may be distributed to the terminal device 10.

S36: The terminal communication unit 61 of the meeting device 60 transmits the voice recognition request (segmented synthesized voice) and base identification information to the information processing system 50.

S37: The communication unit 51 of the information processing system 50 receives the voice recognition request (divided synthesized voice) and the base identification information, and the text conversion unit 56 converts the voice recognition request (divided synthesized voice) into the voice recognition service system. and get the recognition result string.

S38: The information processing system 50 returns the recognition result character string, audio data, and base identification information to the information recording application 41. Since the information processing system 50 returns these to the information recording application 41, the meeting device 60 attaches its own identification information in step S36. Further, the information recording application 41 sets the IP address of the terminal device 10 and the identification information acquired from the meeting device 60 in the information processing system 50 in advance. By doing so, the information processing system 50 can specify the terminal device 10 based on the identification information of the meeting device 60.

S39: The communication unit 51 of the information processing system 50 stores the recognition result character string, audio data, and base identification information in the same storage location as the combined image video. Note that a conference ID is attached to these.

S40: Also, the panoramic image creation unit 62 of the meeting device 60 creates a panoramic image, and the speaker image creation unit 63 creates a speaker image. The speaker change detection unit 66 detects a change in speaker based on the panoramic image and records it.

S41: The device communication unit 16 of the information recording application 41 repeatedly acquires panoramic images and speaker images from the meeting device 60. Further, the device communication unit 16 repeatedly requests the meeting device 60 to obtain the synthesized audio data. These acquisitions may be performed by the device communication unit 16 requesting the meeting device 60. Alternatively, the meeting device 60 may automatically transmit the panoramic image and the speaker image. The meeting device 60 that has received the audio data synthesis request may automatically transmit the synthesized audio data to the information recording application 41.

S42: The display control unit 13 of the information recording application 41 displays the application screen, panoramic image, and speaker image side by side. Further, the recording control unit 17 of the information recording application 41 combines the application screen, panoramic image, and speaker image acquired from the remote conference application 42 and saves the combined image as a moving image. That is, the recording control unit 17 creates a combined image by combining the repeatedly received application screen, panoramic image, and speaker image, and performs the combination by specifying the combined image in the frames that constitute the combined image video. Create an image video. Additionally, the recording control unit 17 stores audio data received from the meeting device 60.

The information recording application 41 repeats the above steps S32 to S42.

S43: When the remote conference ends and recording is no longer necessary, the user instructs the information recording application 41 to end recording. The operation reception unit 12 of the information recording application 41 receives the instruction.

S44: The device communication unit 16 of the information recording application 41 notifies the meeting device 60 of the end of recording. The meeting device 60 continues to create panoramic images and speaker images and to synthesize audio. However, the meeting device 60 may change whether processing is prohibited, such as by changing the resolution or fps, depending on whether recording is in progress.

S45: The recording control unit 17 of the information recording application 41 combines audio data with the combined image video to create a combined image video with audio. If the panoramic image, speaker image, and application screen are not saved at all, the audio data may be independent.

S46: The upload unit 20 of the information recording application 41 stores the combined image video in the combined image video storage destination via the communication unit 11. In the recording information storage unit 5002, the combined image video is associated with a conference ID and a recording ID. Uploaded is recorded in the combined image video.

Note that the processing in steps S32 to S42 does not have to be performed in the order shown in FIG. 20, and the synthesis of audio data and the storage of the combined image video may be performed before or after the other.

S47: The user inputs the end of the meeting into the electronic blackboard 2. The user may input the end of the meeting into the terminal device 10, and the end of the meeting may be transmitted from the terminal device 10 to the electronic blackboard 2. In this case, the end of the meeting may be sent to the electronic blackboard 2 via the information processing system 50.

S48: The communication unit 36 of the electronic whiteboard 2 specifies the conference ID and transmits object information (eg, handwritten) displayed during the conference and speaker switching data to the information processing system 50. The communication unit 36 may transmit the device identification information of the electronic whiteboard 2 to the information processing system 50. In this case, the conference ID is specified by the association information.

S49: Upon receiving the object information and speaker switching data, the communication unit 51 of the information processing system 50 divides the handwritten data every time the time interval between strokes becomes equal to or greater than a threshold. Note that this division may be performed using the electronic whiteboard 2. The communication unit 51 of the information processing system 50 saves the object information and speaker switching data in the same storage destination as the combined image video and the like based on the conference ID.

Since the storage location is notified to the user (in this embodiment, the viewer), the user can share the combined image video with participants by notifying the storage location via e-mail or the like. Even if the combined image video, audio data, text data, and object information are created by different devices, they can be collected and stored in one storage location, and can be easily viewed by a user or the like later.

<Display processing of combined image video>
FIG. 21 is a sequence diagram showing a procedure in which a viewer operates an information recording application to display a combined image video. The combined image video may be displayed by a web application.

S51: The viewer displays conference information as shown in FIG. 11 on the information recording application, and selects the conference for which the combined image video is to be viewed. Additionally, the viewer can input any keyword to determine the activity level. The operation reception unit 12 of the terminal device 10 receives these.

S52: Thereby, the communication unit 11 specifies the conference ID and requests the information processing system 50 for a combined image video.

S53: The communication unit 51 of the information processing system 50 receives the request for a combined image video, and acquires the combined image video specified by the conference ID from the storage service system 70.

S54: The communication unit 51 also obtains speaker switching data attached to the combined image video from the storage service system 70.

S55: Also, the communication unit 51 obtains text data from the storage service system 70. The text data includes a speaker for each recognition result character string.

S56: The communication unit 51 also obtains object information from the storage service system 70. Object information also includes character strings that are recognized from stroke data and handwritten data.

S57: Next, the display information providing unit 53 generates a combined image video display screen 240, which will be described later, and displays the degree of activity at fixed time intervals as a visual heat map, and arranges it along the time indicator 241d. do. Note that the heat map may be created by the terminal device 10. In this case, the information processing system 50 transmits the degree of activity to the terminal device 10 at regular intervals.

S58: Also, the importance level determining unit 57 determines the level of activity. Here, the degree of activity based on all of the plurality of activity level determining factors, that is, the importance determining unit 57 determines the degree of activity in which the degrees of activity based on the plurality of activity level determining factors are integrated. Details will be explained with reference to FIG.

S59: The communication unit 51 of the information processing system 50 transmits the screen information of the combined image moving image display screen 240 to the terminal device 10.

S60: The communication unit 11 of the terminal device 10 receives the screen information of the combined image video display screen 240, and the display control unit 13 displays the combined image video display screen 240.

S61: The viewer can specify an activity determining element for determining the activity level. Here, it is assumed that the viewer specifies only the frequency of utterances. The operation reception unit 12 accepts the operation.

S62: Thereby, the communication unit 11 requests the information processing system 50 to determine the degree of activity with the frequency of remark as the factor determining the degree of activity.

S63: The communication unit 51 of the information processing system 50 receives the message that the frequency of comments is to be used as an activity determining factor, and the importance determining unit 57 determines the degree of activity using only the frequency of comments. The display information providing unit 53 uses this activity level to create a heat map by associating the activity level at fixed time intervals with color and brightness.

S64: The communication unit 51 of the information processing system 50 transmits to the terminal device 10 a heat map created based on the activity level based on the frequency of speech.

S65: The communication unit 11 of the terminal device 10 receives information for displaying the heat map, and the display control unit 13 displays the heat map along the time indicator 241d.

S66: Next, the viewer specified an arbitrary keyword as an activity level determining element for determining the activity level. The operation reception unit 12 receives keywords.

S67: Thereby, the communication unit 11 requests the information processing system 50 to set the activity level determining element as an arbitrary keyword.

S68: The communication unit 51 of the information processing system 50 receives the notification that the keyword is to be used as an activity level determining element, and the importance level determination unit 57 determines the activity level using the keyword. The display information providing unit 53 uses this activity level to create a heat map representing the activity level at regular intervals.

S69: The communication unit 51 of the information processing system 50 transmits the heat map created with the activity level based on the keyword to the terminal device 10.

S70: The communication unit 11 of the terminal device 10 receives the heat map, and the display control unit 13 displays the heat map along the time indicator 241d.

<Display of combined image video>
FIG. 22 is an example of a combined image moving image display screen 240 displayed by the information recording application 41. Note that although the combined image video display screen 240 is a screen that is displayed after the meeting ends, the screen during the meeting is almost the same. The combined image video display screen 240 has a conference bibliographic information field 221, an image display field 241, a transcription button 242, one or more text display fields 243, an automatic scroll button 244, and a search button 245.

- The conference bibliographic information column 221 is a column in which at least part of conference information is displayed. In FIG. 22, the name of the conference, the time and place where the conference will be held are displayed.

- The image display field 241 has a display button 241a, a rewind button 241b, a fast forward button 241c, a time indicator 241d, a display speed button 241e, a volume button 241f, etc. The image display column 241 displays a combined image moving image. In FIG. 22, in the combined image video in the image display field 241, a panoramic image and a speaker image are arranged on the left side, and a screen of the remote conference application 42 is displayed on the right side. The screen of the remote conference application 42 becomes an image of the base or an image of materials during the remote conference. Therefore, the viewer can view a desired scene on the screen by operating various buttons.

- If the audio data of the combined image video currently being displayed in the image display field 241 has been converted to text data, the text display field 243 displays the content of the statement in text. The text display field 243 displays text data obtained by converting synthesized speech data through speech recognition.

- The transcription button 242 is a button that allows the viewer to switch whether or not to display the text data displayed in the text display field 243 in accordance with the display time of the combined image video.

- The automatic scroll button 244 is a button that allows the viewer to switch whether or not to automatically scroll the text data regardless of the display time.

- The search button 245 is a button for the viewer to search for text data by specifying a keyword.

- The heat map 246 visually represents the activity level of the meeting. Details will be described later.

<<How to determine activity level>>
Next, a method for determining the degree of activity will be explained with reference to FIG. 23. FIG. 23 is a flowchart showing the procedure by which the importance determining unit 57 determines the degree of activity. The importance determination unit 57 may be determined when the viewer requests the information processing system 50 to reproduce the combined image video, or may be executed when the recorded information is saved. Once determined, the degree of activity may be stored in association with recorded information.

The importance determining unit 57 obtains the next activity determining element (S101).

・Frequency A of writing strokes at regular intervals
The writing frequency of strokes per fixed time is how many pieces of stroke data shown in FIG. 19 are written per fixed time. Note that the frequency of writing strokes for each certain period of time may be the number of handwritten data for each certain period of time. The certain period of time is, for example, one minute, but a suitable period of time is determined as appropriate.

・Number of statements per fixed time B
The number of utterances per certain period of time is the number of words that speech-recognized text data includes at each certain period of time. Therefore, the importance determining unit 57 morphologically analyzes the text data and breaks it down into words with parts of speech. The number of utterances per certain period of time may be the number of characters.

・Number of speakers per fixed time C
The number of speakers per fixed time is the number of speakers (different number) per fixed time. Since speakers are associated with the text data, the importance determining unit 57 counts speakers who appear in a certain period of time so as not to overlap.

・Frequency of speaker switching at regular intervals D
The speaker switching frequency at fixed time intervals is the number of times the speaker switches. The frequency of speaker switching at regular time intervals can be recorded in speaker switching data (detection of participant movement in a panoramic image) or detected by speakers associated with text data. Further, the importance determining unit 57 may determine that the speaker has changed in at least one of the two.

・Keyword appearance frequency E, F (spoken or handwritten) at regular intervals
The appearance frequency of a keyword at a certain time interval is the number of times a keyword specified by a viewer is included in text data (audio data) or a character string (handwritten data that is character-recognized). For example, by specifying a topic that is being actively discussed by the viewer, such as "electronic blackboard" or "projector," the user can input the information in order to guess which time period of the recorded information should be viewed.

Next, the importance determining unit 57 determines the following (S102).
・The maximum value Amax during the meeting of the stroke writing frequency A for each fixed period of time
・The maximum value Bmax during the meeting among the number of speeches B per certain period of time
・Out of the number of speakers C per fixed time, the maximum value Cmax during the conference
・The maximum value Dmax during the conference among the frequency D of changing speakers at regular intervals
・Among keyword appearance frequencies E and F at regular intervals, maximum values Emax and Fmax during the meeting
Next, the importance determining unit 57 calculates the activity level at regular intervals using the following formula (S103).
Activity level = (A/Amax + B/Bmax + C/Cmax + D/Dmax+E/Emax+F/Fmax)/6
Therefore, the activity level is 0 or more and less than 1. This level of activity is an integrated level of activity based on six activity level determining factors, and the level of activity of each individual activity level determining element is as follows.
・Activity level based on stroke writing frequency A = A/Amax
・Activity level based on number of comments B = B/Bmax
・Activity level based on number of speakers C = C/Cmax
・Activity level based on speaker switching frequency D = D/Dmax
・Activity level based on keyword appearance frequency E, F = E/Emax, F/Fmax,
The importance level determining unit 57 can also determine the integrated level of activity by combining any two or more of the plurality of activity level determining elements.

Further, the importance determining unit 57 may assign weights to the activity determining factors. For example, if it is desired to strongly reflect the writing frequency A of strokes at regular time intervals in the activity level, the importance determining unit 57 multiplies A by a coefficient, such as doubling it. By doing so, the information processing system 50 can create a heat map in which time periods in which the writing frequency A of strokes for each fixed period of time is high are likely to be emphasized.
Activity level = (2A/Amax + B/Bmax + C/Cmax + D/Dmax+E/Emax+F/Fmax)/7
Although the total amount (number of words) of handwritten data and utterances is originally different, by weighting it, it is possible to control the effects of both by averaging out or giving more importance to one.

As described above, since the degree of activity for each fixed period of time has been determined, the display information providing unit 53 can create a heat map by assigning the degree of activity to color and brightness. In this method, the part of each meeting that is evaluated to be the most active is expressed with the highest brightness, regardless of the meeting.

Furthermore, the certain period of time may be changed depending on the time of the meeting (the recording time of the combined image video). This is because there is a restriction on the length of the heat map 246. For example, the importance determining unit 57 sets the certain time to 1/100 to 1/10,000 of the recording time of the combined image video. Depending on the assumed display resolution (DPI: Dots Per Inch), the heat map 246 has a length (the number of pixels in the length direction of the heat map 246) that can be divided into approximately several hundred to 10,000 pieces. Therefore, by determining the predetermined time according to the recording time of the combined image video, the color and brightness for each predetermined time can be reflected in the heat map 246. The loss of the activity level for each shorter time period due to a longer certain period of time can be dealt with by expanding the heat map 246, which will be described later.

In addition, when the activity level is determined by individual activity level determining factors, for a certain period of time, the frequency of writing strokes A, the number of utterances B, the number of speakers C, the frequency of speaker switching D, the frequency of appearance of keywords E, F It may differ depending on the For example, the fixed time period for the number of utterances B, which has a large absolute number, may be shortened, and the appearance frequencies E and F of keywords, which have a small absolute number, may be made longer.

<Example of time indicator display>
Next, display examples of heat maps will be described with reference to FIGS. 24 to 31. First, FIG. 24 shows a heat map 246 that expresses the degree of activity in terms of brightness for one color. In this case, the viewer may decide on one color, or the colors may be decided in advance. In FIG. 24, the level of activity is expressed by the level of brightness. Note that in order to change the brightness while keeping the same color, it is preferable that the display information providing unit 53 uses the HSV color space. HSV is a method of expressing color using three elements: "hue", "saturation", and "value/brightness". Therefore, the display information providing unit 53 may change the brightness while keeping the hue the same. Note that since the display of the terminal device 10 is displayed in RGB, colors expressed in HSV are converted to RGB.

Since the color and brightness for each fixed time are determined in this way, the display information providing unit 53 divides the length of the heat map 246 by the length of the combined image video, and calculates the length on the time indicator 241d to which the fixed time corresponds. Calculate.

Length of heat map (number of pixels) ÷ Length of combined image video (min) = Length of heat map per unit time (number of pixels/min) ...(1)
Length on the heat map corresponding to a certain time (number of pixels) = "Length of heat map per unit time (number of pixels/min)" * Fixed time (number of pixels)... (2)
The display information providing unit 53 creates the heat map 246 by arranging the color and brightness determined from the activity level for each length of the heat map 246 corresponding to a certain period of time. Furthermore, the heat map 246 is associated with the degree of activity (the degree of activity as a numerical value) at regular time intervals.

The heat map 246 arranged along the time indicator 241d in this way shows in color and brightness which time period of the recorded time of the combined image video was lively, that is, whether active discussion was taking place. It becomes information.

When the viewer hovers over the heat map 246 with the pointing device 301 such as a mouse, the display control unit 13 of the terminal device 10 may display the activity level in a pop-up manner. In FIG. 24, an activity level of "0.8" is displayed. By clicking the mouse, the pointer 303 of the progress bar of the time indicator 241d moves to the clicked position, so that the viewer can cause the terminal device 10 to reproduce the combined image video at the time corresponding to the specified activity level.

An activity level button 302 is displayed on the right side of the heat map 246. When the viewer presses the activity level button 302, the pointer 303 moves to the time when the activity level is greater than or equal to the threshold value. As described above, since the degree of activity is associated with the position on the heat map 246, the display information providing unit 53 sequentially compares the degree of activity associated with the threshold and presses the degree of activity button 302. Screen information for displaying a pointer 303 at a position associated with an activity level equal to or higher than a threshold value is created for each case. Although it is difficult for the viewer to accurately read the degree of activity using the heat map 246 alone, the degree of activity button 302 allows the terminal device 10 to reproduce the combined image video at the time when the degree of activity is equal to or higher than the threshold value.

Furthermore, the display information providing unit 53 preferably provides words that appear frequently in the portions of the heat map 246 that have a high degree of activity (the degree of activity is equal to or higher than a threshold value). That is, when the degree of activity associated with a position on the heat map 246 is greater than or equal to the threshold, the display information providing unit 53 acquires words with a high frequency of appearance from the text data or character strings before and after the time corresponding to the part. do. A word with a high frequency of appearance may be a word with a top N number of appearances. The number of occurrences may be one.

When the viewer mouses over the heat map 246 with the pointing device 301, a pop-up display of the word associated with the portion that the pointing device 301 mouses over is displayed. In Figure 24, the word "sales results" is displayed. By doing this, it becomes possible to estimate what kinds of topics were actively discussed in areas with high activity.

Next, with reference to FIG. 25, a heat map 246 in which the level of activity level is expressed by gradation will be described. FIG. 25 shows a heat map 246 that expresses the degree of activity using a plurality of color gradations. For example, if the activity level is "high" and "low" is blue, the display information providing unit 53 assigns the activity level of "1" to red (255, 0, 0), and the activity level of "0" to the red (255, 0, 0). ” is assigned to blue (0, 0, 255). When the activity level is α (α is 0 or more and 1 or less), the color is determined as follows.
R=α*255
G=0
B=(1-α)*255
As a result, the heat map 246 has a color that is a mixture of red and blue, and the higher the activity level, the stronger the reddish color, and the lower the activity level, the stronger the bluish color.

Note that red and blue are just examples, and may be set arbitrarily by the viewer or may be determined in advance by the display information providing unit 53.

Further, as shown in FIG. 26, the terminal device 10 may display the activity levels determined by different activity level determining factors in separate heat maps 246. FIG. 26 shows a heat map 246 representing the activity determined by different activity determining factors. In FIG. 26, the degree of activity based on the writing frequency A of stroke data and the degree of activity based on the speaker switching frequency D are shown in a heat map 246, respectively. Note that the display information providing unit 53 may express the two degrees of activity in different colors, or may express them in the same color. The viewer may customize the color, or the display information providing unit 53 may change the color for each activity level determining element.

FIG. 27 shows a heat map in which the frequency of appearance of keywords is used as the degree of activity. When a viewer specifies a specific keyword, the degree of activity is determined based on the frequency of appearance of that keyword. Keywords are searched from text data and character strings converted from handwritten data. Note that the text box 305 may be used to specify the keyword, or the viewer may select from a pull-down menu that lists frequently occurring words extracted in advance from text data or character strings. In FIG. 27, a heat map 246 is displayed together with the specified keyword, in which the color of the time period in which the keyword is uttered the most is darker.

The heat map 246, which uses the frequency of appearance of keywords as the degree of activity, allows the viewer to estimate when a specific topic was being discussed in a meeting with multiple topics.

FIG. 28 shows a heat map 246 displayed for each activity determining factor. As shown in FIG. 28, the display information providing unit 53, by the viewer's operation or automatically, displays a heat map 246a in which the activity level based on all the activity level determination factors is integrated, and the stroke writing frequency. Heat map 246b based on A, heat map 246c based on number of utterances B, heat map 246d based on number of speakers C, heat map 246e based on speaker switching frequency D, heat map 246f based on keyword appearance frequencies E and F. , 246g can be created separately. Further, regarding the heat maps 246f and 246g based on the keyword appearance frequencies E and F, the terminal device 10 displays the heat maps 246f and 246g based on the frequency with which the voice-recognized text data and handwritten-recognized character strings each include the keyword. can be displayed. The colors forming the heat map 246 may be different or the same. The viewer may be able to specify the color.

Which activity determining factors are used to identify times during which useful discussions are occurring may vary from meeting to meeting. By displaying the heat map 246 for each activity determining factor as shown in FIG. 28, the viewer can arbitrarily select the activity level that suits the way each meeting is to proceed.

Furthermore, although the lengths of the combined image videos differ, the length of the heat map 246 is constant. Therefore, if the combined image video is long, the viewer may not be able to read the detailed activity level from the heat map 246. Therefore, when the viewer mouses over the heat map 246 with the pointing device 301, the terminal device 10 may enlarge and display the heat map 246. For example, the display information providing unit 53 displays a heat map of the activity level for each time period shorter than the fixed time period when the heat map 246 is created, for a total of about 10 minutes before and after the time on the heat map 246 to which the position of the pointing device 301 corresponds. Create. This heat map is referred to as an enlarged heat map 247.

FIG. 29 shows an example of an enlarged heat map 247. In this case, the display information providing unit 53 creates the heat map 246 by changing the fixed time period between the entire heat map 246 and the enlarged heat map 247. For example, in the case of the heat map 246, the display information providing unit 53 determines the degree of activity every minute, and in the case of the enlarged heat map 247, the degree of activity is determined every 10 to 30 seconds. By doing this, viewers can check the activity level in a shorter period of time.

Further, as shown in FIG. 30, the display information providing unit 53 can represent the activity level of each participant in the conference using a heat map 246. FIG. 30 is an example of a heat map 246 in which the degree of activity of the participants is defined as the degree of activity. Depending on the conference, presenters and topics may change depending on the time period, and viewers may want to view only predetermined presenters and topics. In such a case, the viewer specifies the participants in the conference and has the degree of activity displayed in the heat map 246. Since the text data is associated with a speaker and the object information is associated with a scribe, the text data and handwritten data uttered or handwritten by the participant specified by the viewer can be easily identified.

Note that the display information providing unit 53 obtains the conference participants from the conference information in advance, and allows them to be displayed using the pull-down menu 248 or the like. When a participant is designated, the importance determining unit 57 searches text data and object information using the participant's user ID, and calculates the stroke writing frequency A and the number of utterances B (by this participant) at regular intervals. The number of words included in the text data uttered by this participant is calculated. Therefore, a heat map 246 is created in which the time period in which the designated participant was active is emphasized. This makes it possible to estimate each participant's degree of participation in the conference.

FIG. 31 is a diagram illustrating a method of displaying activity levels that displays time periods with high activity levels without using colors. In FIG. 31, an activity bar 249 is displayed along the time indicator 241d instead of the heat map. The activity level bar 249 emphasizes time periods in which the activity level is greater than or equal to the threshold value. In FIG. 31, time periods in which the degree of activity is equal to or greater than the threshold are indicated by rectangles 250a to 250d. Although the viewer does not know the degree of activity, it is easier to identify times when discussions are active than from the heat map 246.

Further, the rectangles 250a to 250d may have different shapes or colors depending on the degree of activity, or marks (large, medium, small, etc.) to indicate the degree of activity.

<Main effects>
As explained above, the recorded information creation system 100 of the present embodiment provides a heat map representing the degree of activity in time series, so that the viewer can quickly identify the parts that should or should be viewed from the combined image video. become discoverable.

<Other application examples>
Although the best mode for carrying out the present invention has been described above using examples, the present invention is not limited to these examples in any way, and various modifications can be made without departing from the gist of the present invention. and substitutions can be added.

For example, the importance level determining unit 57 may determine the level of activity using video. In this case, the importance determining unit 57 converts the magnitude of change in the panoramic image at regular time intervals into the degree of activity. Further, the importance determining unit 57 may convert only the mouth movement into the degree of activity, for example.

Moreover, instead of the heat map 246, a screenshot of a combined image video at a time when the degree of activity is equal to or higher than a threshold value may be displayed. Text data uttered at this time may be attached to the screenshot. Viewers can view the screenshots to understand the status of the meeting and decide whether or not to view it. Alternatively, when the viewer hovers over the heat map 246 with the pointing device 301, a screenshot at the time corresponding to the position of the pointing device 301 may be displayed as a pop-up if the degree of activity is equal to or higher than a threshold value.
Further, the heat map 246 may not be placed below the combined image video, but may be placed above. Heatmap 246 does not have to be displayed with the same width as the combined image video. The heat map 246 may be located entirely separate from the combined image video. In addition to being placed below the combined image video, the heat map 246 may be placed vertically on the right or left side of the combined image video. In this case, the upper side corresponds to the start of the conference, and the lower side corresponds to the end of the conference.

Further, the terminal device 10 and the meeting device 60 may be integrated. The meeting device 60 may be externally attached to the terminal device 10. Further, the meeting device 60 may include a spherical camera, a microphone, and a speaker connected by a cable.

Further, the meeting device 60 may be arranged at other bases 101 as well. The other bases 101 use the meeting device 60 to create a combined image video and text data. Further, a plurality of meeting devices 60 may be arranged at one base. In this case, a plurality of pieces of recorded information are created for each meeting device 60.

Further, the arrangement of the panoramic image 203, the speaker image 204, and the application screen in the combined image video used in this embodiment is only an example. The panoramic image 203 may be on the bottom and the speaker image 204 on the top, or the user may change the arrangement or individually switch between displaying and non-displaying the panoramic image 203 and the speaker image 204 during playback.

Furthermore, the configuration example shown in FIG. 9 and the like is divided according to main functions in order to facilitate understanding of the processing by the terminal device 10, the meeting device 60, and the information processing system 50. The present invention is not limited by the method of dividing the processing units or the names thereof. The processing of the terminal device 10, the meeting device 60, and the information processing system 50 can also be divided into more processing units depending on the processing content. Furthermore, one processing unit can be divided to include more processing.

Additionally, the devices described in the Examples are merely illustrative of one of multiple computing environments for implementing the embodiments disclosed herein. In some embodiments, information handling system 50 includes multiple computing devices, such as a server cluster. The plurality of computing devices are configured to communicate with each other via any type of communication link, including a network, shared memory, etc., to perform the processes disclosed herein.

Further, the information processing system 50 can be configured to share the disclosed processing steps, for example, FIGS. 20 and 21 in various combinations. For example, a process executed by a predetermined unit may be executed by a plurality of information processing devices included in the information processing system 50. Further, the information processing system 50 may be integrated into one server device, or may be divided into multiple devices.

Each function of the embodiments described above can be realized by one or more processing circuits. Here, the "processing circuit" in this specification refers to a processor programmed to perform each function by software, such as a processor implemented by an electronic circuit, or a processor designed to perform each function described above. It includes devices such as ASIC (Application Specific Integrated Circuit), DSP (Digital Signal Processor), FPGA (Field Programmable Gate Array), and conventional circuit modules.
<Additional notes regarding claims>
[Claim 1]
An information processing system that provides information recorded by a device capable of recording audio and video to a terminal device,
Based on at least one of recorded information including the audio or handwritten data associated with the recorded information, importance information regarding the recorded information is determined for each time shorter than a recording time of the recorded information. An importance determining section,
a display information providing unit that provides the terminal device with information for displaying the importance information along the recording time of the recorded information;
An information processing system with
[Claim 2]
The importance determining unit is
2. The information processing system according to claim 1, wherein the importance information is determined by dividing it into periods shorter than a recording time based on the writing frequency of stroke data constituting the handwritten data.
[Claim 3]
The importance determining unit is
3. The information processing system according to claim 1, wherein the importance information is determined by dividing it into periods shorter than a recording time based on the number of times the voice is uttered.
[Claim 4]
The importance determining unit is
The information processing system according to any one of claims 1 to 3, wherein the importance information is determined by dividing it into periods shorter than a recording time based on the number of speakers of the audio.
[Claim 5]
The importance determining unit is
The information processing system according to any one of claims 1 to 4, wherein the importance information is determined by dividing it into periods shorter than a recording time based on a frequency of switching speakers of the audio.
[Claim 6]
The importance determining unit is
Recording the importance information based on at least one of the frequency of appearance of a keyword included in a character string obtained by character recognition of the handwritten data, and the frequency of appearance of a keyword included in text data obtained by voice recognition of the voice. The information processing system according to any one of claims 1 to 5, characterized in that the information processing system is determined by dividing it into periods shorter than a certain period of time.
[Claim 7]
The importance determining unit is
The writing frequency of stroke data constituting the handwritten data, the number of times the voice is uttered, the number of speakers of the voice, the frequency of switching speakers of the voice, and the appearance of keywords included in the character string from which the handwritten data is recognized. The importance information is determined by integrating at least two of the frequency and the frequency of appearance of the keyword included in the text data in which the voice is voice-recognized, and dividing the importance information into periods shorter than the recording time. The information processing system according to claim 1.
[Claim 8]
The importance determining unit is configured to determine the writing frequency of stroke data constituting the handwritten data, the number of times the voice is uttered, the number of speakers of the voice, the frequency of switching speakers of the voice, and the characters in which the handwritten data is recognized. determining the importance information based on the frequency of occurrence of keywords included in the column and the frequency of occurrence of keywords included in the text data in which the voice has been voice-recognized, divided into periods shorter than the recording time;
The display information providing unit includes a writing frequency of stroke data constituting the handwritten data, a number of times the voice is uttered, a number of speakers of the voice, a frequency of switching speakers of the voice, and a character whose character is recognized in the handwritten data. The present invention is characterized by providing information for individually displaying the importance information based on the appearance frequency of the keyword included in the column and the appearance frequency of the keyword included in the text data in which the speech is voice-recognized. The information processing system according to claim 7.
[Claim 9]
The importance determining unit is configured to determine the writing frequency of stroke data constituting the handwritten data, the number of times the voice is uttered, the number of speakers of the voice, the frequency of switching speakers of the voice, and the characters in which the handwritten data is recognized. The importance information, which is integrated by giving weight to one or more of the appearance frequency of the keyword included in the column and the appearance frequency of the keyword included in the text data in which the voice is voice-recognized, is shorter than the recording time. 8. The information processing system according to claim 7, wherein the information processing system is determined in units of time.
[Claim 10]
10. The information processing system according to claim 7, wherein the keyword is specified by a viewer on the terminal device.
[Claim 11]
The keyword is selected by the viewer on the terminal device from among the words included in the character string obtained by character recognition of the handwritten data or the text data obtained by voice recognition of the voice. The information processing system according to claim 10.
[Claim 12]
The display information providing unit provides the importance information displayed along the recording time with words whose frequency of appearance for each time period is equal to or higher than a threshold value, and displays the words according to the viewer's operation. The information processing system according to any one of claims 1 to 11, characterized in that the information processing system provides information for display.
[Claim 13]
The display information providing unit associates a numerical value of the importance level information with the importance level information that is displayed along the recording time, and displays the numerical value that is equal to or higher than a threshold value in response to an operation by a viewer. The information processing system according to any one of claims 1 to 12, characterized in that the information processing system provides information on.
[Claim 14]
Identification information of conference participants is associated with the handwritten data and the voice,
The importance determining unit determines the importance level based on at least one of the writing frequency of stroke data included in the handwritten data associated with the participant, or the number of times the voice is uttered associated with the participant. 14. The information processing system according to claim 1, wherein the importance information is determined by dividing it into periods shorter than a meeting.
[Claim 15]
The frequency at which the speaker of the audio changes is determined based on at least one of the direction of the audio detected by the device or the direction of the speaker detected in the video imaged by the device. The information processing system according to claim 5.
[Claim 16]
16. The display information providing unit provides the terminal device with information for displaying the size of the importance information in a gradation of a plurality of colors. information processing system.
[Claim 17]
16. The display information providing unit provides the terminal device with information for displaying the magnitude of the importance information in terms of brightness for any color. The information processing system according to item 1.

10 Terminal device 50 Information processing system 60 Meeting device 70 Storage service system 100 Recorded information creation system

Japanese Patent Application Publication No. 8-320940

Claims (20)

An information processing system that provides information recorded by a device capable of recording audio and video to a terminal device,
Based on at least one of recorded information including the audio or handwritten data associated with the recorded information, importance information regarding the recorded information is determined for each time shorter than a recording time of the recorded information. An importance determining section,
a display information providing unit that provides the terminal device with information for displaying the importance information along the recording time of the recorded information;
An information processing system with The importance determining unit is
2. The information processing system according to claim 1, wherein the importance information is determined by dividing it into periods shorter than a recording time based on the writing frequency of stroke data constituting the handwritten data. The importance determining unit is
3. The information processing system according to claim 1, wherein the importance information is determined by dividing it into periods shorter than a recording time based on the number of times the voice is uttered. The importance determining unit is
2. The information processing system according to claim 1, wherein the importance information is determined by dividing it into periods shorter than a recording time based on the number of speakers of the audio. The importance determining unit is
2. The information processing system according to claim 1, wherein the importance information is divided into periods shorter than a recording time and determined based on a frequency of switching between speakers of the audio. The importance determining unit is
Recording the importance information based on at least one of the frequency of appearance of a keyword included in a character string obtained by character recognition of the handwritten data, and the frequency of appearance of a keyword included in text data obtained by voice recognition of the voice. 2. The information processing system according to claim 1, wherein the information processing system is determined by dividing it into periods shorter than time. The importance determining unit is
The writing frequency of stroke data constituting the handwritten data, the number of times the voice is uttered, the number of speakers of the voice, the frequency of switching speakers of the voice, and the appearance of keywords included in the character string from which the handwritten data is recognized. The importance information is determined by integrating at least two of the frequency and the frequency of appearance of the keyword included in the text data in which the voice is voice-recognized, and dividing the importance information into periods shorter than the recording time. The information processing system according to claim 1. The importance determining unit is configured to determine the writing frequency of stroke data constituting the handwritten data, the number of times the voice is uttered, the number of speakers of the voice, the frequency of switching speakers of the voice, and the characters in which the handwritten data is recognized. determining the importance information based on the frequency of occurrence of keywords included in the column and the frequency of occurrence of keywords included in the text data in which the voice has been voice-recognized, divided into periods shorter than the recording time;
The display information providing unit includes a writing frequency of stroke data constituting the handwritten data, a number of times the voice is uttered, a number of speakers of the voice, a frequency of switching speakers of the voice, and a character whose character is recognized in the handwritten data. The present invention is characterized by providing information for individually displaying the importance information based on the appearance frequency of the keyword included in the column and the appearance frequency of the keyword included in the text data in which the speech is voice-recognized. The information processing system according to claim 7. The importance determining unit is configured to determine the writing frequency of stroke data constituting the handwritten data, the number of times the voice is uttered, the number of speakers of the voice, the frequency of switching speakers of the voice, and the characters of the handwritten data that are recognized as characters. The importance information, which is integrated by giving weight to one or more of the appearance frequency of the keyword included in the column and the appearance frequency of the keyword included in the text data in which the voice is voice-recognized, is shorter than the recording time. 8. The information processing system according to claim 7, wherein the information processing system is determined in units of time. 10. The information processing system according to claim 7, wherein the keyword is specified by a viewer on the terminal device. The keyword is selected by the viewer on the terminal device from among the words included in the character string obtained by character recognition of the handwritten data or the text data obtained by voice recognition of the voice. The information processing system according to claim 10. The display information providing unit provides the importance information displayed along the recording time with words whose frequency of appearance for each time period is equal to or higher than a threshold value, and displays the words according to the viewer's operation. The information processing system according to claim 1, wherein the information processing system provides information for display. The display information providing unit associates a numerical value of the importance information with the importance information that is displayed along the recording time, and displays the numerical value that is equal to or higher than a threshold value in response to an operation by a viewer. 2. The information processing system according to claim 1, wherein the information processing system provides information on the following information. Identification information of conference participants is associated with the handwritten data and the voice,
The importance determining unit determines the importance level based on at least one of the writing frequency of stroke data included in the handwritten data associated with the participant, or the number of times the voice is uttered associated with the participant. 2. The information processing system according to claim 1, wherein the importance information is determined by dividing it into periods shorter than a meeting. The frequency of switching the speaker of the audio is determined based on at least one of the direction of the audio detected by the device or the direction of the speaker detected in the video imaged by the device. The information processing system according to claim 5. 2. The information processing system according to claim 1, wherein the display information providing unit provides the terminal device with information for displaying the size of the importance information in a gradation of a plurality of colors. The information processing according to claim 1, wherein the display information providing unit provides the terminal device with information for displaying the magnitude of the importance information in terms of brightness for any color. system. A display method performed by an information processing system that provides information recorded by a device capable of recording audio and video to a terminal device, the method comprising:
Based on at least one of recorded information including the audio or handwritten data associated with the recorded information, importance information regarding the recorded information is determined for each time shorter than a recording time of the recorded information. step and
providing the terminal device with information for displaying the importance information along the recording time of the recorded information;
A display method having An information processing system that provides information recorded by a device capable of recording audio and video to a terminal device.
Based on at least one of recorded information including the audio or handwritten data associated with the recorded information, importance information regarding the recorded information is determined for each time shorter than a recording time of the recorded information. An importance determining section,
a display information providing unit that provides the terminal device with information for displaying the importance information along the recording time of the recorded information;
A program to function as An information processing system that provides information recorded by a device capable of recording audio and video to a terminal device, and a recorded information creation system comprising the terminal device,
Based on at least one of recorded information including the audio or handwritten data associated with the recorded information, importance information regarding the recorded information is determined for each time shorter than a recording time of the recorded information. An importance determining section,
a display information providing unit that creates information for displaying the importance information along the recording time of the recorded information;
a display control unit that displays the importance information along the recording time of the recorded information;
A record information creation system with

Images