JP7148840B2 - Information processing system - Google Patents

Description

The present invention relates to an information processing system capable of reproducing data written to a page.

In recent years, the digitization of classes is progressing. For example, recording the lecturer's voice or taking a picture of the content displayed on the monitor in the classroom by the lecturer is recorded as recorded information, and at the timing when the page number of the content changes, the content and the recorded information A technique for creating and recording synchronization information has been proposed (see Patent Document 1, for example). With the technology described in Patent Document 1, it is possible to synchronize and reproduce the content and the recorded information via the network at the timing when the page number of the content changes.

JP 2013-223086 A

However, in the conventional technology, even if a student takes notes during class, the contents written in the notes and the recorded information are not recorded in association with each other, so the students can review the notes while reproducing the recorded information together with the content. Even so, there is a problem that it is not possible to understand what is written in the notebook, and it is not possible to understand the written contents of the notebook.

SUMMARY OF THE INVENTION Accordingly, it is an object of the present invention to provide an information processing system that facilitates understanding of content written in a page.

An information processing system of the present invention is an information processing system comprising a mobile terminal and a data management system connected to the mobile terminal via a network, wherein the mobile terminal is a display device for displaying pages of notes. input receiving means for receiving writing to the page displayed on the display device; application executing means for executing an application program; and displaying the page on the display device and enabling writing to the page. A notebook application realized by executing the notebook application program, which is the application program, by the application executing means, the written data received by the input receiving means, and the execution contents of the application program are each timed. record request means for requesting the data management system to record playback data including stamps; reproduction processing means for synchronously reproducing based on the time stamp, wherein the data management system includes data transmission means for transmitting the playback data recorded in the data management system to the outside of the information processing system. In addition, the reproduction processing means causes the display device to display the page based on the playback data stored outside the information processing system by the notebook application.

With this configuration, the information processing system of the present invention causes the data management system to record the playback data including the data to be written to the page and the execution contents of the application program in a state in which time stamps are added to each of them. Write data and execution contents included in the playback data obtained from the management system are played back synchronously based on the time stamp, so that the writing to the page can be reproduced along with the situation in which it was written. It is possible to facilitate understanding of the content. Further, the information processing system of the present invention transmits playback data recorded in the data management system to the outside of the information processing system, and creates a notebook page based on the playback data stored outside the information processing system. Playback data of other users that is stored outside of the information processing system as it is displayed on the display device by the app, even if the data management system does not allow access to the playback data of other users can be displayed on a display device by a note app.

In the information processing system of the present invention, the data management system comprises a database storing at least part of the playback data for each note, and a database storing at least part of the playback data stored in the database. and a storage unit for storing an entity, wherein the data transmission means transmits the playback data stored in the database without transmitting the entity stored in the storage unit to the outside of the information processing system. is transmitted to the outside of the information processing system, and the reproduction processing means causes the page based on the playback data stored outside the information processing system to be displayed on the display device by the notebook application. , the entity stored in the storage unit may be used.

With this configuration, the information processing system of the present invention can reproduce the play stored in the database of the data management system without transmitting the actual data stored in the storage unit of the data management system to the outside of the information processing system. When sending back data to the outside of the information processing system and displaying a page based on the playback data stored outside the information processing system on the display device by the notebook application, the data stored in the storage unit Since the entity of the data is used, even if the size of the entity of the data is large, the load on the external system of the information processing system can be suppressed.

The information processing system of the present invention can facilitate understanding of the content written to the page.

1 is a block diagram of an information processing system according to one embodiment of the present invention; FIG. 2 is a block diagram of the mobile terminal shown in FIG. 1; FIG. Figure 3 is a front view of the mobile terminal shown in Figure 2 in an open state; Figure 3 is a front view of the mobile terminal shown in Figure 2 in a closed state; 4 is a front view of the mobile terminal shown in FIG. 3 when the phone application and the notebook application are linked; FIG. 2 is a block diagram of the data management system shown in FIG. 1 when configured by one computer; FIG. 7 is a diagram showing the directory structure of the database shown in FIG. 6; FIG. 8 is a diagram showing an example of playback data shown in FIG. 7; FIG. Figure 3 is a flow chart of the operation of the mobile terminal shown in Figure 2 when writing to a page; 4 is a front view of the mobile terminal shown in FIG. 3 when document data or a web page is referenced when writing on a notebook page; FIG. FIG. 4 is a front view of the mobile terminal shown in FIG. 3 when document data is copied and pasted on a notebook page; FIG. 4 is a front view of the mobile terminal shown in FIG. 3 when writing content of another user's page is copied and pasted to a notebook page; FIG. 13 is a diagram showing the directory structure of the database when the contents written by another user shown in FIG. 12 are copied and pasted to the page of the notebook; 13 is a diagram showing playback data corresponding to the copy target area shown in FIG. 12; FIG. 13 is a diagram showing pages when the copy target area shown in FIG. 12 is divided across a plurality of pages and copied; FIG. FIG. 13 is a diagram showing a directory structure of a database when the copy target area shown in FIG. 12 is divided across a plurality of pages and copied; FIG. 13 is a diagram showing playback data when the copy target area shown in FIG. 12 is divided across a plurality of pages and copied; 3 is a flowchart of the operation of the mobile terminal shown in FIG. 2 when playing back playback data; 4 is a front view of the mobile terminal shown in FIG. 3 when reproducing playback data; FIG. 3 is a diagram showing playback data when a timer interrupt is generated during playback of playback data by the mobile terminal shown in FIG. 2; FIG. 2 is a diagram showing a directory structure of playback data in an online storage service external to the information processing system shown in FIG. 1; FIG.

BEST MODE FOR CARRYING OUT THE INVENTION Hereinafter, embodiments of the present invention will be described with reference to the drawings.

First, the configuration of an information processing system according to one embodiment of the present invention will be described.

FIG. 1 is a block diagram of an information processing system 10 according to this embodiment.

As shown in FIG. 1, the information processing system 10 includes mobile terminals 20 . The information processing system 10 can include at least one mobile terminal similar to the mobile terminal 20 in addition to the mobile terminal 20 . For example, the mobile terminals included in the information processing system 10 are used by students as notebooks.

The information processing system 10 includes a data management system 40 that manages data. The data management system 40 may be composed of one computer or may be composed of a plurality of computers. The data management system 40 may be a system that operates on the cloud.

Mobile terminals included in the information processing system 10 and the data management system 40 can communicate via a network 11 such as a LAN (Local Area Network) or the Internet.

FIG. 2 is a block diagram of the mobile terminal 20. As shown in FIG.

As shown in FIG. 2, the mobile terminal 20 includes an operation unit 21, which is an operation device such as a button for inputting various operations, and a display device such as an LCD (Liquid Crystal Display) for displaying various information. A communication device that communicates with an external device via a display unit 22, a microphone 23, a speaker 24, a camera 25, a network such as a LAN or the Internet, or directly by wire or wirelessly without a network. a communication unit 26 that stores various information, a storage unit 27 that is a non-volatile storage device such as a semiconductor memory or HDD (Hard Disk Drive), and a control unit 28 that controls the entire mobile terminal 20. ing.

FIG. 3 is a front view of mobile terminal 20 in an open state. FIG. 4 is a front view of mobile terminal 20 in a closed state.

As shown in FIGS. 2 to 4, the operation section 21 has a plurality of buttons 21a for receiving various operations.

The display unit 22 includes a first display unit 22a that forms a first touch panel together with part of the operation unit 21, and a second display unit 22b that forms a second touch panel together with part of the operation unit 21. ing. The first display section 22a is configured by, for example, a liquid crystal display, an organic EL display, or a plasma display. The second display unit 22b is made of electronic paper, for example. Note that not only the second display section 22b but also the first display section 22a may be made of electronic paper. Also, the sizes of the first display section 22a and the second display section 22b may each be A4 size, which is easy to use as a notebook.

The mobile terminal 20 includes a hinge portion 20a that rotatably connects the first display portion 22a and the second display portion 22b. In the mobile terminal 20, the screen of the first display unit 22a and the screen of the second display unit 22a are displayed as shown in FIG. A spread state in which the screens of 22b are adjacent to each other on the same plane, and a screen of the first display unit 22a (see FIG. 3) and a screen of the second display unit 22b (see FIG. 3) as shown in FIG. can be in a closed state in which the are closed facing each other.

The camera 25 includes a front camera 25a for photographing the front side of the mobile terminal 20 when the mobile terminal 20 is in a spread state, and a front camera 25a for photographing the rear side of the mobile terminal 20 when the mobile terminal 20 is in a spread state. and a rear camera 25b.

The storage unit 27 stores various application programs. For example, the storage unit 27 includes a notebook application program 27a, which is an application program for creating a notebook including a plurality of pages and enabling writing to the created notebook via the operation unit 21, Microsoft (registered trademark) Word, Microsoft A document application program 27b, which is an application program for creating, displaying, and editing documents such as Excel (registered trademark) and PDF; a browser application program 27c, which is an application program for displaying Web pages on the display unit 22; A telephone application program 27d, which is an application program for realizing telephone functions, is stored. The notebook application program 27a, the document application program 27b, the browser application program 27c, and the telephone application program 27d may be web applications.

The first display unit 22a shown in FIG. 3 displays an icon 31a associated with the document application program 27b, an icon 31b associated with the browser application program 27c, and an icon 31c associated with the telephone application program 27d. is displayed. The second display unit 22b shown in FIG. 3 displays an icon 31d associated with the note application program 27a for creating an English note and an icon 31d corresponding to the note application program 27a for creating a math note. An icon 31e attached thereto and an icon 31f associated with the note application program 27a for creating a note for memos are displayed.

The control unit 28 includes, for example, a CPU (Central Processing Unit), a ROM (Read Only Memory) storing programs and various data, and a RAM (Random Memory) as a memory used as a work area for the CPU of the control unit 28. Access Memory). The CPU of the control unit 28 executes programs stored in the storage unit 27 or the ROM of the control unit 28 .

The control unit 28 executes a program stored in the storage unit 27 or the ROM of the control unit 28, thereby providing an input reception unit 28a that receives input to the screen displayed on the display unit 22, and a user authentication unit that performs user authentication. The data management system requests recording of the playback data of the page 32 (see FIG. 5) displayed on the second display unit 22b by the authentication means 28b, the application execution means 28c for executing the application program, and the notebook application program 27a. 40 (see FIG. 1); A means 28e and a reproduction processing means 28f for reproducing the playback data acquired by the acquisition request means 28e are implemented.

The input reception unit 28a displays, on the page 32, points indicating position coordinates input from the operation unit 21 with respect to the page 32 displayed on the second display unit 22b by the notebook application program 27a, for example. That is, the trajectory of the position coordinates input from the operation unit 21 is drawn on the page 32 as handwritten data. For example, when one of the icons 31a to 31f shown in FIG. 3 is touched, the input reception unit 28a receives an operation to start an application program associated with the touched icon.

The user authentication means 28b displays an authentication screen on the display unit 22 for accepting input of authentication information. Then, the user authentication means 28b transmits the authentication information input to the authentication screen through the input reception means 28a to the data management system 40, and performs user authentication with the data management system 40. FIG. For example, the data management system 40 sends an authentication success notification indicating that the authentication is successful when there is authentication information in the pre-registered account information that matches the authentication information sent from the user authentication means 28b. It is transmitted to the user authentication means 28b. The user authentication means 28b logs in the user when an authentication success notification is sent from the data management system 40. FIG. For example, the authentication information is a combination of user ID and password.

When the input accepting unit 28a accepts the activation operation of the application program, the application executing unit 28c launches the application program whose activation operation has been accepted by the input accepting unit 28a. Application executing means 28c can execute a plurality of application programs in parallel.

The recording request unit 28d adds a time stamp to the write data written via the input reception unit 28a on the page 32 displayed on the second display unit 22b as playback data of the written page 32. , a recording request is transmitted to the data management system 40 in real time via the communication unit 26 . Further, the recording requesting means 28d executes the document application program 27b, the browser application program 27c, and the telephone application program 27d executed by the application executing means 28c when writing to the page 32 is accepted by the input accepting means 28a. The contents and the like are added as playback data of the written page 32 with a time stamp, and a recording request is transmitted to the data management system 40 in real time via the communication section 26 . The time stamp is information indicating the current time clocked by a clock unit (not shown). Also, the recording request means 28d can include audio data generated by the microphone 23, image data captured by the camera 25, etc. in the playback data.

When the page 32 whose playback data has been transmitted to the data management system 40 by the recording requesting unit 28d is displayed on the second display unit 22b, the acquisition requesting unit 28e requests acquisition of the playback data of this page 32. This playback data is received from the data management system 40 by sending it to the data management system 40 via the communication unit 26 .

The reproduction processing means 28f synchronizes and reproduces the write data and the execution content included in the playback data acquired by the acquisition request means 28e based on the time stamps added to each.

The control unit 28 realizes a notebook application 28g by executing the notebook application program 27a. The notebook application 28g accepts creation, display and editing of a new notebook including multiple pages via the second touch panel.

The control unit 28 realizes the document application 28h by executing the document application program 27b. The document application 28h accepts creation, display and editing of documents such as teaching material content via the first touch panel.

The control unit 28 realizes the browser application 28i by executing the browser application program 27c. The browser application 28i displays a website obtained from the Internet or the like via the communication unit 26 on the first display unit 22a.

The control unit 28 realizes a telephone application 28j by executing the telephone application program 27d. The phone application 28j controls voice calls with other users via the communication unit 26. FIG. The phone application 28j is capable of not only one-to-one calls but also group calls with a plurality of people.

The phone application 28j is linked with the notebook application 28g. When the application executing means 28c activates the telephone application program 27d, the telephone application 28j displays the address book 33 (see FIG. 5) for selecting a call destination on the first display section 22a. When starting the telephone application program 27d, the application executing means 28c also starts the notebook application program 27a. When the application executing means 28c activates the notebook application program 27a, the notebook application 28g displays a new writable page 32 (see FIG. 5) on the second display section 22b. Therefore, the caller can write a memo on the page 32 displayed on the second display section 22b via the operation section 21 when making a call.

The page 32 displayed on the second display unit of the mobile terminal of the caller is registered in the data management system 40 as a page shared between the callers and shared with the called party. When receiving an incoming call from the phone application of the mobile terminal of the calling party, the mobile terminal of the called party activates the phone application program by means of the application execution means, thereby enabling a video call with the caller and identifying the other party of the call. An image 34 (see FIG. 5) shown is displayed on the first display unit. Then, the application executing means of the mobile terminal on the receiving side activates the note application program to acquire the page 32 shared between the callers from the data management system 40 and displays it on the second display unit. Thus, the caller and the called party can share the writing via the page 32 displayed on the second display of the mobile terminal.

Note that when the input receiving means 28a receives writing on the page 32 displayed on the second display section 22b by the notebook application program 27a, a microphone is displayed on the second display section 22b as shown in FIG. A button 35 is displayed. The microphone button 35 is switched between ON and OFF each time it is touched via the operation unit 21 . While the microphone button 35 is ON, the microphone 23 generates voice data based on the input voice and records it. When the microphone button 35 is turned on while the call is not in progress, the voice data generated by the microphone 23 is acquired as microphone recording data by the recording request means 28d. When the microphone button 35 is turned on during a call, the voice data generated by the microphone 23 is acquired by the recording request means 28d as outgoing recorded data. When the recording requesting means 28d acquires the outgoing call recording data, the voice data of the called party receiving the call via the network 11 is acquired by the recording requesting means 28d as the incoming call recording data.

FIG. 6 is a block diagram of the data management system 40 when configured by one computer.

A data management system 40 shown in FIG. 6 includes an operation unit 41, which is an operation device such as a keyboard and a mouse, for inputting various operations, and a display unit 42, which is a display device such as an LCD for displaying various information. A communication unit 43, which is a communication device that communicates with an external device via a network such as , LAN, or the Internet, or directly by wire or wirelessly without a network, a semiconductor memory that stores various information, It has a storage unit 44 that is a non-volatile storage device such as an HDD, and a control unit 45 that controls the entire data management system 40 .

The data management system 40 may include, for example, an MQTT (Message Queue Telemetry Transport) server in order to share page writing among callers via a plurality of mobile terminals. The MQTT server accepts caller registration for each call, and notifies in real time the operation performed on the mobile terminal by one registered caller to the mobile terminals of other callers. Page writing can be shared between users. The MQTT server can share not only pages but also document data by document application programs and websites by browser application programs among callers.

The storage unit 44 may be configured by, for example, a database server. The storage unit 44 includes a database 44a and a large-capacity storage unit 44b.

FIG. 7 is a diagram showing the directory structure of the database 44a.

As shown in FIG. 7, the database 44a has a hierarchical structure in which directories of "data", "user", "notebook", and "page" are arranged in this order. A “user” is provided for each user so as to be identifiable by a user ID or the like. "Note" is provided so that each note created by the user can be identified by a note name or the like. A “page” is provided for each page constituting a notebook so that it can be identified by a page ID or the like.

Playback data 50 transmitted from a mobile terminal included in the information processing system 10 (see FIG. 1) is stored under "page" as page data. The playback data 50 includes microphone recording data 51 , incoming recording data 52 , outgoing recording data 53 , handwritten data 54 , copy and paste data 55 , date bookmark data 56 and application events 57 . The handwritten data 54 and the copy-and-paste data 55 are written contents drawn on the page. This is reference information. In other words, the database 44a stores together, for each page of each notebook owned by each user, the contents of writing on the page and the reference information used as a reference for writing on the page.

FIG. 8 is a diagram showing an example of playback data 50. As shown in FIG.

As shown in FIG. 8, microphone recording data 51, incoming call recording data 52, and outgoing call recording data 53 are recorded in an audio file format such as wav. The handwritten data 54 is recorded in a vector format such as SVG. The copy-and-paste data 55 is recorded including characters, images, URLs of link destinations, and the like. Date and time information is recorded in the date bookmark data 56 . The application event 57 records the execution content of the application.

Note that in FIG. 7, document data is stored under "user". When the playback data 50 includes an application event 57 of the document application 28h (see FIG. 2), this application event 57 includes the file name of the document data displayed by the document application 28h. Therefore, when the playback data 50 includes the application event 57 of the document application 28h, the document application 28h selects the document data to be displayed based on the file name of the document data included in the application event 57. can be obtained from and displayed.

The large-capacity storage unit 44b is a large-capacity storage unit such as an HDD, and stores various data such as WAV, XML, SVG, JSON, PDF, Microsoft Word, and Microsoft Excel. Large amounts of data, such as microphone recording data 51, incoming call recording data 52, outgoing call recording data 53, handwritten data 54, and document data, are stored in large-capacity storage unit 44b.

The control unit 45 includes, for example, a CPU, a ROM that stores programs and various data, and a RAM that serves as a work area for the CPU of the control unit 45 . The CPU of the control unit 45 executes programs stored in the storage unit 44 or the ROM of the control unit 45 .

By executing a program stored in the storage unit 44 or the ROM of the control unit 45, the control unit 45 connects a mobile terminal included in the information processing system 10 (see FIG. 1) to the network 11 (see FIG. 1). a recording request receiving means 45a for receiving a recording request for playback data transmitted via and storing the playback data in the storage unit 44; acquisition request response means 45b for receiving a transmitted playback data acquisition request, searching for this playback data in the storage unit 44, and returning the retrieved playback data to the mobile terminal via the network 11; Application processing means 45c for performing various application processes in response to requests from each application of the mobile terminal included in the system 10; A data transmission means 45d for transmission to the outside is realized. The recording request receiving means 45a, the acquisition request responding means 45b, the application processing means 45c, and the data transmitting means 45d may each be composed of a plurality of Web servers.

The application processing means 45c, for example, performs call control via a network such as a public line in response to a request from a phone application of a mobile terminal. The data management system 40 may include a WebRTC gateway, IP-PBX, and SIP server for performing telephone call control processing. The application processing unit 45c includes a notebook application server that provides the functions of a mobile terminal notebook application, a document application server that provides the functions of a mobile terminal document application, and a browser server that provides the functions of a mobile terminal browser application. and a phone server that provides the functions of a mobile terminal phone application.

Next, operations of the information processing system 10 will be described.

First, the operation of the information processing system 10 when the playback data 50 transmitted to the data management system 40 by the mobile terminal 20 is stored in the data management system 40 will be described.

When the page 32 is displayed on the second display unit 22b by the notebook application program 27a, the user logged in by the user authentication means 28b writes on the page 32 or reproduces the playback data 50, which will be described later. It can be selected via the operation part 21 whether to perform. When writing to the page 32 is selected, the recording request means 28d of the mobile terminal 20 starts the operation shown in FIG.

FIG. 9 is a flowchart of the operation of mobile terminal 20 when page 32 is being written.

As shown in FIG. 9, the recording request unit 28d of the mobile terminal 20 creates date bookmark data indicating the recording start date and time, adds a time stamp to the created date bookmark data, and returns the date bookmark data to which the time stamp has been added. A recording request is transmitted to the data management system 40 as the date bookmark data of the playback data of the page 32 (S101). Therefore, the recording request receiving means 45a of the data management system 40 receives the recording request transmitted from the mobile terminal 20 in S101, and transfers the date bookmark data, which is the target of the received recording request, to the playback data 50 of the page 32. It is stored under the page 32 in the storage unit 44 as at least part of the date bookmark data 56 .

When the process of S101 ends, the recording requesting means 28d of the mobile terminal 20 determines whether or not an application program other than the notebook application program 27a, that is, another application program has been executed by the application executing means 28c (S102).

When the record requesting means 28d determines in S102 that another application program such as the document application program 27b, the browser application program 27c, or the telephone application program 27d is executed by the application executing means 28c, the application executing means 28c The data management system 40 acquires an application event indicating the execution content of the executed application program, adds a time stamp to the acquired application event, and treats the time-stamped application event as the application event of the playback data of the page 32. (S103). Therefore, the recording request receiving means 45a of the data management system 40 receives the recording request transmitted from the mobile terminal 20 in S103, and converts the application event, which is the target of the received recording request, into the application event of the playback data 50 of the page 32. As at least a part of the event 57 , it is stored under the page 32 in the storage unit 44 . For example, as shown in FIG. 10, when a page 32 displayed on the second display unit 22b is being written, the document data 36 by the document application program 27b and the website by the browser application program 27c are displayed. 37 is displayed on the first display unit 22a, the recording request unit 28d of the mobile terminal 20 generates an application event indicating the execution details of the document application program 27b and an application event indicating the execution details of the browser application program 27c. It acquires, adds a time stamp to each of the acquired application events, and sends a recording request to the data management system 40 with each of the time-stamped application events as an application event of the playback data of the page 32 . Therefore, the recording request receiving unit 45 a of the data management system 40 can record the document data and website information that the user referred to when writing on the page 32 on the mobile terminal 20 in association with the page 32 .

The record requesting means 28d of the mobile terminal 20 determines in S102 that another application program is not being executed by the application executing means 28c, or when the process of S103 ends, the handwritten input is made on the page 32 by the input receiving means 28a. (S104).

When the record requesting means 28d determines in S104 that the input receiving means 28a has handwritten input on the page 32, the recording requesting means 28d obtains the handwritten data written on the page 32 by the input receiving means 28a, and adds a time stamp to the obtained handwritten data. Then, a recording request is transmitted to the data management system 40 with the handwritten data with the timestamp added as the handwritten data of the playback data of the page 32 (S105). Therefore, the recording request receiving means 45a of the data management system 40 receives the recording request transmitted from the mobile terminal 20 in S105, and converts the handwritten data, which is the target of the received recording request, into the handwritten playback data 50 of the page 32. It is stored under the page 32 in the storage unit 44 as at least part of the data 54 .

The record requesting means 28d of the mobile terminal 20 determines in S104 that no handwritten input has been made to the page 32 by the input receiving means 28a, or when the process of S105 ends, the microphone button 35 is turned on by the input receiving means 28a. (S106).

When the recording requesting means 28d determines in S106 that the microphone button 35 has been turned on by the input receiving means 28a, it acquires any one of the microphone recording data, the incoming call recording data, and the outgoing call recording data according to the call situation, A time stamp is added to the acquired recorded data, and a record request is sent to the data management system 40 with the recorded data with the time stamp added as recorded data of the playback data of page 32 (S107). Therefore, the recording request receiving means 45a of the data management system 40 receives the recording request transmitted from the mobile terminal 20 in S107, and transfers the recorded data, which is the target of the received recording request, to the recording of the playback data 50 of the page 32. As at least part of the data, it is stored under the page 32 in the storage unit 44 . Here, if the recorded data that is the target of the recording request sent from the mobile terminal 20 in S107 is microphone recorded data, the recording request receiving means 45a converts this microphone recorded data to the microphone recording of the playback data 50 of the page 32. Stored as at least part of the data 51 . In addition, when the recording data that is the object of the recording request sent from the mobile terminal 20 in S107 is incoming call recording data, the recording request receiving means 45a converts this incoming call recording data into the incoming call recording data of the playback data 50 of the page 32. Store as at least part of 52. In addition, when the recorded data that is the object of the recording request sent from the mobile terminal 20 in S107 is the outgoing recorded data, the recording request receiving means 45a converts the outgoing recorded data into the outgoing recorded data of the playback data 50 of the page 32. Store as at least part of 53. Therefore, for example, when the note application program 27a of the mobile terminal 20 accepts writing on the page 32 displayed on the second display unit 22b by the input acceptance unit 28a, the voice of the lecturer or the like recorded by the mobile terminal 20 and call audio can be recorded in association with page 32 in data management system 40 .

The record requesting means 28d of the mobile terminal 20 determines in S106 that the microphone button 35 is not turned ON by the input receiving means 28a, or when the process of S107 is completed, the input receiving means 28a copies and It is determined whether or not pasting has been performed (S108). For example, as shown in FIG. 11, the user selects the copy target area 38 from the document data 36 or the website 37 via the operation unit 21, and the characters, images, and link information in the selected copy target area 38 are displayed. can be copied and pasted to a specified location within the page 32 for copy and paste. In addition, the user can also copy and paste the content written on the page of the notebook created by another user to the page 32 to be written by himself/herself.

When the recording requesting means 28d determines in S108 that the input receiving means 28a has copied and pasted the page 32, the recording requesting means 28d acquires the copy-and-paste data written on the page 32 by the input receiving means 28a, and performs the acquired copy-and-paste data. A time stamp is added to the paste data, and a record request is sent to the data management system 40 with the time stamped copy-and-paste data as the copy-and-paste data of the playback data of the page 32 (S109). Therefore, the recording request receiving means 45a of the data management system 40 receives the recording request transmitted from the mobile terminal 20 in S109, and transfers the copy-and-paste data, which is the target of the received recording request, to the playback data 50 of the page 32. is stored under the page 32 in the storage unit 44 as at least part of the copy-and-paste data 55 of .

The recording request unit 28d of the mobile terminal 20 determines in S108 that the input reception unit 28a has not copied and pasted the page 32, or when the process of S109 ends, the input reception unit 28a does not accept writing. It is determined whether or not the process has ended (S110). For example, when an operation to close a notebook is performed via the operation unit 21 or an operation to display the next page is performed via the operation unit 21, the input reception unit 28a writes to the displayed page 32. end the acceptance of

If the recording request unit 28d of the mobile terminal 20 determines in S110 that the input reception unit 28a has not finished accepting the writing, it executes the processing of S102. That is, the recording requesting means 28d makes a recording request for the playback data of the page 32 while the writing to the page 32 is being accepted by the input accepting means 28a.

When the recording request unit 28d of the mobile terminal 20 determines in S110 that the input reception unit 28a has finished accepting the writing, it creates date bookmark data indicating the recording end date and time, and adds a time stamp to the created date bookmark data. , a recording request is transmitted to the data management system 40 with the date bookmark data with the time stamp added as the date bookmark data of the playback data of the page 32 (S111). Therefore, the recording request receiving means 45a of the data management system 40 receives the recording request transmitted from the mobile terminal 20 in S111, and converts the date bookmark data, which is the target of the received recording request, into the playback data 50 of the page 32. It is stored under the page 32 in the storage unit 44 as at least part of the date bookmark data 56 .

The recording request unit 28d of the mobile terminal 20 ends the operation shown in FIG. 9 when the process of S111 ends.

Next, the operation of the information processing system 10 will be described in detail when a user copies and pastes written content on a page of a notebook created by another user to the page 32 to be written by the user.

As shown in FIG. 12, a user (hereafter referred to as "user A") displays a notebook page created by another user (hereafter referred to as "user B") on the second display of user A's own mobile terminal 20. It is also possible to display it on the part 22b and copy and paste the writing contents of the user B's page to the writing target page 32 of the user A himself. In this case, user A can display a page permitted by user B on his own mobile terminal 20 by inputting a preset authentication key. User B communicates with the data management system 40 in advance via the notebook application 28g of his own mobile terminal, and sets an authentication key such as a password for the page that User A is permitted to display. good.

When the recording request unit 28d of the user A's mobile terminal 20 transmits a request to record the copy-and-paste data of the content written in the user B's page to the data management system 40 in S109, the recording request reception unit 45a of the data management system 40 , as shown in FIG. 13, the copy-and-paste data including the playback data of the user B corresponding to the copy target area 38 is stored as at least part of the copy-and-paste data 55 of the playback data 50 of the page 32. It is stored under the page 32 in the unit 44 . That is, not only the written data (handwritten data 54 and copy-and-paste data 55) in the copy target area 38, but also the data written by user B corresponding to the first time stamp to the last time stamp in the written data in the copy target area 38. All playback data are copied.

For example, in FIG. 14, user B's playback data corresponding to copy target area 38 includes microphone recording data, handwritten data, copy-and-paste data, and application events. Therefore, the written data in the copy target area 38 copied by the user A from the page of the user B is pasted on the page 32 of the user A with the information such as the document data and the website that the user B referred to in writing in the backbone. attached.

As shown in FIG. 15, when the copy target area 38 copied from the user B's page is pasted onto the user A's page 32, the size of the copy target area 38 is the remaining writable area (paste area) of the page 32. If the copy target area 38 is larger than the size of the pastable area), the copy target area 38 is pasted across the page 32 and the page next to the page 32 . In the example shown in FIG. 15 , the copy target area 38 is divided into a copy target area 38 a pasted on the page 32 and a copy target area 38 b pasted on the next page of the page 32 . Therefore, first, the recording request means 28d of the mobile terminal 20 of the user A adds a time stamp to the copy and paste data based on the copy target area 38a, and converts the time stamped copy and paste data to the playback data of the page 32. A recording request is transmitted to the data management system 40 as copy-and-paste data (S109). Next, the recording requesting means 28d determines that writing has ended because there is no more writable area on the page 32, creates date bookmark data indicating the recording end date and time, adds a time stamp to the created date bookmark data, and adds a time stamp to the created date bookmark data. A recording request is transmitted to the data management system 40 with the date bookmark data with the stamp added as the date bookmark data of the playback data of the page 32 (S111). Next, in order to paste the copy target area 38b, the page next to the page 32 is displayed by the notebook application program 27a, and the entry is accepted by the input accepting means 28a. Next, the recording request means 28d of the mobile terminal 20 of the user A creates date bookmark data indicating the recording start date and time, adds a time stamp to the created date bookmark data, and sends the date bookmark data with the time stamp to the page 32. A recording request is transmitted to the data management system 40 as the date bookmark data of the playback data of the next page (S101). Next, since the copy target area 38b is pasted to the next page of the page 32, the recording request means 28d adds a time stamp to the copy and paste data based on the copy target area 38b, and adds the time stamp to the copy and paste data. A recording request is transmitted to the data management system 40 with the paste data as copy and paste data of the playback data of the next page of the page 32 (S109). Therefore, as shown in FIGS. 16 and 17, the recording request receiving means 45a of the data management system 40 stores the user's The playback data corresponding to the copy target area 38a of user B and the playback data corresponding to the copy target area 38b of user B are copied.

Note that if user B, who is the copy source, has created the writing in the copy target area 38 by including write data copied from the page of user C, the copy target area 38 is the playback data of user B. , and user C's playback data. In this case, when the recording request receiving means 45a of the data management system 40 receives a request to record copy-and-paste data as playback data from the user A, the playback data of a plurality of users corresponding to the copy target area 38 are stored. In the section 44, it is copied as copy-and-paste data 55 under the user A's page.

Next, the operation of the information processing system 10 when the playback data 50 stored in the data management system 40 is reproduced by the mobile terminal 20 will be described.

A user who has logged in using the user authentication means 28b can input an operation to open a notebook via the operation unit 21 of the mobile terminal 20. FIG. When the input reception unit 28a of the mobile terminal 20 receives an operation to open the notebook, the acquisition request unit 28e of the mobile terminal 20 starts the operation shown in FIG.

FIG. 18 is a flow chart of the operation of the mobile terminal 20 when playing back the playback data 50. FIG.

As shown in FIG. 18, the acquisition requesting means 28e of the mobile terminal 20 sends an acquisition request for the playback data of the page last displayed on the second display unit 22b, that is, the last updated page, which is the most recent page. It is transmitted to the management system 40 (S131). In the case of a notebook that has never been opened, such as a new notebook, the acquisition request unit 28e sets the first page as the last updated page. The acquisition request response unit 45b of the data management system 40 receives the acquisition request transmitted from the mobile terminal 20 in S131, and searches the storage unit 44 for the playback data of the page that is the target of the received acquisition request. Here, the acquisition request response unit 45b may specify the page storing the latest date and time among the date bookmark data and time stamps stored in the storage unit 44 as the last updated page. When the acquisition request response unit 45b of the data management system 40 identifies the playback data of the page that is the object of the acquisition request, the date bookmark data, application event, and handwritten data stored in the directory of the last updated page from the storage unit 44 are retrieved. Data, recorded data, copy-and-paste data, etc. are acquired as playback data, and the playback data of the acquired page is transmitted to the mobile terminal 20 . It should be noted that in the case of a new page for which writing has never been accepted, playback data is not stored in the directory of this page. Therefore, the acquisition request response unit 45b of the data management system 40 transmits empty information (for example, NULL) as playback data to the mobile terminal 20 when an acquisition request for a new page is made.

The acquisition request unit 28e of the mobile terminal 20 continues until the reception of the playback data from the data management system 40 is completed until it determines that the reception of the playback data from the data management system 40 is completed after the processing of S131 is completed. (S132).

When it is determined in S132 that the reception of the playback data from the data management system 40 is completed, the reproduction processing means 28f of the mobile terminal 20 reproduces the playback data determined in S132 that the reception of the playback data from the data management system 40 is completed. The page in which the handwritten data and the copy-and-paste data contained in . Here, if the playback data determined in S132 that the reception from the data management system 40 has been completed is empty information, the playback processing means 28f displays only pages in which nothing is written in the second display. It is displayed on the part 22b.

When the process of S133 is completed, the reproduction processing means 28f of the mobile terminal 20 reproduces data other than the notebook application program 27a based on the application event included in the playback data determined in S132 that the reception from the data management system 40 is completed. The last screen displayed by another application program is displayed on the display unit 22 (S134). That is, the reproduction processing unit 28f displays on the display unit 22 the screen of the other application program that was displayed on the display unit 22 when the writing to the last update page was completed last time. For example, the document data and website that were displayed on the display unit 22 when writing to the final update page was completed last time are displayed on the first display unit 22a.

After completing the process of S134, the reproduction processing means 28f of the mobile terminal 20 displays the reproduction bar 39 shown in FIG. 19 for reproducing the playback data on the second display section 22b (S135). The reproduction bar 39 is a reproducible area 39a indicating the reproduction time length of the playback data, and a cursor indicating the reproduction position within the reproducible area 39a. A playback cursor 39b that can be slid to an arbitrary position, a playback start button 39c for playing back playback data from a time corresponding to the playback position of the playback cursor 39b, and a playback stop button for stopping playback of the playback data. button 39d.

When the processing of S135 is completed, the reproduction processing means 28f of the mobile terminal 20 continues to check whether the reproduction bar 39 has received the reproduction operation by the reproduction start button 39c until it determines that the reproduction bar 39 has received the reproduction operation by the reproduction start button 39c. (S136).

When the reproduction processing means 28f of the mobile terminal 20 determines in S136 that the reproduction bar 39 has received the reproduction operation by the reproduction start button 39c, the reproduction processing means 28f is based on the reproduction data after the time stamp corresponding to the reproduction position indicated by the reproduction cursor 39b. The display is temporarily turned off, and the playback data are reproduced sequentially from the time stamp corresponding to the reproduction position indicated by the reproduction cursor 39b (S137). That is, the reproduction processing means 28f has a time stamp indicating the closest time after the time indicated by the reproduction cursor 39b in the playback data determined in S132 that reception from the data management system 40 has been completed. Data to the data having the time stamp of the final time are reproduced in chronological order. Note that when any of the written data in the page displayed on the second display section 22b is touched, the reproduction processing means 28f changes the time stamp of the touched written data to the reproduction position. You may reproduce playback data as .

When playing back the playback data in S137, the playback processing means 28f of the mobile terminal 20 may sequentially acquire the playback data to be played back from the data management system 40 via the acquisition requesting means 28e and perform streaming playback. good. For example, as shown in FIG. 20, the reproduction processing means 28f generates a timer interrupt every specific time such as every second during reproduction of the playback data, and starts the next timer interrupt from the time corresponding to the generated timer interrupt. The acquisition request means 28e requests the data management system 40 to acquire the playback data having the corresponding time stamp within a specific time such as the time until the timer interrupt generated in the data management system 40. Obtained from system 40 . When the playback data is acquired from the data management system 40, the playback processing means 28f passes the acquired data of the corresponding period to at least one or more of the note application 28g, the document application 28h, the browser application 28i and the telephone application 28j for synchronization. to play. When the mobile terminal 20 sequentially acquires playback data from the data management system 40 and performs streaming playback, it is possible to reduce the amount of playback data received when displaying the last updated page on the second display unit 22b. Therefore, the time to display the last update page can be shortened.

In the above description, reproduction of playback data for the final update page has been described, but the same applies to reproduction of playback data for pages other than the final update page. For example, after displaying the last updated page on the second display unit 22b, the reproduction processing unit 28f of the mobile terminal 20 performs an operation to display the next page or an operation to display the previous page via the operation unit 21. When input, a playback data acquisition request for a new page to be displayed is transmitted to the data management system 40, and the page to be displayed is displayed on the second display unit based on the playback data acquired from the data management system 40. 22b, and the playback data for this page is reproduced according to the operation via the operation unit 21. FIG.

Next, the operation of the data management system 40 when transmitting playback data to the outside of the information processing system 10 will be described.

When an instruction to transmit playback data to a specific location outside the information processing system 10 is input via the operation unit 41 or the communication unit 43, the data transmission unit 45d of the data management system 40 reproduces the instructed playback data. Send the data to the indicated location. For example, the data transmission means 45d can download playback data to any PC (Personal Computer). The user can upload the playback data downloaded from the data management system 40 to an online storage service such as Google Drive, Dropbox, or the like. It should be noted that the user can set the online storage service to share the playback data stored in the online storage service external to the information processing system 10 with other users.

FIG. 21 is a diagram showing the directory structure of playback data in an online storage service external to the information processing system 10. As shown in FIG.

In the directory structure shown in FIG. 21, playback data of three notes "note1", "note2" and "note3" are included. 'note1' is a note containing three pages: 'page1', 'page2' and 'page3'.

The playback processing means 28f of the mobile terminal 20 can display pages based on the playback data stored in the data management system 40 on the second display section 22b by the note application program 27a, as described above. Similarly, the reproduction processing means 28f can display a page based on the playback data stored outside the information processing system 10 on the second display section 22b by the notebook application program 27a. For example, the reproduction processing means 28f can cause the second display section 22b to display pages based on playback data stored in an online storage service external to the information processing system 10 by means of the notebook application program 27a. Here, even if the playback data stored in the online storage service external to the information processing system 10 belongs to user B, the setting for sharing this playback data with user A is set for this online storage service. If it is done, the playback processing means 28f of the mobile terminal 20 logged in by the user A causes the page based on this playback data to be displayed on the second display unit 22b by the note application program 27a. can be done.

As described above, the mobile terminal 20 can copy and paste the written content of the page based on the playback data stored in the data management system 40 to another page. Similarly, the mobile terminal 20 can copy and paste the written contents of the page based on the playback data stored outside the information processing system 10 to another page.

The mobile terminal 20 can reproduce the playback data stored in the data management system 40 as described above. Similarly, the mobile terminal 20 can reproduce playback data stored outside the information processing system 10 .

As described above, the information processing system 10 time-stamps document data, websites, voice data from the microphone 23, and the like that were referred to when the page displayed on the mobile terminal 20 was written. It is added and recorded as playback data in association with the page together with the write data. When the written page is to be displayed, the information processing system 10 can acquire the playback data of the page and reproduce each data included in the acquired playback data in synchronization with each other based on the time stamp. . Therefore, the information processing system 10 can reproduce the writing on the notebook page together with the writing situation, and can allow the user to easily understand the contents of writing on the notebook page.

The information processing system 10 transmits the playback data recorded in the data management system 40 to the outside of the information processing system 10, and the page based on the playback data stored outside the information processing system 10 is displayed in the notebook application. 28g to display on the second display unit 22b, even if the data management system 40 does not permit access to the playback data of other users, the data stored outside the information processing system 10 can be displayed. A page based on the user's playback data can be displayed on the second display unit 22b by the note application 28g. It should be noted that, in the present embodiment, the information processing system 10, of the playback data recorded in the data management system 40, transfers the playback data of users other than the logged-in user to the mobile terminal 20 from the mobile terminal 20. Authorize access by authentication key. However, the information processing system 10 prohibits the mobile terminal 20 from accessing the playback data of users other than the logged-in user of the mobile terminal 20 among the playback data recorded in the data management system 40. Thus, security in the data management system 40 can be improved.

When transmitting the playback data stored in the data management system 40 to the outside of the information processing system 10, the data management system 40 transfers the actual data stored in the large-capacity storage unit 44b to the information processing system. The playback data stored in the database 44 a may be transmitted to the outside of the information processing system 10 instead of being transmitted to the outside of the information processing system 10 . If the actual data stored in the large-capacity storage unit 44b is not transmitted outside the information processing system 10, the playback processing means 28f of the mobile terminal 20 reproduces the playback data stored outside the information processing system 10. is displayed on the second display unit 22b by the notebook application 28g, the substance of the data stored in the large-capacity storage unit 44b is used. The information processing system 10 stores the actual data stored in the large-capacity storage unit 44b of the data management system 40 in the database 44a of the data management system 40 without transmitting the data to the outside of the information processing system 10. When the playback data is transmitted to the outside of the information processing system 10 and a page based on the playback data stored outside the information processing system 10 is displayed on the second display unit 22b by the notebook application 28g, Since the actual data stored in the capacity storage unit 44b is used, the load on the external system of the information processing system 10 can be suppressed even if the size of the actual data is large.

10 information processing system 11 network 20 mobile terminal 22 display unit (display device)
22a first display unit 22b second display unit 27a notebook application program (application program)
27b document application program (application program)
27c browser application program (application program)
27d Telephone application program (application program)
28a Input reception means 28c Application execution means 28d Recording request means 28f Reproduction processing means 28g Note application 32 Page 40 Data management system 44a Database 44b Mass storage unit (storage unit)
45d data transmission means 50 playback data 54 handwritten data (written data)
55 Copy and paste data (write data)
57 Application event (application execution content)

Claims (2)

a mobile device and
and a data management system connected to the mobile terminal via a network, wherein
The mobile terminal is
a display device for displaying a page of notes;
input receiving means for receiving writing to the page displayed on the display device;
an application execution means for executing an application program;
a notebook application realized by executing the notebook application program, which is the application program for displaying the page on the display device and enabling writing to the page, by the application executing means;
The write data received by the input receiving means is included in a state to which a time stamp indicating the time corresponding to when the write data was received by the input receiving means is added, and the execution contents of the application program are included in this recording request means for requesting the data management system to record the playback data including the execution content with a time stamp indicating the time corresponding to when the application execution means was executed ;
playback processing means for synchronizing and playing back the written data and the execution content contained in the playback data acquired from the data management system based on the time stamps added thereto ,
The data management system comprises data transmission means for transmitting the playback data recorded in the data management system to the outside of the information processing system,
The information processing system, wherein the reproduction processing means displays the page based on the playback data stored outside the information processing system on the display device by the notebook application. The data management system includes:
a database storing at least part of the playback data for each note;
a storage unit that stores the entity of at least part of the playback data stored in the database;
The data transmission means transmits the playback data stored in the database to the outside of the information processing system without transmitting the entity stored in the storage unit to the outside of the information processing system. ,
The playback processing means, when displaying the page based on the playback data stored outside the information processing system on the display device by the note application, reproduces the content of the entity stored in the storage unit. 2. The information processing system according to claim 1, wherein the information processing system uses:

Images