JP2020027327A - Information processing system - Google Patents

Abstract

To provide an information processing system that facilitates understanding of the details of writing in a page.SOLUTION: In an information processing system, a mobile terminal requests a data management system to record playback data including data of writing in a page displayed on a second display unit by a note application and the details of execution of an application program, with time stamps respectively added thereto (S101 to S111), and synchronizes and plays back, on the basis of the time stamps, the writing data and the details of execution included in the playback data acquired from the data management system. The data management system transmits the playback data recorded in the data management system to the outside of the information processing system, and the mobile terminal displays, with the note application, a page based on the playback data stored in the outside of the information processing system on the second display unit.SELECTED DRAWING: Figure 9

Description

  The present invention relates to an information processing system capable of reproducing data written to a page.

  In recent years, digitization of classes has been promoted. For example, while recording the voice of the instructor or recording the content that the instructor displayed on the monitor of the classroom as recording information, at the timing when the page number of the content changes, the content and the recording information A technique for creating and recording the synchronization information has been proposed (for example, see Patent Document 1). According to the technology described in Patent Literature 1, it is possible to reproduce the content and the recorded information in synchronization with the timing at which the page number of the content changes via a network.

JP 2013-223086 A

  However, in the conventional technology, even if a student takes a note during class, the content written in the note and the recorded information are not recorded in association with each other, so the student reviews the note while playing the recorded information together with the content. However, there is a problem in that it is not possible to understand the writing content of the note without knowing the writing base of the note.

  Therefore, an object of the present invention is to provide an information processing system capable of facilitating the understanding of written contents on a page.

  An information processing system according to the present invention is an information processing system including a mobile terminal and a data management system connected to the mobile terminal via a network, wherein the mobile terminal is a display device that displays a page of a notebook. Input receiving means for receiving writing to the page displayed on the display device, application executing means for executing an application program, and displaying the page on the display device and enabling writing to the page A note application realized by executing the note application program, which is the application program, by the application executing means, write data received by the input receiving means, and execution contents of the application program Recording request means for requesting the data management system to record playback data including a time stamp added thereto, and the write data and the write data included in the playback data acquired from the data management system. Reproduction processing means for synchronously reproducing the execution contents based on the time stamp, wherein the data management system transmits the playback data recorded in the data management system to outside the information processing system Data reproduction means, wherein the reproduction processing means displays the page based on the playback data stored outside the information processing system on the display device by the note application.

  With this configuration, the information processing system of the present invention causes the data management system to record the playback data including the data written to the page and the execution content of the application program in a state where a time stamp is added to each of the data. Since the write data and execution contents included in the playback data acquired from the management system are reproduced synchronously based on the time stamp, the writing to the page can be reproduced together with the writing status, and the writing to the page It is easy to understand the contents. Further, the information processing system of the present invention transmits playback data recorded in the data management system to the outside of the information processing system, and notes a page based on the playback data stored outside the information processing system. Displayed on the display device by the app, even if the data management system is not allowed to access the playback data of other users, the playback data of other users stored outside the information processing system The page based on the can be displayed on the display device by the note application.

  In the information processing system according to the present invention, the data management system may include a database that stores at least a part of the playback data for each note, and a database of at least a part of the playback data that is stored in the database. A storage unit for storing an entity, wherein the data transmission unit transmits the entity stored in the database without transmitting the entity stored in the storage unit to the outside of the information processing system. Is transmitted to the outside of the information processing system, and the reproduction processing unit causes the note application to display the page based on the playback data stored outside the information processing system on the display device. Alternatively, the entity stored in the storage unit may be used.

  With this configuration, the information processing system of the present invention does not transmit the substance of the data stored in the storage unit of the data management system to the outside of the information processing system but stores the data stored in the database of the data management system. When the back data is transmitted to the outside of the information processing system and a page based on the playback data stored outside the information processing system is displayed on the display device by the note application, the data stored in the storage unit. Therefore, even when the size of the data entity is large, it is possible to reduce the load on a system external to the information processing system.

  The information processing system according to the present invention can facilitate understanding of the contents written to a page.

1 is a block diagram of an information processing system according to one embodiment of the present invention. FIG. 2 is a block diagram of the mobile terminal shown in FIG. 1. FIG. 3 is a front view of the mobile terminal shown in FIG. 2 in an open state. FIG. 3 is a front view of the mobile terminal shown in FIG. 2 in a closed state. FIG. 4 is a front view of the mobile terminal shown in FIG. 3 when a telephone application and a note application cooperate. FIG. 2 is a block diagram of the data management system shown in FIG. 1 when configured by one computer. FIG. 7 is a diagram illustrating a directory configuration of a database illustrated in FIG. 6. FIG. 8 is a diagram showing an example of the playback data shown in FIG. 7. 3 is a flowchart of an operation of the mobile terminal shown in FIG. 2 when writing is performed on a page. FIG. 4 is a front view of the mobile terminal shown in FIG. 3 when document data or a Web page is referred to when writing a notebook page. FIG. 4 is a front view of the mobile terminal shown in FIG. 3 when document data is copied and pasted on a notebook page. FIG. 4 is a front view of the mobile terminal shown in FIG. 3 in a case where written contents of another user's page are copied and pasted on a notebook page. FIG. 13 is a diagram illustrating a directory configuration of a database in a case where contents written by another user illustrated in FIG. 12 are copied and pasted on a page of a note. FIG. 13 is a diagram showing playback data corresponding to the copy target area shown in FIG. 12. FIG. 13 is a diagram illustrating a page when the copy target area illustrated in FIG. 12 is divided and copied across a plurality of pages. FIG. 13 is a diagram illustrating a directory configuration of a database when the copy target area illustrated in FIG. 12 is divided and copied across a plurality of pages. FIG. 13 is a diagram illustrating playback data when the copy target area illustrated in FIG. 12 is divided and copied across a plurality of pages. 3 is a flowchart of an operation of the mobile terminal shown in FIG. 2 when playing back playback data. FIG. 4 is a front view of the mobile terminal shown in FIG. 3 when playing back playback data. FIG. 3 is a diagram showing playback data when a timer interrupt is generated during playback of playback data by the mobile terminal shown in FIG. 2. FIG. 2 is a diagram illustrating a directory configuration of playback data in an online storage service external to the information processing system illustrated in FIG. 1.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

  First, the configuration of an information processing system according to an embodiment of the present invention will be described.

  FIG. 1 is a block diagram of an information processing system 10 according to the present embodiment.

  As shown in FIG. 1, the information processing system 10 includes a mobile terminal 20. The information processing system 10 can include at least one mobile terminal similar to the mobile terminal 20 in addition to the mobile terminal 20. For example, each of the mobile terminals included in the information processing system 10 is used as a notebook by a student.

  The information processing system 10 includes a data management system 40 that manages data. The data management system 40 may be configured by one computer, or may be configured by a plurality of computers. The data management system 40 may be a system that operates on a cloud.

  The mobile terminal included in the information processing system 10 and the data management system 40 can communicate with each other via a network 11 such as a LAN (Local Area Network) or the Internet.

  FIG. 2 is a block diagram of the mobile terminal 20.

  As shown in FIG. 2, the mobile terminal 20 includes an operation unit 21 which is an operation device such as a button to which various operations are input, and a display device such as an LCD (Liquid Crystal Display) which displays various information. A communication device that communicates with a certain display unit 22, a microphone 23, a speaker 24, a camera 25, and an external device via a network such as a LAN or the Internet, or directly by wire or wirelessly without using a network. A communication unit 26 that stores various information; a storage unit 27 that is a non-volatile storage device such as a semiconductor memory and an HDD (Hard Disk Drive); and a control unit 28 that controls the entire mobile terminal 20. ing.

  FIG. 3 is a front view of the mobile terminal 20 in an open state. FIG. 4 is a front view of the mobile terminal 20 in a closed state.

  As shown in FIGS. 2 to 4, the operation unit 21 includes a plurality of buttons 21 a for receiving various operations.

  The display unit 22 includes a first display unit 22a that forms a first touch panel together with a part of the operation unit 21, and a second display unit 22b that forms a second touch panel together with a part of the operation unit 21. ing. The first display unit 22a includes, for example, a liquid crystal display, an organic EL display, and a plasma display. The second display unit 22b is made of, for example, electronic paper. Note that not only the second display unit 22b but also the first display unit 22a may be made of electronic paper. Further, the size of the first display unit 22a and the size of the second display unit 22b may be A4 size which is easy to use as a notebook.

  The mobile terminal 20 includes a hinge portion 20a that rotatably connects the first display portion 22a and the second display portion 22b. The mobile terminal 20 rotates the first display unit 22a and the second display unit 22b with each other by the hinge unit 20a, and thereby the screen of the first display unit 22a and the second display unit as shown in FIG. The two-page spread state in which the screens 22b are adjacent to each other on the same surface, and the screens of the first display unit 22a (see FIG. 3) and the second display unit 22b (see FIG. 3) as shown in FIG. Can be closed to face to face.

  The camera 25 includes a front camera 25a for photographing the front side of the mobile terminal 20 when the mobile terminal 20 is in the facing state, and a camera 25a for photographing the back side of the mobile terminal 20 when the mobile terminal 20 is in the facing state. Rear camera 25b.

  The storage unit 27 stores various application programs. For example, the storage unit 27 creates a note including a plurality of pages, and writes a note to the created note via the operation unit 21. The note application program 27a is an application program, and Microsoft (registered trademark) Word, Microsoft A document application program 27b that is an application program for creating, displaying, and editing documents such as Excel (registered trademark) and PDF; a browser application program 27c that is an application program for displaying a Web page on the display unit 22; A telephone application program 27d which is an application program for realizing a telephone function is stored. The note application program 27a, document application program 27b, browser application program 27c, and telephone application program 27d may be Web applications.

  The first display unit 22a shown in FIG. 3 includes an icon 31a associated with the document application program 27b, an icon 31b associated with the browser application program 27c, and an icon 31c associated with the telephone application program 27d. And are displayed. The second display unit 22b shown in FIG. 3 has an icon 31d associated with the note application program 27a for creating a note for English and a note application program 27a for creating a note for mathematics. An attached icon 31e and an icon 31f associated with the note application program 27a for creating a memo note are displayed.

  The control unit 28 includes, for example, a CPU (Central Processing Unit), a ROM (Read Only Memory) storing programs and various data, and a RAM (Random) as a memory used as a work area of the CPU of the control unit 28. Access Memory). The CPU of the control unit 28 executes a program stored in the storage unit 27 or the ROM of the control unit 28.

  The control unit 28 executes, by executing a program stored in the storage unit 27 or the ROM of the control unit 28, an input receiving unit 28a that receives an input to the screen displayed on the display unit 22, and a user that executes user authentication. The authentication unit 28b, the application execution unit 28c for executing the application program, and the note management program 27a send the recording request of the playback data of the page 32 (see FIG. 5) displayed on the second display unit 22b to the data management system. Recording request means 28d for transmitting to the data management system 40 a request to acquire the playback data of the page 32 displayed on the second display unit 22b by the note application program 27a. Means 28e and acquisition request means 28 Play playback data obtained to realize the reproduction processing unit 28f by.

  The input receiving unit 28a displays, on the page 32, points indicating the position coordinates input from the operation unit 21 with respect to the page 32 displayed on the second display unit 22b by the note application program 27a, for example. That is, the locus of the position coordinates input from the operation unit 21 is drawn on the page 32 as handwritten data. The input accepting unit 28a accepts, for example, when one of the icons 31a to 31f illustrated in FIG. 3 is touched, accepts a start operation of an application program associated with the touched icon.

  The user authentication unit 28b displays an authentication screen for accepting input of authentication information on the display unit 22. Then, the user authentication unit 28b transmits the authentication information input to the authentication screen via the input reception unit 28a to the data management system 40, and performs user authentication with the data management system 40. For example, the data management system 40 sends an authentication success notification indicating that the authentication has been successful when the authentication information in the pre-registered account information matches the authentication information transmitted from the user authentication means 28b. It is transmitted to the user authentication means 28b. The user authentication unit 28b logs in the user by receiving an authentication success notification from the data management system 40. For example, the authentication information is a combination of a user ID and a password.

  When the input receiving unit 28a receives the start operation of the application program, the application executing unit 28c starts the application program whose start operation has been received by the input receiving unit 28a. The application execution unit 28c can execute a plurality of application programs in parallel.

  The recording request unit 28d adds a time stamp to the page 32 displayed on the second display unit 22b, as the playback data of the written page 32, with the write data written via the input reception unit 28a. The recording request is transmitted to the data management system 40 in real time via the communication unit 26. The recording request unit 28d executes the document application program 27b, the browser application program 27c, and the telephone application program 27d executed by the application execution unit 28c when writing to the page 32 is received by the input reception unit 28a. A time stamp is added to the content as playback data of the written page 32, and a recording request is transmitted to the data management system 40 in real time via the communication unit 26. The time stamp is information indicating the current time measured by a clock unit (not shown). In addition, the recording request unit 28d can include audio data generated by the microphone 23, image data captured by the camera 25, and the like in the playback data.

  The acquisition requesting unit 28e issues an acquisition request for the playback data of the page 32 when the page 32 whose playback data has been transmitted to the data management system 40 by the recording requesting unit 28d is displayed on the second display unit 22b. The playback data is received from the data management system 40 by transmitting the playback data to the data management system 40 via the communication unit 26.

  The reproduction processing unit 28f synchronously reproduces the write data and the execution content included in the playback data acquired by the acquisition request unit 28e, based on the time stamp added to each.

  The control unit 28 realizes the note application 28g by executing the note application program 27a. The note application 28g receives creation, display, and editing of a new note including a plurality of pages via the second touch panel.

  The control unit 28 realizes the document application 28h by executing the document application program 27b. The document application 28h receives creation, display, and editing of a document such as educational material content via the first touch panel.

  The control unit 28 realizes the browser application 28i by executing the browser application program 27c. The browser application 28i displays a Web site acquired from the Internet or the like via the communication unit 26 on the first display unit 22a.

  The control unit 28 realizes the telephone application 28j by executing the telephone application program 27d. The telephone application 28j controls a voice call with another user via the communication unit 26. The telephone application 28j can perform not only a one-to-one call but also a group call for talking with a plurality of people.

  The telephone application 28j cooperates with the note application 28g. When the application executing unit 28c activates the telephone application program 27d, the telephone application 28j displays an address book 33 (see FIG. 5) for selecting a call destination on the first display unit 22a. When activating the telephone application program 27d, the application executing means 28c also activates the note application program 27a. When the application executing unit 28c activates the note application program 27a, the note application 28g displays a new writable page 32 (see FIG. 5) on the second display unit 22b. Therefore, the caller can make notes on the page 32 displayed on the second display unit 22b via the operation unit 21 when making a call.

  The page 32 displayed on the second display unit of the calling mobile terminal is registered in the data management system 40 as a page shared between the callers, and is shared with the other party. Upon receiving an incoming call from the telephone application of the calling mobile terminal, the called mobile terminal activates the telephone application program by the application executing means, thereby enabling a video call with the caller, and enabling the called party to communicate with the caller. The displayed image 34 (see FIG. 5) is displayed on the first display unit. Then, the application execution means of the mobile terminal on the receiving side activates the note application program, acquires the page 32 shared between the callers from the data management system 40, and displays the page 32 on the second display unit. Therefore, the caller and the callee can share the writing via the page 32 displayed on the second display unit of the mobile terminal.

  When the input receiving unit 28a receives the writing to the page 32 displayed on the second display unit 22b by the note application program 27a, the second display unit 22b displays the microphone as shown in FIG. Button 35 is displayed. The microphone button 35 switches between ON and OFF each time the microphone button 35 is touched via the operation unit 21. While the microphone button 35 is ON, the microphone 23 generates audio data based on the input audio, and performs recording. When the microphone button 35 is turned on while not in a call, the audio data generated by the microphone 23 is acquired by the recording requesting unit 28d as microphone recording data. When the microphone button 35 is turned on during a call, the voice data generated by the microphone 23 is acquired by the recording request unit 28d as outgoing recording data. When the outgoing recording data is acquired by the recording requesting unit 28d, the voice data of the call destination that arrives via the network 11 is acquired by the recording requesting unit 28d as incoming recording data.

  FIG. 6 is a block diagram of the data management system 40 when configured by one computer.

  The data management system 40 shown in FIG. 6 includes an operation unit 41 which is an operation device such as a keyboard and a mouse to which various operations are input, and a display unit 42 which is a display device such as an LCD for displaying various information. A communication unit 43 that is a communication device that communicates with an external device via a network such as a LAN or the Internet, or directly by wire or wirelessly without using a network, and a semiconductor memory that stores various information, A storage unit 44, which is a nonvolatile storage device such as an HDD, and a control unit 45 that controls the entire data management system 40 are provided.

  The data management system 40 may include, for example, an MQTT (Message Queue Telemetry Transport) server to share page writing between callers via a plurality of mobile terminals. The MQTT server accepts a caller's registration for each call, notifies the other terminal's mobile terminal of the operation performed by one registered caller on the mobile terminal and reflects the operation in real-time to the caller. Page writing can be shared between users. Further, the MQTT server can share not only pages but also document data by a document application program and a Web site by a browser application program among callers.

  The storage unit 44 may be configured by, for example, a database server. The storage unit 44 includes a database 44a and a large-capacity storage unit 44b.

  FIG. 7 is a diagram showing a directory configuration of the database 44a.

  As shown in FIG. 7, the database 44a has a hierarchical structure composed of directories of "data", "user", "notes", and "pages" in this order. The “user” is provided to be identifiable by a user ID or the like for each user. “Note” is provided so as to be identifiable by a note name or the like for each note created by the user. The “page” is identifiably provided by a page ID or the like for each page constituting the note.

  Playback data 50 transmitted from a mobile terminal included in the information processing system 10 (see FIG. 1) is stored under “page” as page data. Playback data 50 includes microphone recording data 51, incoming recording data 52, outgoing recording data 53, handwriting data 54, copy and paste data 55, date bookmark data 56, and application event 57. The handwritten data 54 and the copy-and-paste data 55 are writing contents drawn on the page, but the microphone recording data 51, the incoming recording data 52, the outgoing recording data 53, the application event 57, and the like are recorded by the user when writing to the page. This is reference information for reference. In other words, the database 44a stores, for each page of each note possessed by each user, together with the content written to the page and reference information used as a reference for writing to the page.

  FIG. 8 is a diagram showing an example of the playback data 50.

  As shown in FIG. 8, microphone recording data 51, incoming recording data 52, and outgoing recording data 53 are recorded in an audio file format such as wav. The handwritten data 54 is recorded in a vector format such as SVG. Further, the copy and paste data 55 is recorded including characters, images, urls of link destinations, and the like. In the date bookmark data 56, date and time information is recorded. The application event 57 records the execution contents of the application.

  In FIG. 7, the document data is stored under “user”. When the playback data 50 includes the application event 57 of the document application 28h (see FIG. 2), the application event 57 includes the file name of the document data displayed by the document application 28h. Therefore, when the application event 57 of the document application 28h is included in the playback data 50, the document application 28h stores the document data to be displayed on the data management system 40 based on the file name of the document data included in the application event 57. And can be displayed.

  The large-capacity storage unit 44b is a large-capacity storage unit such as an HDD, and stores various data such as WAV, XML, SVG, JSON, PDF, Microsoft Word, and Microsoft Excel. Entities of large data amounts, such as microphone recording data 51, incoming recording data 52, outgoing recording data 53, handwritten data 54, and document data, are stored in the large-capacity storage unit 44b.

  The control unit 45 includes, for example, a CPU, a ROM storing programs and various data, and a RAM serving as a memory used as a work area of the CPU of the control unit 45. The CPU of the control unit 45 executes a program stored in the storage unit 44 or the ROM of the control unit 45.

  The control unit 45 executes a program stored in the storage unit 44 or the ROM of the control unit 45, so that the mobile terminal included in the information processing system 10 (see FIG. 1) transmits the network 11 (see FIG. 1). Via a network 11 from a recording request receiving unit 45a for receiving a recording request for playback data transmitted via the network 11 and storing the playback data in the storage unit 44, and a mobile terminal included in the information processing system 10. An acquisition request responding unit 45b for receiving the transmitted playback data acquisition request, searching the storage unit 44 for the playback data, and returning the searched playback data to the mobile terminal via the network 11, Request from each application of mobile terminal included in system 10 An application processing unit 45c which performs various application processing according to realize a data transmission means 45d for transmitting at least a portion of the playback data recorded in the data management system 40 to the outside of the information processing system 10. The recording request receiving unit 45a, the acquisition request responding unit 45b, the application processing unit 45c, and the data transmitting unit 45d may each be configured by a plurality of Web servers.

  The application processing unit 45c performs, for example, call control via a network such as a public line in response to a request from a telephone application of a mobile terminal. The data management system 40 may include a WebRTC gateway, an IP-PBX, and a SIP server for performing a telephone call control process. The application processing unit 45c includes a notebook application server that provides a function of a note application of a mobile terminal, a document application server that provides a function of a document application of a mobile terminal, and a browser server that provides a function of a browser application of a mobile terminal. And a telephone server that provides a function of a telephone application of the mobile terminal.

  Next, the operation of the information processing system 10 will be described.

  First, the operation of the information processing system 10 when the playback data 50 transmitted by the mobile terminal 20 to the data management system 40 is stored in the data management system 40 will be described.

  When the page 32 is displayed on the second display unit 22b by the note application program 27a, the user logged in by the user authentication unit 28b writes the page 32 or plays back the playback data 50 described later. Whether to perform the operation can be selected via the operation unit 21. The recording request unit 28d of the mobile terminal 20 starts the operation shown in FIG. 9 when writing to the page 32 is selected.

  FIG. 9 is a flowchart of the operation of the mobile terminal 20 when writing to the page 32 is performed.

  As shown in FIG. 9, the recording request unit 28d of the mobile terminal 20 creates date bookmark data indicating the recording start date and time, adds a time stamp to the created date bookmark data, and converts the date bookmark data with the time stamp added thereto. A recording request is transmitted to the data management system 40 as date bookmark data of the playback data of the page 32 (S101). Therefore, the recording request receiving unit 45a of the data management system 40 receives the recording request transmitted from the mobile terminal 20 in S101, and adds the date bookmark data that is the target of the received recording request to the playback data 50 of the page 32. The date bookmark data 56 is stored in the storage unit 44 under the page 32 as at least a part.

  When the process of S101 ends, the recording requesting unit 28d of the mobile terminal 20 determines whether an application program other than the note application program 27a, that is, another application program has been executed by the application executing unit 28c (S102).

  When the recording request unit 28d determines in S102 that another application program, that is, an application program such as the document application program 27b, the browser application program 27c, or the telephone application program 27d has been executed by the application execution unit 28c, the recording execution unit 28c An application event indicating the execution content of the executed application program is acquired, a time stamp is added to the acquired application event, and the application event to which the time stamp is added is regarded as an application event of the playback data of the page 32, and the data management system 40 A recording request is transmitted to the server (S103). Therefore, the recording request receiving unit 45a of the data management system 40 receives the recording request transmitted from the mobile terminal 20 in S103, and sends the application event that is the target of the received recording request to the application of the playback data 50 of the page 32. At least a part of the event 57 is stored in the storage unit 44 under the page 32. For example, as shown in FIG. 10, when writing is performed on the page 32 displayed on the second display unit 22b, the document data 36 by the document application program 27b or the Web site by the browser application program 27c When 37 is displayed on the first display unit 22a, the recording request unit 28d of the mobile terminal 20 displays an application event indicating the execution content of the document application program 27b and an application event indicating the execution content of the browser application program 27c. A timestamp is added to each of the acquired application events, and each of the application events to which the timestamp is added is added to the playback data of the page 32. As application event, and transmits a recording request to the data management system 40. Therefore, the recording request receiving unit 45a of the data management system 40 can record the document data and the information on the website that the user referred to when writing the page 32 in the mobile terminal 20 in association with the page 32.

  The recording request unit 28d of the mobile terminal 20 determines in S102 that another application program has not been executed by the application execution unit 28c, or when the process of S103 ends, a handwriting input is performed on the page 32 by the input reception unit 28a. It is determined whether or not it has been (S104).

  When the recording requesting unit 28d determines in S104 that handwriting has been input to the page 32 by the input receiving unit 28a, the recording requesting unit 28d obtains the handwritten data written on the page 32 by the input receiving unit 28a and adds a time stamp to the obtained handwritten data. Then, the recording request is transmitted to the data management system 40 as the handwritten data of the playback data of the page 32 with the time stamp added thereto (S105). Therefore, the recording request receiving unit 45a of the data management system 40 receives the recording request transmitted from the mobile terminal 20 in S105, and writes the handwritten data that is the target of the received recording request by handwriting the playback data 50 of the page 32. At least a part of the data 54 is stored in the storage unit 44 under the page 32.

  The recording requesting unit 28d of the mobile terminal 20 determines in S104 that handwriting input has not been performed on the page 32 by the input receiving unit 28a, or when the process of S105 ends, the microphone button 35 is turned on by the input receiving unit 28a. It is determined whether or not it has been (S106).

  When the recording requesting unit 28d determines in S106 that the microphone button 35 has been turned ON by the input receiving unit 28a, the recording requesting unit 28d acquires any one of the microphone recording data, the incoming recording data, and the outgoing recording data according to the call situation, A time stamp is added to the obtained recording data, and the recording data to which the time stamp is added is transmitted to the data management system 40 as recording data of the playback data of the page 32 (S107). Therefore, the recording request receiving unit 45a of the data management system 40 receives the recording request transmitted from the mobile terminal 20 in S107, and converts the recording data that is the target of the received recording request into the recording of the playback data 50 of the page 32. The data is stored under the page 32 in the storage unit 44 as at least a part of the data. If the recording data transmitted from the mobile terminal 20 in S107 is the microphone recording data, the recording request receiving unit 45a converts the microphone recording data into the microphone recording data of the playback data 50 on the page 32. It is stored as at least a part of the data 51. When the recording data transmitted from the mobile terminal 20 in S107 is the incoming recording data, the recording request receiving unit 45a transmits the incoming recording data to the incoming recording data of the playback data 50 of the page 32 when the recording data is the incoming recording data. 52 is stored as at least a part thereof. If the recording data transmitted from the mobile terminal 20 in S107 is the outgoing recording data, the recording request receiving means 45a transmits the outgoing recording data to the outgoing recording data 50 of the playback data 50 on the page 32. 53 is stored as at least a part. Therefore, for example, when the input receiving unit 28a receives writing to the page 32 displayed on the second display unit 22b by the note application program 27a in the mobile terminal 20, the voice of the lecturer or the like recorded by the mobile terminal 20 is received. And the call voice can be recorded in the data management system 40 in association with the page 32.

  The recording requesting unit 28d of the mobile terminal 20 determines in S106 that the microphone button 35 has not been turned ON by the input receiving unit 28a, or when the processing of S107 ends, the input receiving unit 28a copies and executes the page 32. It is determined whether or not the paste has been performed (S108). For example, as shown in FIG. 11, the user selects a copy target area 38 from the document data 36 or the Web site 37 via the operation unit 21, and selects characters, images, and link information in the selected copy target area 38. Can be copied and pasted at a designated position in the page 32 to copy and paste. The user can also copy and paste the contents of the page of the note created by another user to the page 32 to which the user writes.

  When the recording requesting unit 28d determines in S108 that the input receiving unit 28a has copied and pasted the page 32, the recording requesting unit 28d acquires the copy and paste data written on the page 32 by the input receiving unit 28a, and acquires the acquired copy and paste. A time stamp is added to the paste data, and a copy request is transmitted to the data management system 40 as the copy and paste data of the playback data of the page 32 with the time stamp added (S109). Therefore, the recording request receiving unit 45a of the data management system 40 receives the recording request transmitted from the mobile terminal 20 in S109, and copies the copy and paste data, which is the target of the received recording request, to the playback data 50 of the page 32. Is stored under the page 32 in the storage unit 44 as at least a part of the copy and paste data 55.

  The recording requesting unit 28d of the mobile terminal 20 determines in S108 that the copy and paste has not been performed on the page 32 by the input receiving unit 28a, or when the process of S109 ends, the input receiving unit 28a stops accepting the writing. It is determined whether the process has been completed (S110). For example, the input accepting unit 28a writes to the displayed page 32 when an operation of closing a note is performed through the operation unit 21 or a display operation of a next page is performed through the operation unit 21. Ends reception.

  When the recording request unit 28d of the mobile terminal 20 determines in S110 that the reception of writing by the input receiving unit 28a has not been completed, the recording request unit 28d executes the process of S102. That is, the recording requesting unit 28d requests recording of the playback data of the page 32 while the input receiving unit 28a receives the writing to the page 32.

  When the recording request unit 28d of the mobile terminal 20 determines in S110 that the reception of the writing by the input receiving unit 28a has been completed, the recording request unit 28d creates date bookmark data indicating the recording end date and time, and adds a time stamp to the created date bookmark data. The recording request is transmitted to the data management system 40 using the date bookmark data to which the time stamp is added as the date bookmark data of the playback data of the page 32 (S111). Therefore, the recording request receiving unit 45a of the data management system 40 receives the recording request transmitted from the mobile terminal 20 in S111, and stores the date bookmark data as the target of the received recording request in the playback data 50 of the page 32. The date bookmark data 56 is stored in the storage unit 44 under the page 32 as at least a part.

  When the processing of S111 ends, the recording request unit 28d of the mobile terminal 20 ends the operation illustrated in FIG.

  Next, the operation of the information processing system 10 when the user copies and pastes the written content of the note page created by another user to the page 32 to be written by the user will be described in detail.

  As shown in FIG. 12, the user (hereinafter, referred to as “user A”) displays a note page created by another user (hereinafter, referred to as “user B”) on the user A's own mobile terminal 20 in the second display mode. The content written on the page of the user B can be copied and pasted on the page 32 to be written by the user A himself / herself by displaying the content on the page 22b. In this case, the user A can display the page permitted by the user B on the mobile terminal 20 of the user A by inputting a preset authentication key. The user B communicates with the data management system 40 via the note application 28g of the user B's own mobile terminal in advance, and sets an authentication key such as a password on a page that permits the user A to display. good.

  When the recording request unit 28d of the mobile terminal 20 of the user A transmits a recording request of copy and paste data of the written content of the page of the user B to the data management system 40 in S109, the recording request receiving unit 45a of the data management system 40 As shown in FIG. 13, the copy and paste data including the playback data of the user B corresponding to the copy target area 38 is stored as at least a part of the copy and paste data 55 of the playback data 50 of the page 32. The data is stored under the page 32 in the unit 44. That is, not only the write data (the handwritten data 54 and the copy and paste data 55) in the copy target area 38 but also the user B corresponding to the write data in the copy target area 38 from the first time stamp to the last time stamp. All of the playback data is to be copied.

  For example, in FIG. 14, the playback data of the user B corresponding to the copy target area 38 includes microphone recording data, handwritten data, copy and paste data, and an application event. Therefore, the write data of the copy target area 38 copied by the user A from the page of the user B is pasted on the page 32 of the user A while holding the document data and the website information referred to by the user B on the backbone. Attached.

  As shown in FIG. 15, when the copy target area 38 copied from the page of the user B is pasted on the page 32 of the user A, the size of the copy target area 38 is changed to the remaining writable area of the page 32 (paste area). When the size is larger than the size of the attachment area, the copy target area 38 is pasted over the page 32 and the next page after the page 32. In the example shown in FIG. 15, the copy target area 38 is divided into a copy target area 38a to be pasted on the page 32 and a copy target area 38b to be pasted on the page following the page 32. Therefore, first, the recording request unit 28d of the mobile terminal 20 of the user A adds a time stamp to the copy and paste data based on the copy target area 38a, and outputs the copy and paste data with the time stamp added to the playback data of the page 32. The recording request is transmitted to the data management system 40 as the copy and paste data (S109). Next, the recording requesting unit 28d determines that the writing has been completed because there is no more writable area in the page 32, creates date bookmark data indicating the recording end date and time, adds a time stamp to the created date bookmark data, The recording request is transmitted to the data management system 40 using the date bookmark data with the stamp as the date bookmark data of the playback data of the page 32 (S111). Next, in order to paste the copy target area 38b, the page next to the page 32 is displayed by the note application program 27a, and writing is received by the input receiving unit 28a. Next, the recording request unit 28d of the mobile terminal 20 of the user A creates date bookmark data indicating the recording start date and time, adds a time stamp to the created date bookmark data, and converts the date bookmark data with the time stamp added to the page 32. A recording request is transmitted to the data management system 40 as the date bookmark data of the playback data of the next page (S101). Next, since the copy target area 38b is pasted on the page next to the page 32, the recording request unit 28d adds a time stamp to the copy and paste data based on the copy target area 38b, and adds the time stamp to the copy and paste data. The recording request is transmitted to the data management system 40 as the copy and paste data of the playback data of the page next to the page 32 with the paste data (S109). Therefore, as shown in FIGS. 16 and 17, the recording request receiving unit 45a of the data management system 40 stores the user A as the copy-and-paste data 55 under the plurality of continuous pages of the user A, The playback data corresponding to the copy target area 38a of the user B and the playback data corresponding to the copy target area 38b of the user B are copied.

  When the copy source user B has created the writing in the copy target area 38 by including the write data copied from the page of the user C, the copy target area 38 stores the user B's playback data. , Playback data of the user C. In this case, when the recording request receiving unit 45a of the data management system 40 receives the recording request of the copy and paste data as the playback data from the user A, the recording request receiving unit 45a stores the playback data of the plurality of users corresponding to the copy target area 38. The copy is performed as copy and paste data 55 under the page of the user A in the unit 44.

  Next, an operation of the information processing system 10 when the playback data 50 stored in the data management system 40 is reproduced by the mobile terminal 20 will be described.

  The user logged in by the user authentication unit 28b can input an operation of opening a notebook via the operation unit 21 of the mobile terminal 20. When the input receiving unit 28a of the mobile terminal 20 receives the operation of opening the notebook, the acquisition requesting unit 28e of the mobile terminal 20 starts the operation shown in FIG.

  FIG. 18 is a flowchart of the operation of the mobile terminal 20 when reproducing the playback data 50.

  As shown in FIG. 18, the acquisition requesting unit 28 e of the mobile terminal 20 sends a request to acquire playback data of the page last displayed on the second display unit 22 b, that is, the latest updated page that is the latest page. It is transmitted to the management system 40 (S131). In the case of a note that has never been opened, such as a new note, the acquisition request unit 28e sets the first page as the last updated page. The acquisition request response unit 45b of the data management system 40 receives the acquisition request transmitted from the mobile terminal 20 in S131, and searches the storage unit 44 for playback data of the page that is the target of the received acquisition request. Here, the acquisition request responding unit 45b may specify the page in which the latest date and time is stored as the last updated page among the date bookmark data and the time stamp stored in the storage unit 44. When the playback request response unit 45b of the data management system 40 specifies the playback data of the page for which the acquisition request is made, the date bookmark data, the application event, and the handwriting stored in the directory of the last updated page from the storage unit 44. Data, recording data, copy and paste data, etc. are acquired as playback data, and the playback data of the acquired page is transmitted to the mobile terminal 20. In the case of a new page for which writing has never been accepted, no playback data is stored in the directory of this page. Therefore, the acquisition request response unit 45b of the data management system 40 transmits empty information (eg, NULL) to the mobile terminal 20 as playback data when a request to acquire a new page is made.

  Upon completion of the process in S131, the acquisition request unit 28e of the mobile terminal 20 completes the reception of the playback data from the data management system 40 until determining that the reception of the playback data from the data management system 40 is completed. It is determined whether or not it is (S132).

  When it is determined in S132 that the reception of the playback data from the data management system 40 has been completed, the playback processing unit 28f of the mobile terminal 20 determines that the reception of the playback data from the data management system 40 has been completed in S132. Is displayed on the second display unit 22b by the note application program 27a (S133). Here, when the playback data determined in S132 that the reception from the data management system 40 has been completed is empty information, the reproduction processing unit 28f displays only the page in which nothing is written in the second display. It is displayed on the section 22b.

  When the processing of S133 is completed, the reproduction processing unit 28f of the mobile terminal 20 determines, based on the application event included in the playback data determined in S132 that the reception from the data management system 40 has been completed, based on the application event other than the note application program 27a. The screen displayed last by another application program is displayed on the display unit 22 (S134). That is, the reproduction processing unit 28f displays on the display unit 22 a screen of another application program that was being displayed on the display unit 22 at the time when the writing to the last updated page was completed last time. For example, the document data and the website displayed on the display unit 22 at the time when the writing to the last updated page is completed last time are displayed on the first display unit 22a.

  When the processing in S134 ends, the reproduction processing unit 28f of the mobile terminal 20 displays a reproduction bar 39 for reproducing the playback data shown in FIG. 19 on the second display unit 22b (S135). The playback bar 39 is a playback area 39a which is an area indicating the playback time length of the playback data, and a cursor indicating a playback position in the playback area 39a. A playback cursor 39b slidable to an arbitrary position, a playback start button 39c for playing back playback data from a time corresponding to the playback position of the playback cursor 39b, and a playback stop for stopping playback of the playback data. Button 39d.

  When the processing of S135 ends, the playback processing unit 28f of the mobile terminal 20 determines whether or not the playback bar 39 has accepted the playback operation by the playback start button 39c until it determines that the playback bar 39 has accepted the playback operation by the playback start button 39c. It is determined (S136).

  When the playback processing unit 28f of the mobile terminal 20 determines in S136 that the playback bar 39 has accepted the playback operation by the playback start button 39c, the playback processing unit 28f based on the playback data after the time stamp at the time corresponding to the playback position indicated by the playback cursor 39b. The display is once erased, and the playback data is sequentially reproduced from the time stamp corresponding to the reproduction position indicated by the reproduction cursor 39b (S137). That is, the reproduction processing unit 28f has a time stamp indicating the time closest to this time after the time indicated by the reproduction cursor 39b in the playback data determined in S132 that the reception from the data management system 40 has been completed. From the data to the data having the time stamp of the last time, the data is reproduced in chronological order. The reproduction processing unit 28f, when any of the write data in the page displayed on the second display unit 22b is touched, changes the time stamp of the touched write data to the reproduction position. And the playback data may be reproduced.

  When playing back the playback data in S137, the playback processing unit 28f of the mobile terminal 20 obtains the playback data to be played back sequentially from the data management system 40 via the acquisition requesting unit 28e and performs streaming playback. good. For example, as shown in FIG. 20, the reproduction processing unit 28f generates a timer interrupt at a specific time, such as every one second, during reproduction of the playback data, and starts the next time from the time corresponding to the generated timer interrupt. An acquisition requesting unit 28e makes a request to the data management system 40 to acquire playback data having a time stamp corresponding to a specific time such as a time until a timer interrupt is generated. Obtained from the system 40. When the playback processing unit 28f acquires the playback data from the data management system 40, the playback processing unit 28f passes the acquired data for the corresponding period to at least one or more of the note application 28g, the document application 28h, the browser application 28i, and the telephone application 28j, and synchronizes the data. To play. When the mobile terminal 20 sequentially obtains playback data from the data management system 40 and performs streaming playback, it is possible to reduce the amount of playback data received when displaying the last updated page on the second display unit 22b. Therefore, the time required to display the last updated page can be reduced.

  Although the reproduction of the playback data for the last updated page has been described above, the same applies to the reproduction of the playback data for pages other than the last updated page. For example, after displaying the last updated page on the second display unit 22b, the reproduction processing unit 28f of the mobile terminal 20 performs an operation of displaying the next page or an operation of displaying the previous page via the operation unit 21. If the input is received, a request for acquiring playback data for the new page to be displayed is transmitted to the data management system 40, and the page to be displayed is displayed on the second display unit based on the playback data acquired from the data management system 40. 22b, and playback data for this page is reproduced according to an operation via the operation unit 21.

  Next, the operation of the data management system 40 when transmitting playback data to the outside of the information processing system 10 will be described.

  When an instruction to transmit playback data to a specific location outside the information processing system 10 is input via the operation unit 41 or the communication unit 43, the data transmission unit 45d of the data management system 40 Send data to the indicated location. For example, the data transmitting unit 45d can download the playback data to an arbitrary PC (Personal Computer). The user can upload the playback data downloaded from the data management system 40 to an online storage service such as a Google (registered trademark) drive or Dropbox (registered trademark). Note that the user can make settings for sharing the playback data stored in the online storage service outside the information processing system 10 with other users with respect to this online storage service.

  FIG. 21 is a diagram showing a directory configuration of playback data in an online storage service external to the information processing system 10.

  The directory structure shown in FIG. 21 includes playback data of three notes “note 1”, “note 2”, and “note 3”. “Note1” is a note including three pages “page1”, “page2”, and “page3”.

  As described above, the reproduction processing unit 28f of the mobile terminal 20 can cause the note application program 27a to display a page based on the playback data stored in the data management system 40 on the second display unit 22b. Similarly, the reproduction processing unit 28f can display a page based on the playback data stored outside the information processing system 10 on the second display unit 22b by the note application program 27a. For example, the reproduction processing unit 28f can display a page based on the playback data stored in the online storage service external to the information processing system 10 on the second display unit 22b by the note application program 27a. Here, even if the playback data stored in the online storage service external to the information processing system 10 is that of the user B, the setting for sharing the playback data with the user A is set for the online storage service. If the playback processing has been performed, the playback processing unit 28f of the mobile terminal 20 to which the user A is logged in causes the note application program 27a to display the page based on the playback data on the second display unit 22b. Can be.

  As described above, the mobile terminal 20 can copy and paste the written content of the page based on the playback data stored in the data management system 40 to another page. Similarly, the mobile terminal 20 can copy and paste the written content of the page based on the playback data stored outside the information processing system 10 to another page.

  The mobile terminal 20 can play back the playback data stored in the data management system 40, as described above. Similarly, the mobile terminal 20 can play back playback data stored outside the information processing system 10.

  As described above, the information processing system 10 sets the time stamp on the document data, the website, the audio data from the microphone 23, and the like referred to when the page displayed on the mobile terminal 20 is written. In addition, it is recorded as playback data in association with the page together with the write data. Then, when displaying the written page, the information processing system 10 can acquire the playback data of the page, and can synchronously reproduce each data included in the acquired playback data based on the time stamp. . Therefore, the information processing system 10 can reproduce the writing of the note on the page together with the writing state, and can make the user easily understand the contents of the writing of the note on the page.

  The information processing system 10 transmits the playback data recorded in the data management system 40 to the outside of the information processing system 10 and writes a page based on the playback data stored outside the information processing system 10 to the note application. Since the data is displayed on the second display unit 22b by 28g, even if the data management system 40 is not permitted to access the playback data of another user, the data stored outside the information processing system 10 may be displayed. The page based on the playback data of the user can be displayed on the second display unit 22b by the note application 28g. Note that, in the present embodiment, the information processing system 10 transmits, to the playback data of a user other than the user logged in to the mobile terminal 20, the playback data recorded in the data management system 40 from the mobile terminal 20. Access is permitted by the authentication key. However, the information processing system 10 prohibits the mobile terminal 20 from accessing the playback data of a user other than the user who is logging in to the mobile terminal 20 among the playback data recorded in the data management system 40. Thereby, security in the data management system 40 can be improved.

  When transmitting the playback data stored in the data management system 40 to the outside of the information processing system 10, the data management system 40 transmits the data stored in the large-capacity storage unit 44 b to the data processing system 10. The playback data stored in the database 44a may be transmitted to the outside of the information processing system 10 without being transmitted to the outside of the information processing system 10. When the substance of the data stored in the large-capacity storage unit 44b is not transmitted to the outside of the information processing system 10, the reproduction processing unit 28f of the mobile terminal 20 outputs the playback data stored outside the information processing system 10. When displaying the page based on the data on the second display unit 22b by the note application 28g, the entity of the data stored in the large-capacity storage unit 44b is used. The information processing system 10 is stored in the database 44a of the data management system 40 without transmitting the substance of the data stored in the large-capacity storage unit 44b of the data management system 40 to the outside of the information processing system 10. When transmitting the playback data to the outside of the information processing system 10 and displaying a page based on the playback data stored outside the information processing system 10 on the second display unit 22b by the note application 28g, a large Since the data entity stored in the capacity storage unit 44b is used, even if the size of the data entity is large, the load on a system external to the information processing system 10 can be suppressed.

Reference Signs List 10 information processing system 11 network 20 mobile terminal 22 display unit (display device)
22a First display unit 22b Second display unit 27a Note application program (application program)
27b Document application program (application program)
27c Browser application program (application program)
27d Telephone application program (application program)
28a input receiving means 28c application executing means 28d recording requesting means 28f reproduction processing means 28g note application 32 page 40 data management system 44a database 44b large capacity storage unit (storage unit)
45d Data transmission means 50 Playback data 54 Handwritten data (write data)
55 Copy and paste data (write data)
57 Application Event (Application Execution Contents)

Claims (2)

Mobile devices,
An information processing system comprising: a data management system connected to the mobile terminal via a network;
The mobile terminal,
A display device for displaying the note page,
Input receiving means for receiving writing to the page displayed on the display device,
An application execution means for executing the application program;
A note application realized by being executed by the application executing means, the note application program being the application program for displaying the page on the display device and enabling writing to the page,
Record request means for requesting the data management system to record playback data including the write data received by the input reception means, and execution contents of the application program, each of which has a time stamp added thereto,
Playback processing means for synchronously playing back the write data and the execution content included in the playback data obtained from the data management system based on the time stamp,
The data management system includes a data transmission unit that transmits the playback data recorded in the data management system to outside the information processing system,
The information processing system, wherein the reproduction processing means displays the page based on the playback data stored outside the information processing system on the display device by the note application. The data management system comprises:
A database for storing at least a part of the playback data for each note;
A storage unit that stores an entity of at least a part of the playback data stored in the database,
The data transmission unit transmits the playback data stored in the database to the outside of the information processing system without transmitting the entity stored in the storage unit to the outside of the information processing system. ,
The playback processing unit may be configured to display the page based on the playback data stored outside of the information processing system on the display device by the note application, the entity stored in the storage unit. The information processing system according to claim 1, wherein the information processing system utilizes the information.

Images