JP6810515B2 - Handwriting information processing device - Google Patents

Description

The present invention relates to a handwritten information processing device that enables viewing of handwritten information written on a predetermined entry medium using a digital pen.

In recent years, the introduction of IT technology has been promoted in which the contents of utterances of the speaker and the state of the viewers who watch the utterances, such as various presentations and classes at schools, are converted into video files and later reproduced as feedback information. For example, in order to feed back the contents of the manifestation of intention of many participants to the presentation at the meeting, the participants receive the signal of manifestation of intention to be transmitted at any time as the log data of the manifestation of intention, and the contents of the presentation. When the recorded data is received, the recorded data is displayed in a time-series array, the mark generated from the log data is displayed on the array display as the timing of manifestation of intention, and the desired mark is specified. , A system has been proposed in which recorded data is reproduced and displayed from a reproduction position corresponding to a designated mark (see Patent Document 1).

According to the system according to Patent Document 1, the manifestation of intention of the viewer who is the participant can be confirmed as feedback information at the time of reproduction, but since the manifestation of intention is performed via the dedicated remote controller, the manifestation of intention is displayed. There is a problem that the expression is limited to the range that can be operated by the dedicated remote controller.

To solve this problem, handwritten information using a digital pen may be displayed as written data. For example, a digital pen having a writing means and a means for transmitting writing data, an answer sheet on which a two-dimensional pattern that can be read by the digital pen is printed on the surface, and a communication terminal that receives writing data transmitted from the digital pen. , A control table connected to the communication terminal on the network and having a plurality of icons and control codes corresponding to each of the plurality of icons is provided, and when the icons are specified, the corresponding control codes are followed. A lesson support system having a lesson support server that executes the processed contents and a display means for displaying the written data has been proposed (see Patent Document 2).

If the system according to Patent Document 2 is applied, the manifestation of intention of the participant can be used as feedback information by converting the specific content into data as it is without any restriction. In addition, by handwriting the input of the manifestation of intention, it can be expected that the viewer can intuitively describe the specific content without intervening complicated operations.

Japanese Patent No. 4848329 Japanese Patent No. 5319458

However, although all of the above-mentioned conventional techniques are useful for confirmation, analysis, etc. at the time of reproduction as feedback information, it is said that the reaction of the viewer in real time cannot be sufficiently grasped while the speaker is speaking. There was a problem. That is, although it is possible to realize the immediacy of recording the reaction by converting the reaction of the viewer into handwritten information, the immediacy of browsing on the speaker side has not been realized by the above-mentioned prior art.

In addition, for the speaker, the reason for each reaction is more important than the conclusion of the reaction, regardless of whether the utterance progresses or the feedback is reproduced after the utterance ends. In many cases. For example, when the viewer is an evaluator of the speaker's presentation, etc., the speaker knows the specific reason why it is judged that there is room for improvement in the speaking style rather than the quality of the speaking style itself. Is more important.

Further, for example, in a school class, that is, when the speaker is a teacher and the viewer is a student, the answer to the teacher's question is described in the handwritten information and can be viewed on the monitor. Can be constructed from the above-mentioned advanced technology, but in cases such as such classes, the thinking process of each student who came to the answer is often more important than the correctness of the answer, and the thinking process is observed in real time. That is very important for the progress of the teacher's lessons.

Conventionally, there is a system that enables the process of describing handwritten information such as the answer to be viewed simultaneously on a monitor or the like, but the viewer responds (understands) according to the intention of the speaker. In order to grasp in real time whether or not it is done, it is necessary to rely not a little on the old-fashioned inter-desk patrol and inter-desk guidance, or to read the description contents of each viewer one by one with the monitor etc. There was a problem that the burden on the person was not sufficiently reduced and it was difficult to use it on a daily basis in actual classes.

The present invention is for solving the above-mentioned problems, and grasps a specific viewer's reaction and a process leading to the reaction at both the utterance progress and the feedback reproduction without imposing a load on the speaker. It is an object of the present invention to provide a device capable of performing.

In order to achieve the above object, the handwritten information processing apparatus according to the present invention sequentially converts the handwritten information described by the viewer into vector data and collects it, and also stores the moving image data for photographing the speaker or the viewer. When a predetermined keyword is specified, the keyword included in the vector data is changed to highlighting data, and the vector data is sequentially displayed so that it can be viewed in comparison with the real-time playback display of the moving image data while the speaker is speaking. The most important feature is to display in a series and to display the highlighted data sequentially.

That is, it is a handwritten information processing device provided with a display means that enables a plurality of viewers to view handwritten information written on a predetermined entry medium using a digital pen for the speaker.
A receiving means that converts the handwritten information described by the plurality of viewers into vector data based on the coordinate values set in the entry medium and collects the handwritten information corresponding to the transmission time data.
A moving image data storage means for storing moving image data including audio obtained by shooting the speaker or the viewer together with shooting time data, and
A reproduction position setting means for the reproduction position of said received transmission time data and the photographing time data and moving image data of the time said stored matching by said receiving means,
Data generating means for generating text data from the vector data received by said receiving means,
A search means for searching text data that matches a keyword specified by a predetermined condition from the generated text data, and
It has a display changing means for changing the vector data corresponding to the searched text data into a display that can be distinguished from other vector data to be highlighted data.
The predetermined entry medium includes at least items to be described in time series in advance according to the progress of speech as designated items for describing the handwritten information, and each designated item and a corresponding handwritten information description column. Is specified by the coordinate data and provided, and when handwritten information is described in each of the entry fields according to the progress of the speech, the receiving means receives the converted vector data and the corresponding designated item data. A display list is displayed in which the transmission time data and the designated item data are associated with the vector data that is specified and received by the coordinate data and displayed in comparison with the moving image data by the display means. Equipped with a display list generation means to generate
The display means, specifying a vector data corresponding to the transmission time data set as the playback position, and the vector data including the highlight data, the reproduction of the moving image data corresponding thereto, a comparison viewable in displayed, when real-time playback of the moving image data, the vector data is sequentially displayed on the chronological received by said receiving means, as well as changes to sequentially highlight data vector data by the display changing means, the moving image data It is characterized by displaying in a viewable manner in contrast to real-time playback.

According to this configuration, vector data related to handwritten information can be viewed together with moving image data not only during so-called feedback reproduction after the end of the utterance but also in real time, and further, by highlighting data, for example, utterance. It is possible to confirm in real time whether or not the viewer understands the keyword intended by the person. When there are many viewers who describe handwritten information, if the vector data of all the viewers is displayed by the display means, the screen may become complicated and the convenience of browsing may be impaired. Therefore, the viewer may be provided with a form provided with designated items for which handwritten information should be described in the entry medium, and a column for describing handwritten information may be provided corresponding to each designated item. If the designated items include at least items described in time series in advance according to the progress of the utterance, it is highly convenient when describing handwritten information and when browsing after conversion to vector data.

In the present invention, the "speaker" means a person who speaks to the listener regardless of the number of people, for example, a presenter in a presentation, a school teacher, a seminar lecturer, various instructors, and others. , Job interviews, medical interviews, etc., including one-on-one or one-on-one speakers. Further, in the present invention, the "viewer" means a person who watches the utterance of the speaker, and includes a student who directly watches the utterance, a listener, and an evaluator who evaluates the utterance content of the speaker. ..

In addition to specifying the keyword in advance, the keyword may be newly specified while the utterance is in progress. When the keyword is newly specified, the display changing means is different from the highlighted data changed in advance. May be changed to.

Also, for example, a symbol indicating a weighting to each item specified provided, it may be a prominent display than other vector data vector data according column symbol is described showing a predetermined weighting. If the designated items are the same and the transmission time data is within a predetermined range, the display may be grouped. Further, if the vector data having the keyword in common is integrated to generate a single data and changed to the highlighting data, the complexity of the display space on the screen can be eliminated.

The display list generation means displays, at least, for the same designated item, only the number in which the same weighting symbol is described or the number of vector data having the same keyword in common is equal to or greater than a predetermined number. You may make it selectable.

The receiving means has a display unit that receives vector data from the digital pen and can receive the moving image data and reproduce it in real time. The moving image data displayed on the display unit is the handwritten information. When the superimposition mark data is entered in the place related to the content of the above, the receiving means receives the superimposition mark data corresponding to the vector data and the transmission time data from the communication terminal, and the reproduction position. When vector data is specified as, the display means can also reproduce and display moving image data in which the superimposed mark data is combined.

In some cases, the superimposed mark cannot be directly combined with the moving image data of real-time reproduction due to the communication environment or other reasons, or it may be preferable not to directly combine the superimposed mark. In such a case, the receiving means receives vector data from the digital pen and has a display unit capable of receiving the moving image data and reproducing the moving image data in real time , and the handwritten information is entered from the moving image data. The frame image data corresponding to the time is extracted, the extracted frame image data is displayed together with the handwritten information on the display unit, and the displayed frame image data is a portion related to the content of the handwritten information. It suffices to be able to fill in so that the superimposed mark data can be combined. In this case, when the receiving means receives the frame image data in which the vector data and the superimposed mark data are combined from the communication terminal and specifies the vector data as the reproduction position, the display means receives the superimposed mark data. It suffices to make it possible to reproduce and display the moving image data from the position corresponding to the frame image data in which the above is combined.

According to this configuration, in addition to the highlighting data, the specific content regarding the reaction of the viewer can be intuitively grasped from the moving image data.

In recent years, as a teaching and learning method for schools and the like, active learning in which students and students corresponding to the viewer actively participate in the class has been actively adopted, and the present invention relates to such active learning. When using a handwritten information processing device, it is assumed that a plurality of groups composed of a predetermined number of viewers for describing the handwritten information are formed, and the moving image data is photographed for each group, at the request of the speaker. Correspondingly, when the viewers of each group describe the handwritten information on the entry medium, the receiving means receives the vector data and the transmission time data together with the identification data for identifying each group, and stores the moving image data. The means stores the moving image data of each group together with the identification data, and the display means distinguishes the vector data and the moving image data for each group and displays them in a list , and also displays the vector data for each group. If there is highlighting data in, the highlighting data may be displayed .

Further, in addition to the handwritten information processing device used by a speaker such as a teacher, the vector data and the moving image data of each group displayed by the display means are handwritten information processing devices for groups assigned to each group. also in the display unit, and displays only the vector data and moving picture data in the group, in order to display the highlighted data on the display unit of the previous SL handwritten information processing apparatus for the group that is assigned to each group, it may be configured to have a transmitting means for transmitting the highlighted data on the handwriting information processing apparatus for the group.

Furthermore, among the group or a predetermined number of viewers, the vector data of the handwritten information of the same group or the same viewer, has a storage means for storing as log data including the highlight data, wherein display means, along with a list allows displaying the log data of the same group or the same viewer may be the viewable display the transition of highlight data.

According to this configuration, changes in the thinking process of the group or individual viewers can be viewed.

It should be noted that the storage means has a specific means for specifying the speech count data of each viewer as the comment data associated with the viewer ID data and the speech time data specified at the time of speech, and the storage means stores the speech data. The reproduction position setting means sets the speech data as a reproduction position, and when the display means reads the speech data from the storage means, the speech data is reproduced in chronological order and the speech data being reproduced is displayed. , It may be configured to display the number of all remarks.

The present invention may be made available even when the speaker and the viewer are at a remote location. That is, the receiving means or transmitting means receives or transmits via the remote communication relay device connected to the Internet, the moving image data storage means, before Symbol Internet moving image data obtained by photographing the viewer It may be acquired and stored via the remote communication relay device connected to the Internet.

The handwritten information processing device according to the present invention is said to be able to grasp a specific viewer's reaction and the process leading to the reaction at both the utterance progress and the feedback reproduction without imposing a load on the speaker. It works.

That is, it is possible to view not only the conclusions such as whether the viewer's reaction is good or bad, but also the reason for reaching such a conclusion and the process that reached the conclusion, and it is possible to view the keywords without reading all the handwritten information. By highlighting the part, it is possible to quickly grasp the reaction of the entire viewer.

Further, since the speaker can grasp the reaction of the viewer in real time, it is possible to flexibly and accurately correct the trajectory of the utterance while the utterance is in progress.

FIG. 1 is a block configuration diagram of a handwriting information processing device according to the present invention. FIG. 2 is a processing flow diagram of the handwritten information processing apparatus according to the present invention. FIG. 3 is a diagram showing a form and an example of an entry form for filling in with a digital pen. FIG. 4 is a block configuration diagram of a handwritten information processing apparatus having a display list generation unit. FIG. 5 is a diagram showing a screen display example of the handwriting information processing apparatus. FIG. 6 is a diagram showing an example of a display list displayed on the screen. FIG. 7 is a processing flow diagram when the display items of the display list are limited to those having a predetermined threshold value or more. FIG. 8 is a diagram showing an example of a display list when the display items of the display list are limited to those having a predetermined threshold value or more. FIG. 9 is a device configuration diagram of a handwritten information processing device when a communication terminal is used together with an entry medium. FIG. 10 is a diagram showing an example of displaying a transmission screen of a communication terminal. FIG. 11 is a device configuration diagram of a handwriting information processing device for active learning. FIG. 12 is a block configuration diagram of the handwriting information processing device of FIG. FIG. 13 is a diagram showing a screen display example of the handwriting information processing apparatus of FIG. FIG. 14 is a block configuration diagram of a first modification of the handwriting information processing apparatus for active learning. FIG. 15 is a block configuration diagram of a second modification of the handwriting information processing apparatus for active learning. FIG. 16 is a diagram showing a device configuration when the handwritten information processing device according to the present invention is used in a remote location.

Hereinafter, preferred embodiments of the handwritten information processing apparatus according to the present invention will be described with reference to the drawings.

FIG. 1 is a block configuration diagram of the handwritten information processing apparatus 1 according to the present invention. The handwriting information processing device 1 may be a dedicated device for exerting various functions described later, but is preferably composed mainly of a central processing unit (CPU), a main memory, a magnetic disk, and other peripheral devices. Any personal computer can be used. The CPU mainly controls the operation of each component of the handwriting information processing apparatus 1 according to the present invention. The main memory stores a control program executed by the CPU and provides a work area when the program is executed by the CPU. The magnetic disk stores various applications including an operating system, a device drive of a peripheral device, and a program for performing various processes according to the present invention. In FIG. 1, only the functions necessary for explaining the functions of the present invention are described, and the description of the various functions and the like that function for general purposes is omitted.

The handwritten information processing device 1 enables the speaker to electronically view the handwritten information described by a plurality of viewers. The handwritten information processing device 1 can be used in various scenes such as mock training, mock negotiation, research presentation, trainer training, debate, role playing, school lessons / lectures, seminars, employee training, questionnaires, and events. is there. The viewer includes students and participants who directly listen to the speaker's utterance (for example, a lecture), as well as an evaluator who evaluates the utterance content, utterance attitude, and the like of the speaker who makes the utterance.

The plurality of viewers A use the digital pens P1, P2 ... Pn, respectively, to describe evaluations, impressions, answers, etc. regarding the utterance contents of the speaker on the predetermined entry forms F1, F2 ... Fn, respectively. Here, the digital pen is, for example, as shown in FIG. 1, a pen tip on which characters and the like can be written on a dedicated paper on which a dot pattern E is set in advance, and a pen tip, like a normal pen. A small built-in camera that shoots the characters written on the paper together with the dot pattern E, a sensor that activates the built-in camera when pen pressure is applied to the pen tip, and the handwritten characters shot by the built-in camera. A processor that calculates information such as the above as coordinate values from the dot pattern E and converts it into vector data, a memory that temporarily stores the handwritten information converted into vector data, and a communication that transfers the vector data from the memory to the handwriting information processing device 1. Any known material having a processor and a battery as a drive source for operating each of the above functions may be used (not shown), and is not particularly limited. However, as will be described later, since real-time confirmation by the handwriting information processing device 1 is required, the processing speed of vector data conversion should be high, and data transfer is also performed directly from the digital pen without the intervention of a cradle. Those that can be transferred are preferable.

The handwritten information described by each viewer A on the entry forms F1, F2 ... Fn using the digital pens P1, P2 ... Pn is converted into vector data by each digital pen P1, P2 ... Pn, and then each vector. The receiving unit 101 of the handwritten information processing apparatus 1 receives the data in correspondence with the transmission time data.

On the other hand, the moving image data of the speaker or the viewer is recorded by the video camera C together with the voice, and is stored in the moving image data storage unit 102 of the handwritten information processing apparatus 1 together with the shooting time data.

The time at which the transmission time data corresponding to the vector data and the shooting time data match is set as a playback position that becomes a so-called cue point when the shot moving image is reproduced by the playback position setting unit 103. Set.

The vector data received by the receiving unit 101 is converted into text data by the text data generation unit 104, and as will be described later, the search unit 105 searches for text data that matches the keyword specified under a predetermined condition.

The vector data corresponding to the searched text data is changed to a display that can be distinguished from other vector data by the display change unit 106. That is, the display changing unit 106 performs a process of changing the vector data to highlighting data, and specifically, for example, highlight display, enlarged display, reverse display, or the like may be used.

The display unit 107 displays the video data on the monitor so that the video data can be reproduced and the vector data can be viewed in comparison with each other. The video data corresponds to the transmission time data set as the playback position by the playback position setting unit 103. By designating the vector data to be generated on the display unit 107, the shooting time data matching the transmission time data can be reproduced as a cue point. At this time, the display unit 107 also displays the vector data changed to the highlighted data.

In the handwritten information processing device 1 according to the present invention, the utterance of the speaker is completed, the moving image data can be viewed as feedback information as described above, and real-time such as streaming playback during the utterance is in progress. Even during playback, the above-mentioned browsing becomes possible. In this case, the vector data received by the receiving unit 101 is sequentially displayed in chronological order on the monitor of the display unit 107, changed to the sequentially highlighted data by the display changing unit 106, and displayed, and the moving image data reproduced in real time. Is displayed so that it can be viewed in comparison.

Hereinafter, the specific operation of the handwritten information processing apparatus 1 will be described using the processing flows of FIGS. 1 and 2.

When vector data is received from the digital pens P1, P2 ... Pn (S1), in the case of moving image data processing (Y in S2), the moving image data stored in the moving image data storage unit 102 is read out (S3). The reproduction position setting process is performed from the transmission time data and the shooting time data associated with the received vector data (S4).

On the other hand, in the case of not moving image data processing (N in S2), the received vector data is generated as text data (S5). As a specific method for generating text data, for example, OCR (Optical Character Recognition), which optically reads a character and collates it with a pre-stored pattern to specify the character, may be used.

For the generated text data, the search unit 105 determines whether to execute the search with the set keyword (S6), and when searching with the set keyword (Y in S6), the set keyword is used. Perform a search (S7). Here, the keyword is set if the speaker's intention of utterance is understood and evaluated so that the speaker can grasp the reaction of the viewer to the utterance content by the handwritten information obtained from the viewer. , Naturally done by identifying the words that are expected to be listed. Further, at least a word that should be avoided from the description may be set as a keyword from the intention of the speaker's utterance. The search of the search unit 105 is performed based on whether or not the keyword matches the set keyword. Here, the predetermined condition includes a condition that completely matches the set keyword, as well as similar words and synonyms. It may be a condition to be extended, and a condition to be an opposite word or an opposite word to the set keyword. In the case of a condition other than an exact match, the search unit 105 may be provided with a dictionary function such as a thesaurus, and the search target may be determined according to the condition.

When a hit is made in the set keyword search (Y in S8), the display changing unit 106 performs a process of highlighting the vector data corresponding to the hit keyword (S9). If no hit is found in the search (N in S8), the process returns to S7 and the search for the set keyword is looped.

On the other hand, when a search is performed with a new keyword instead of the set keyword (N in S6), a search is executed with the new keyword set (S10), and a hit is made in such a search (Y in S11). A new highlighting process is performed (S12). Here, the new highlighting is processed with a highlighting different from the highlighting after the search for the set keyword. The new keyword can be set when the speaker needs to confirm the understanding and evaluation of the viewer while the utterance is in progress, and after the search for the new keyword, real-time playback is performed. In that case, the new highlighting will be reflected. If there is no hit in the search (N in S11), the process returns to S10 and the search for a new keyword is looped.

When the playback position of S4 is set and the highlighting process of S9 or S12 is performed, the moving image data and the vector data are displayed in comparison on the display unit 107, and when viewing as feedback information after the end of the speech, the playback position (cue point). By specifying any of the vector data that becomes), the moving image can be reproduced from the shooting time data that matches the transmission time data of the vector data, and if the speech is in progress, the moving image can be reproduced in real time. , Vector data is displayed sequentially in chronological order, and sequentially highlighted data is also displayed.

FIG. 3 shows an example of an entry form F as an entry medium such as a digital pen P1. Specifically, Fig. 3 was created based on the lesson progress plan of the school teacher, and it is evaluated whether the lesson of the teacher who is the speaker is proceeding according to the progress plan. Is for the evaluator who is the viewer (for example, a veteran teacher) to describe.

The entry form F is a designated item field Fa (“learning flow”) in which designated items for which handwritten information should be described are described in chronological order, and a handwritten information entry field Fc (“learning flow”) in which the reason for evaluation is described corresponding to the designated item. “Reason for evaluation”) is particularly important, but in the present embodiment, the evaluation column Fb (“evaluation”) in which the evaluation is indicated by a stepwise symbol and the weighting of the importance of the designated item are also described. It is configured to include the priority column Fd (“priority”).

By making the entry form F have such a configuration, the evaluator only needs to write the evaluation by hand according to the designated items in the time series, and as in the conventional case, the dedicated device distributed for the evaluation There is no need to bother with the operation, and there is no confusion such as the order of writing being changed in chronological order as in the case of writing on a completely blank sheet, and it becomes possible to focus on observing the evaluation. Also, for the speaker side, the items to be described are organized to some extent in a unified manner among the evaluators according to the above-mentioned designated items, so that it is possible to relatively encourage the description of keywords, and the lesson (utterance) is efficient. Progress and feedback information can be obtained.

In the present embodiment, the designated items describe the order in which the handwritten information should be described in chronological order, but the evaluation as a whole (for example, "loudness", "motion", etc.), etc. There may be some items that cannot be described in chronological order. Such designated items may be described in one place, such as the last column of the entry form F (not shown). Since the items other than the time-series items are provided in one place on the viewer side, the description positions are not scattered and the description can be performed smoothly.

FIG. 4 is a block configuration diagram of the handwritten information processing apparatus 1 when a predetermined entry form F as shown in FIG. 3 is used. Common symbols and numbers will be used for the components that have the same functions and functions as those in FIG. 1, and duplicate detailed explanations will be omitted for the matters described in FIG.

As shown in FIG. 3, when at least the entry form F having the time-series designated item field Fa and the corresponding handwritten information entry field Fc is used, the designated items and the vector are displayed when the vector data is displayed on the display unit 107. A display list generation unit 108 for generating a display list in which data is displayed corresponding to the data may be provided, and the generated display list may be displayed on the display unit 107.

The display list generation unit 108 has coordinate data for specifying the position of the designated item field Fa in advance, and when the receiving unit 101 receives the vector data converted from the handwritten information, the corresponding designated item is obtained from the coordinate values of the vector data. By specifying the coordinate data of, the designated item data and the vector data are arranged at the corresponding positions, and the transmission time data can also be acquired from the received vector data. Therefore, the transmission time data can be obtained from the designated item data and the vector data. Generate a display list by arranging with.

FIG. 5 is a diagram showing a screen display example by the display unit 107 including the display list. The screen display V is composed of at least a display list unit Va, a moving image display unit Vb, and an evaluation total display unit Vc. The display list part Va is displayed in a scrollable manner, and in the case of real-time playback, when the latest vector data (evaluation) is added, for example, this is displayed at the bottom and automatically scrolled in sequence. Scroll. In the present embodiment, the items displayed in the display list unit Va are arranged in the horizontal direction such as vector data (handwritten information) and designated item data corresponding to the transmission time data, and are displayed on the screen. , Each item is a cue point in the corresponding transmission time data, so if you specify any item on the screen, the moving image data will be played back from the shooting time data associated with the corresponding transmission time data. can do.

In the present embodiment, the evaluation total display unit Vc exemplifies a graph display in which the evaluations of the evaluation column Fb in FIG. 3 are totaled in stages. The horizontal axis is time, the vertical axis is the aggregated value of evaluation (by time), and a line graph with different colors for each evaluation type is shown. Here, the type of evaluation means, for example, each stage when a symbol of a plurality of stages is set for the judgment of good or bad in the evaluation column Fb. In the graph of the evaluation summary display unit Vc, the time on the horizontal axis of the graph is a cue point, and when this is specified on the screen, the moving image display unit Vb reproduces the moving image data from that time. ..

FIG. 6 is an enlarged view showing an example of the display list unit Va among the screen display examples of FIG.

Va1 is the evaluation time, Va3 is the evaluation, Va4 is the evaluator number, Va5 is the designated item, Va6 is the priority, and Va7 is the column where the reason for evaluation is displayed. Note that Va2 is a sort button based on the evaluation time Va1, and the ascending order, descending order, and other sorting methods can be selected from the pull-down menu. The sort buttons are also provided in Va3, Va4, Va5, and Va6 so that sorting can be performed based on each column.

By the way, when the number of evaluators is large or when there are many designated items, the listability of the display list may be hindered. Therefore, the display list generation unit 108 has a function for making the generated display list easy to see as follows.

First, the evaluation time 05:40:00 (evaluator A3), the evaluation time 05:40:03 (evaluator A1), and the evaluation time 05:40:55 (evaluator A2) differ in both evaluation and evaluation reason. The specified items are common to the "product outline description". As in this case, when the specified items are common and the evaluation times are close to each other (when the time is within a predetermined range), it is possible to display H1 by enclosing and grouping three items. is there.

Next, in the priority column of Va6, the "product spec" in which the designated item is weighted as priority 1 can be displayed H2 more prominently than the description in the other evaluation reason columns. ..

In addition, when highlighting by keyword search, the evaluation time is 08:16:45 (evaluator A3), the evaluation time is 08:16:57 (evaluator A1), and the evaluation time is 08:10:00 (evaluator A2). As described above, when the designated item "comparison of products of other companies" is evaluated by the common keyword "clear" for all three parties, it is possible to set the display H3 to integrate the evaluations of each of the three parties into one. In the present embodiment, the display of one evaluator is displayed as a representative display, but the purpose is not limited to this.

In principle, the display list unit Va in FIG. 6 is in a format of displaying all the evaluation reasons, but may display only a specific evaluation reason. Hereinafter, a mode in which only a specific evaluation reason is displayed will be described with reference to FIGS. 7 and 8.

When there are many evaluators, or when only important evaluation reasons are viewed, feedback information can be efficiently viewed by displaying only specific evaluation reasons. In the present invention, only those keywords having the same priority and the same reason for evaluation that exceed a predetermined threshold value are displayed.

FIG. 7 is a processing flow of the display list generation unit 108 when displaying only a specific evaluation reason. From the items described in the entry form F, the column of priority or evaluation reason is specified, and i = 0 is set as the initial value for calculating the threshold value (S101). Here, the setting of the initial value is a function that can be selectively set when all the evaluation reasons are displayed in principle as shown in FIG. Specifically, an edit button or the like capable of the selective setting may be provided on the display screen as shown in FIG. 5 by the display unit 107 of the handwriting information processing apparatus 1.

Next, each entry form F is searched for the description of the priority or the reason for evaluation (S102). The text data is generated from the vector data before the search.

As a result of the search, it is determined whether or not the number of hit items in the search is less than the set threshold value (S103). In the present embodiment, the threshold value is set to 3, but the purpose is not limited to this. When the number of hit items in the search is less than the threshold value (Y in S103), it is determined whether or not there is still a search target (S104), and when there is still a search target (Y in S104), the above i The value is incremented as i = i + 1 (S105), and the value is returned to the search process of S102 again. After that, in the determination of S103, the processes of S102 to S105 are looped until the threshold value is reached. In the determination of S104, when the search is completed to the end (N of S104), it is determined that the item is less than the threshold value, and the item is hidden in the display list unit Va (S106).

On the other hand, in S103, when the threshold value is exceeded (N in S103), it is determined as a matter to be displayed in the display list unit Va (S107). Items to be hidden and displayed are determined in S106 and S107, and a display list unit Va is generated based on this (S108).

FIG. 8 is a diagram showing an example of a display list unit Va displaying only the evaluation reasons above the threshold value. Common symbols and numbers will be used for the components that have the same functions and functions as those in FIG. 6, and duplicate detailed explanations will be omitted for the matters described in FIG.

FIG. 8 is an example of a display list in which a doctor's mock interview is evaluated. At the evaluation times 01:53:00, 01:53:09, and 01:53:20, the keyword "critical" is common in the description of the reason for evaluation. Since the keyword is described by all three evaluators A1 to A3, it exceeds the threshold value and is displayed in the evaluation reason Va7.

Further, at the evaluation times 07:18:33, 07:20:22, and 07:25:15, the priority is 1, which is equal to or higher than the threshold value, and is displayed in the evaluation reason Va7.

Since the designated items "course", "primary medical history", "past medical history", and "family history" other than the above were below the threshold value, the reason for evaluation is hidden.

Since the display through the determination of the threshold value is for making it easy to see at the time of browsing, it may be any one that meets the purpose and is not limited to the display method shown in FIG. For example, in FIG. 8, both the priority and the reason for evaluation are displayed, but either one may be selected. Further, even if the low priority is equal to or higher than the threshold value, it may not be displayed as a precondition. Further, in FIG. 8, as a display method, all the evaluation reasons are simply listed, but each is displayed on a plurality of layers, superimposed and composited, and displayed like a page turn by clicking on the screen or the like. You may try to do it. In addition, in order to link with this, the evaluation time Va1, the evaluation Va2, the evaluator number Va3, the designated item Va4, and the priority Va5 are also displayed on multiple layers, and the top surface is always the item described by one evaluator. Only may be displayed.

FIG. 9 is a device configuration diagram when the communication terminals D1, D2 ... Dn that receive the vector data transmitted from the digital pens P1, P2 ... Pn of FIG. 1 are used. In the case of this configuration, the viewer A'transmits the handwritten information to the communication terminals D1, D2 ... Dn using the digital pens P1, P2 ... Pn and displays them on the display unit of the communication terminals D1, D2 ... Dn. .. The items to be described other than the handwritten information are directly input on the communication terminals D1, D2 ... Dn. The communication terminals D1, D2 ... Dn and the handwritten information processing device 1 are connected so as to be capable of data communication, and the handwritten information described by the digital pens P1, P2 ... Pn and the direct input by the communication terminals D1, D2 ... Dn. The collected items are collectively transmitted as transmission data from the communication terminals D1, D2 ... Dn by data communication, and are received by the receiving unit 101 of the handwriting information processing apparatus 1. In the present embodiment, the communication terminals D1, D2 ... Dn and the handwriting information processing device 1 are connected by a wireless LAN via the router R, but the present invention is not limited to this.

FIG. 10 is a diagram showing a transmission screen display example when the transmission data is transmitted on the display unit of the communication terminal D. Da is a moving image data display unit. The communication terminal D can receive the moving image data captured by the video camera C connected to the handwriting information processing device 1 by data communication and display it in real time. Therefore, the viewer using the communication terminal D can describe the handwritten information while viewing the moving image data in real time. In the present invention, when the handwritten information is described, the superimposed mark data De is synthesized in the portion by touching the portion related to the handwritten information on the screen displaying the moving image data. Has the function of being able to. A known image synthesis technique may be used as a technique for superimposing data synthesis by touching moving image data or the like. In the present embodiment, attention is paid to the orientation of the teacher's face, and by touching the face part of the moving image data, an image in which the superimposed mark data De that surrounds the entire face with a circle is synthesized is displayed. There is.

Db is vector data of handwritten information described on the entry form. The vector data Db displays the description related to the superimposed mark data De. Further, in Dc under the vector data Db, a data set of the superimposed mark data De, the evaluation time data of the vector data Db, the evaluation data, and the designated item data is displayed. The data set Dc may be set in advance in the communication terminal D and may be generated by checking with a radio button, a check button, or the like (not shown).

After confirming the superimposed mark data De, vector data Db, and data set Dc to be transmitted on the screen of the communication terminal D, when the transmission button Db is pressed, these data are transmitted to the handwritten information processing device 1 via the router R. It is received by the receiving unit 101.

In FIG. 10, the superimposed mark data is directly synthesized on the moving image data displayed in real time. However, for example, when the communication environment is poor, the handwritten information and the superimposed mark data are described at the same time. When it is difficult to do so, the frame image data corresponding to the time when the handwritten information is entered is extracted from the moving image data, and the moving image data display unit Da displays the handwritten information together with the extracted frame image. The data may be displayed, and the displayed frame image data may be entered so as to synthesize the superimposed mark data at a portion related to the content of the handwritten information. It should be noted that whether the superimposed mark data is combined on the moving image data or the frame image data may be selected in advance on the communication terminal D (not shown).

When combined on the frame image data, the receiving unit 101 receives the frame image data in which the vector data and the superimposed mark data are combined from the communication terminal D, and specifies the vector data as the playback position. , The display unit 107 reproduces and displays the moving image data from the position corresponding to the frame image data in which the superimposed mark data is combined.

The communication terminal D may have the same function as the handwriting information processing device 1 when the communication terminal D is played back to acquire the feedback information after the utterance is completed. Further, the transmitted superimposed mark data De, vector data Db, and data set Dc can be edited after the fact from the communication terminal D, and have a function of synchronizing data with the handwritten information processing device 1. May be good.

By the way, in the case of mutual participation type active learning in which a plurality of groups consisting of a plurality of viewers (for example, students) are formed and discussions are held with the speaker (for example, a teacher), the viewer responds to the speaker's request. However, when the handwritten information processing device 1 is used in such a case, it is effective for learning guidance if the description contents of each viewer can be listed for each group.

In the present invention, when the viewers form the above-mentioned group, the moving image data is photographed for each group, and when each viewer describes the handwritten information, the receiving unit 101 of the handwritten information processing apparatus 1 may display the handwritten information. The vector data and transmission time data of the handwritten information described by each viewer are received together with the identification data that identifies each group, and the moving image data storage unit 102 stores the moving image data of each group together with the identification data. It was configured. Since the vector data, the transmission time data, and the moving image data are associated with the identification data, the display unit 107 can distinguish the vector data and the moving image data for each group and display them in a list. Become.

Further, when forming the group, as shown in FIG. 11, for example, a group handwriting information processing device 2 having the same function as the handwriting information processing device 1 in three groups of group G1, group G2, and group G3. Vector data and transmission time obtained by converting handwritten information described by a plurality of viewers belonging to each group (4 people in the present embodiment, but not limited to this) are assigned to 3 to 3 respectively. The data is transmitted to the handwritten information processing device 1 via the router R connected by the wireless LAN, and is also transmitted to the handwritten information processing devices 2 to 3 for each group. Cameras C2 to C4 for capturing moving image data for each group are connected to the handwritten information processing devices 2 to 3 for each group.

The group handwritten information processing devices 2 to 3 shall display only the moving image data captured by the cameras C2 to C4 and the vector data related to the handwritten information written on the entry form F in the group on the display unit. Only the handwritten information processing device 1 can view all of the vector data and moving image data of each group G1 to G3.

In the configuration as shown in FIG. 11, when the vector data displayed by the group on the display unit 107 of the handwriting information processing device 1 includes highlighting data, the handwriting information processing device 1 is displayed as highlighted data as shown in FIG. Is configured to include a transmission unit 109 for transmitting the above to the group handwritten information processing devices 2 to 3, so that the highlighting data is also displayed in each of the groups G1 to G3. That is, as shown in FIG. 13, the keyword K (the keyword K () is displayed on the display unit 107 (monitor) of the handwriting information processing apparatus 1 in a state where the moving image data and the vector data of the groups G1 to G3 are displayed for each group. In the present embodiment, when "Hello") is input, the "Hello" part of the viewer's vector data in which "Hello" is described in each group is highlighted, and when the send button T is pressed, the handwriting for the group is performed. Within the range allowed by the display units of the information processing devices 2 to 3, that is, the vector data of the viewers belonging to the group is highlighted. In the present embodiment, the display state of the group handwriting information processing apparatus 3 of the group G2 is exemplified.

14 and 15 are block configuration diagrams showing a modified example of the handwritten information processing apparatus 1 in the usage mode when the viewers described in FIGS. 11 to 13 form a group. Common symbols and numbers will be used for the components that have the same functions and functions as those in FIGS. 1, 4 and 12, and the matters described in FIGS. 1, 4 and 12 will be duplicated in detail. The explanation is omitted.

FIG. 14 is a diagram showing a configuration having a storage unit 110 that stores vector data of handwritten information of the same group or the same viewer as log data including highlighting data.

By reading out the log data accumulated in the storage unit 110, the display unit 107 can display the log data of the same group or the same viewer in a listable manner and display the transition of the highlighted data in a listable manner. it can. Therefore, in the present embodiment, it is assumed that the same viewer writes the handwritten information a plurality of times.

The transition of highlighted data can be displayed even when the transition of the description timing of different keywords is tracked by sequentially adding new keywords in addition to viewing the log data that follows the same keyword. Is.

The method of displaying log data is not particularly limited as long as it can be displayed in a list, for example, one that is displayed on one screen by a thumbnail method, one that is stacked in a hierarchical manner and sequentially fed.

FIG. 15 is a diagram showing a configuration having a specific unit 111 that specifies the speech count data of each viewer as speech data associated with the viewer ID data and speech time data specified at the time of speech.

Specifically, for example, a peripheral device or the like that transmits ID information that uniquely identifies the viewer at the time of speaking to the specific unit 111 may be given to the viewer (not shown). For peripheral devices, in addition to a dedicated remote controller such as a clicker, a remark icon indicating the viewer ID is printed in advance on the entry form F distributed to each viewer, and when speaking, tap on the remark icon with the digital pen P. Then, the ID information may be transmitted to the specific unit 111.

When the speech data is acquired by the specific unit 111, it is accumulated by the storage unit 110. Further, the reproduction position setting unit 103 sets the speech data as the reproduction position, and when the display unit 107 reads the speech data from the storage unit 110, the speech data is reproduced in chronological order and the speech being reproduced. The data may display the number of all remarks.

FIG. 16 is a device configuration diagram for using the handwritten information processing device 1 when the speaker and the viewer are at a remote location. When handwritten information is written on the entry forms F1, F2 ... Fn with the digital pens P1, P2 ... Pn, the vector data is received by the handwritten information processing device 1 via the remote communication relay device M connected to the Internet I. Received by unit 101. Further, the highlighting data transmitted from the transmission unit 109 of the handwritten information processing device 1 is displayed on the group handwritten information processing devices 2 to 3 described with reference to FIG. 11 via the remote communication relay device M (FIG. Although not shown in 16, the remote area communication relay device M and the group handwritten information processing devices 2 to 3 are freely connected to each other for data communication). Further, the moving image data storage unit 102 acquires and stores the moving image data obtained by photographing the viewer via the remote communication relay device M connected to the Internet I.

1 Handwritten information processing device 2-4 Handwritten information processing device for groups
101 Receiving unit 102 Moving image data storage unit 103 Playback position setting unit 104 Text data generation unit 105 Search unit 106 Display change unit 107 Display unit 108 Display list generation unit 109 Transmission unit 110 Storage unit 111 Specific unit

Claims (14)

A handwritten information processing device provided with a display means capable of allowing a plurality of viewers to view handwritten information described in a predetermined entry medium using a digital pen for the speaker.
A receiving means that converts the handwritten information described by the plurality of viewers into vector data based on the coordinate values set in the entry medium and collects the handwritten information corresponding to the transmission time data.
A moving image data storage means for storing moving image data including audio obtained by shooting the speaker or the viewer together with shooting time data, and
A reproduction position setting means for the reproduction position of said received transmission time data and the photographing time data and moving image data of the time said stored matching by said receiving means,
Data generating means for generating text data from the vector data received by said receiving means,
A search means for searching text data that matches a keyword specified by a predetermined condition from the generated text data, and
It has a display changing means for changing the vector data corresponding to the searched text data into a display that can be distinguished from other vector data to be highlighted data.
The predetermined entry medium includes at least items to be described in time series in advance according to the progress of speech as designated items for describing the handwritten information, and each designated item and a corresponding handwritten information description column. Is specified by the coordinate data and provided, and when handwritten information is described in each of the entry fields according to the progress of the speech, the receiving means receives the converted vector data and the corresponding designated item data. A display list is displayed in which the transmission time data and the designated item data are associated with the vector data that is specified and received by the coordinate data and displayed in comparison with the moving image data by the display means. Equipped with a display list generation means to generate
The display means, specifying a vector data corresponding to the transmission time data set as the playback position, and the vector data including the highlight data, the reproduction of the moving image data corresponding thereto, a comparison viewable in displayed, when real-time playback of the moving image data, the vector data is sequentially displayed on the chronological received by said receiving means, as well as changes to sequentially highlight data vector data by the display changing means, the moving image data A handwritten information processing device characterized by displaying in a viewable manner in contrast to real-time playback. When the speaker specifies a new keyword while the utterance is in progress, the display changing means is different from the highlighting data in which the vector data corresponding to the text data matching the new keyword is changed in advance. The handwritten information processing apparatus according to claim 1, wherein the data is changed to highlighted data. The display list generation means provides a symbol column indicating the weighting of each designated item in the display list, and makes vector data corresponding to the column in which a symbol indicating a predetermined weighting is described more remarkable than other vector data. The handwritten information processing apparatus according to claim 1 or 2 , characterized in that the display is simple. From the plurality of vector data collected by the receiving means, the display list generating means sets the designated items to be the same and the transmission time data within a predetermined range to be grouped. The handwritten information processing apparatus according to any one of claims 1 to 3 . The display list generating means of claims 1 to integrate vector data having the keyword in common to generate a single data, and changes it to the highlighted data to claim 4 The handwritten information processing apparatus according to any one item. The display list generation means displays, at least, for the same designated item, only the number in which the same weighting symbol is described or the number of vector data having the same keyword in common is equal to or greater than a predetermined number. The handwritten information processing apparatus according to claim 5 , wherein the information processing apparatus can be selected. The receiving unit is configured to receive vector data from the digital pen, it has a real-time reproducible display unit receives the previous SL moving picture data, the moving picture data displayed on the display unit, the handwritten at locations related to the content of the information, the fill capable communication terminal so as to synthesize the superimposed mark data, receives the superimposed mark data to correspond to said vector data and the transmission time data, vector data as the playback position The handwritten information processing apparatus according to any one of claims 1 to 6 , wherein the display means reproduces and displays moving image data in which the superimposed mark data is combined. The receiving unit is configured to receive vector data from the digital pen, it has a real-time reproducible display unit receives the previous SL moving image data from the moving image data, corresponding to the time that filled the handwritten information The frame image data to be extracted is extracted, the extracted frame image data is displayed together with the handwritten information on the display unit, and a superimposed mark is placed on the displayed frame image data at a place related to the content of the handwritten information. data from the entry capable communication terminal so as to synthesize and the vector data and the superimposed mark data receives the frame image data are combined to specify the vector data as the playback position, the display means, the superposition The handwritten information processing apparatus according to any one of claims 1 to 6 , wherein the moving image data is reproduced and displayed from a position corresponding to the frame image data in which the mark data is synthesized. It is assumed that a plurality of groups composed of a predetermined number of viewers for describing the handwritten information are formed, and the moving image data is photographed for each group.
When the viewers of each group describe the handwritten information on the entry medium in response to the request of the speaker, the receiving means receives the vector data and the transmission time data together with the identification data for identifying each group. The moving image data storage means stores the moving image data of each group together with the identification data, and the display means distinguishes vector data and moving image data for each group and displays them in a list, and also displays the group. The handwritten information processing apparatus according to any one of claims 1 to 8 , wherein if the vector data to be displayed for each includes highlighting data, the highlighting data is displayed . The vector data and the moving image data of each group displayed by the display means are only the vector data and the moving image data in the group even in the display unit of the handwriting information processing device for the group assigned to each group. Is displayed and
To display the highlighted data on the display unit of the previous SL handwritten information processing apparatus for the group that is assigned to each group, characterized in that it has a transmission means for transmitting the highlighted data on the handwriting information processing apparatus for the group 9. The handwritten information processing device according to claim 9. Before Among Kigu loop or a predetermined number of viewers, have a storage means for storing vector data of the same group or handwritten information of the same viewer <br/>, as log data including the highlight data and, the display means, along with a list allows displaying the log data of the same group or the same viewer, claim 9 or claim, characterized in that the list can display the transition of highlight data Item 10. The handwritten information processing apparatus according to item 10 . The storage means has a specific means for specifying the speech count data of each viewer as speech data associated with the viewer ID data and the speech time data specified at the time of speech, and the storage means stores the speech data and reproduces the data. The position setting means sets the remark data as a reproduction position, and when the display means reads the remark data from the storage means, the remark data is reproduced in chronological order, and all the remark data being reproduced is displayed. The handwritten information processing apparatus according to claim 11, wherein the number of remarks is displayed. Said receiving means receives via the remote communication relay device connected to the Internet, the moving image data storage means, said remote the moving image data obtained by photographing the viewer is connected before Symbol Internet The handwritten information processing device according to any one of claims 1 to 12 , wherein the handwritten information processing device is acquired and stored via a communication relay device. The transmission unit via the remote communication relay device connected to the Internet, the highlight data to be displayed on the display unit of the handwritten information processing apparatus for the group, and transmitting the highlighted data handwritten information processing apparatus according to claim 10 Symbol mounting features.