JP2005284490A - Data recording device and data regenerating device - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To record board written contents, images and voice in a semantically structured form at the end of a class in the class or the like started from a state of no data displayed beforehand. <P>SOLUTION: This data recording device is further provided with: an instruction presence detecting part 101 for detecting the presence of instructions to a two-dimensional display; a position specifying part 102 for specifying a coordinate position on the two-dimensional display of the instructions detected by the instruction presence detecting part 101; a region specifying part 103 for specifying a region including the specified coordinate position based on the coordinate position specified by the position specifying part 102; a stream data input part 104 for inputting stream data including the voice; and a data recording part 107 for recording link corresponding information of the region and a division position by the stream data input part 104. <P>COPYRIGHT: (C)2006,JPO&NCIPI

Description

  The present invention relates to a data recording device and a data reproducing device, and in particular, in a class or the like, by automatically associating two-dimensionally displayed board data and stream data including sound, the subsequent editing is unnecessary. The present invention relates to a data recording apparatus and a data reproducing apparatus capable of

  Conventionally, there is a method of saving a lesson scene with a video camera as one method for saving lesson contents. However, the video stored in this way has many redundant parts such as a teacher writing on a long class, and the truly necessary parts are limited. Also, it takes time and effort to find a part that you want to hear again from a series of images. Therefore, it was necessary to edit the video after class and put it together.

Moreover, a meeting recording system can be mentioned as a system for storing lesson contents. In Patent Document 1, the contents explaining each slide and the video and audio data corresponding to the question are divided based on the slide switching operation as a trigger based on the prepared presentation materials and separated from each slide. Methods for associating video and audio data have been proposed. This allows the presentation to be automatically and structurally saved. At the time of reproduction, the material, video and audio can be reproduced in synchronization with each other, and the contents necessary for presentation can be almost reproduced on the PC.
JP 2002-109099 A

  As described above, the conventional conference recording system is applied when presentation materials such as a presentation are prepared in advance. Video and audio data can be recorded in real time and structure based on the structure (slide) of the material. However, if the lesson does not have materials to be displayed in advance, such as writing on a board at elementary and junior high schools, the underlying structure (slides, etc.) does not exist as a shape at the start of the lesson. For this reason, since the trigger for structurally separating the video and audio is not clear, the board writing content and the video and audio cannot be recorded structurally in real time.

  The present invention has been made in order to solve the above-mentioned problems, and means that the contents of the written book, the video, and the sound are at the end of the class in a class or the like that starts from a state where there is no pre-displayed material. The purpose is to record in a structured way.

  A data recording apparatus according to an aspect of the present invention includes an instruction presence / absence detecting unit for detecting the presence / absence of an instruction for two-dimensional display, and a coordinate position of the instruction detected by the instruction presence / absence detecting unit on the two-dimensional display. Position specifying means for specifying, area specifying means for specifying an area including the specified coordinate position based on the coordinate position specified by the position specifying means, and stream data including audio A stream data input unit; and a voice position specifying unit for detecting the presence or absence of sound on the stream data input by the stream data input unit and specifying a voiced part as a voice position. The segment position on the stream data can be specified. The area specified by the area specifying means and the utterance position specifying means are specified. Link correspondence specifying means for associating the division position, a plurality of specified areas, coordinate positions associated with each specified area, stream data, and areas associated with the link correspondence specifying means Recording means for recording link correspondence information with a position is further provided.

  Preferably, the instruction presence / absence detecting means detects the instruction, and when the distance the instruction has moved on the two-dimensional display is a predetermined value or more, the first state, and when the distance is less than the predetermined value, the second state When the state and the instruction are not detected, it is determined as the third state, and the recording unit controls the recording process of the specified area based on the first to third states.

  Preferably, when the instruction presence / absence detecting means determines that the instruction is in the first state and the coordinate position of the instruction specified by the position specifying means is not included in the existing area, the area specifying means Create a new area.

  Preferably, when the instruction presence / absence detecting means determines that the instruction is in the first or second state and the coordinate position of the instruction specified by the position specifying means is included in the existing area, the area specifying The means performs control by selecting the existing area.

  Preferably, the link correspondence specifying unit specifies an instruction start time when the first or second state instruction is first detected for the region specified by the region specifying unit, and specifies the utterance position at the instruction start time. Based on whether or not the utterance position specified by the means is detected, a segment position to be associated with the specified region is searched.

  Preferably, the link correspondence specifying unit specifies an instruction start time when the first or second state instruction is first detected for the region specified by the region specifying unit, and specifies the utterance position at the instruction start time. When the utterance position specified by the means is detected, the section position for associating with the specified area is searched by going back in time.

  Preferably, the link correspondence specifying unit specifies an instruction start time when the first or second state instruction is first detected for the region specified by the region specifying unit, and specifies the utterance position at the instruction start time. When the utterance position specified by the means is not detected, the time is advanced to search for a segment position to be associated with the specified area.

  Preferably, the link correspondence specifying unit is configured such that the segment position associated with the first region identified by the region identifying unit and the segment position associated with the second region temporally identified next by the region identifying unit. The utterance position existing between the two is associated with the first region.

  Preferably, the link correspondence specifying unit combines the two regions into one region when the division position associated with the region specified by the region specifying unit is already associated with the existing region.

  Preferably, the link correspondence specifying unit does not detect the segment position for associating with the region specified by the region specifying unit, and when the region specifying unit specifies another region, the former region includes the associated segment position. Shall not exist.

  Preferably, the link correspondence specifying means does not have any associated region for the remaining utterance positions that cannot be associated with any region.

  A data reproducing device according to another aspect of the present invention includes a plurality of regions on a two-dimensional display, coordinate positions associated with each region, stream data including audio, and regions and segment positions on the stream data. Two-dimensional display for transmitting a signal for displaying recording means for storing associated link correspondence information, a plurality of areas stored in the recording means, and coordinate positions associated with each area A reproduction means, a signal for designating a coordinate position on a two-dimensional display, a position designation means for specifying a region including the coordinate position, and a link correspondence information stored in the recording means Link correspondence analyzing means, the link correspondence analyzing means specifying the segment position on the stream data associated with the area specified by the position specifying means, and storing it in the recording means. The stream data reproduction means further transmits a signal for outputting the stream data, and the stream data reproduction means transmits a signal for outputting the stream data corresponding to the segment position specified by the link correspondence analysis means. .

  According to the data recording apparatus of the present invention, it is possible to automatically specify the region from the coordinate position on the two-dimensional display and automatically specify the utterance position and the segment position of the stream data including audio. In addition, the identified area and the segment position on the stream data can be associated in real time, and subsequent editing can be made unnecessary.

  Furthermore, according to the data reproducing apparatus of the present invention, it is possible to reproduce the stream data corresponding to the utterance position associated with the area by specifying the area from the coordinate position on the two-dimensional display. Therefore, only necessary portions can be taken out and reproduced without trouble.

  Embodiments of the present invention will be described in detail with reference to the drawings. In the drawings, the same or corresponding parts are denoted by the same reference numerals and description thereof will not be repeated.

  First, an outline of the embodiment of the present invention will be described.

  A class usually consists of a blackboard part and an utterance explanation part. The board writing part is a collection of characters and drawings, and the utterance explanation part is a voice or action for supplementing the contents of the board writing part. Therefore, it is considered that the board writing part and the utterance explanation part can be recorded in a semantically related manner. Then, based on the data recorded in association with each other, the lesson can be reproduced by viewing the postcard part and the utterance explanation part at the same time later.

  In order to associate these semantically, it is necessary to semantically divide a board that starts from a blank state and to semantically divide the description of video and audio. However, it is difficult to analyze and categorize the semantic content of the board and explanation. Therefore, based on the behavior pattern of the teacher who conducts the lesson, a position that can be classified semantically is found, and the divided board writing part and the explanation part are associated with each other.

  When a teacher writes on a board, meaningful letters / sentences are written continuously on the board. And if you want to go to the next, start writing on the board again. In this way, the teacher thinks that there is a natural break in the series of board writing actions.

  This board is composed of a combination of words. This word is composed of a combination of characters, and this character is composed of a combination of lines. In other words, I think that a board can be expressed by a combination of lines. However, the combination of lines is a formal structure and has no semantic structure.

  Based on these, based on the teacher's behavior as described above, a set of lines, which is a formal structure based on a series of board writing, is collected into an arbitrary area. In this way, the entire board is divided into a plurality of areas to make it semantically structured.

  Stream data such as video / audio is segmented and recorded in association with each region, triggered by an operation of segmenting such a series of board writings for each region.

  At the time of reproduction, by selecting an arbitrary area, it is possible to view the explanation related to the area, and it is possible to take out only a necessary part without trouble.

  Next, with reference to FIG. 1 and FIG. 2, a data recording apparatus according to an embodiment of the present invention will be described.

  FIG. 1 is a functional block diagram of a data recording apparatus 100 according to an embodiment of the present invention. FIG. 2 is a conceptual diagram of data recording apparatus 100 in the embodiment of the present invention.

  Referring to FIG. 2, the data recording device 100 acquires a two-dimensional display device 11 and a position specifying device 13 that are used for a teacher to write on a board, a voice explaining a lesson content by a teacher, and a question voice by a student. The acquired input information is recorded by using the stream data input device 12 for this purpose.

  The two-dimensional display device 11 can electronically display and record characters or images like an electronic blackboard. The position designation device 13 can enter characters on the two-dimensional display device 11 or can specify an instruction target. The stream data input device 12 is for inputting a speaker's utterance, and is a microphone or a video camera.

  In addition, this lesson form may be a distance lesson where the teacher and the student are in a remote place. Moreover, not only a class but a lecture, a meeting, etc. may be sufficient. Further, the two-dimensional display device 11 is not limited to the electronic blackboard, and may be any device that can detect characters and the like.

  Referring to FIG. 1, a data recording apparatus 100 includes an instruction presence / absence detecting unit 101, a position specifying unit 102, a region specifying unit 103, a stream data input unit 104, a utterance position specifying unit 105, and a link correspondence specifying unit. 106 and a data recording unit 107.

  Hereinafter, each configuration in the data recording apparatus 100 will be described in detail.

  The instruction presence / absence detection unit 101 turns ON when the position specifying device 13 and the two-dimensional display device 11 are in contact with each other and turns OFF when not in contact with each other. If the distance that the position specifying device 13 has moved on the two-dimensional display device 11 is equal to or greater than a certain coordinate distance when the ON signal is detected, the operation is determined to be in the “write” state. Further, when the ON signal is detected and the distance that the position specifying device 13 has moved on the two-dimensional display device 11 is less than a certain coordinate distance, the operation is determined to be a “pointing” state. Further, the operation when the OFF signal is detected over a certain time is determined to be a “leaving” state.

  FIG. 3 is a flowchart showing the flow of the board writing operation state detection process executed by the instruction presence / absence detection unit 101.

  First, at the start of the lesson, since the teacher usually does not start writing on the board, the position specifying device 13 and the two-dimensional display device 11 are in the OFF state. Therefore, the instruction presence / absence detecting unit 101 sets the operation state to be detected to “leave” (step S11). Here, the instruction presence / absence detecting unit 101 waits until the position specifying device 13 and the two-dimensional display device 11 come into contact with each other and is turned on (NO in step S12).

  When the instruction presence / absence detecting unit 101 detects that the position specifying device 13 and the two-dimensional display device 11 are turned on (YES in step S12), the position specifying device 13 is turned on on the two-dimensional display device 11. It is determined whether the distance moved in the state is a certain coordinate distance or more (step S13). If it is detected that the position specifying device 13 has moved on the two-dimensional display device 11 by a certain coordinate distance or more (YES in step S13), the instruction presence / absence detecting unit 101 sets the operation state to be detected to “write” (step) S14). On the other hand, when it is not detected that the position designation device 13 has moved over the two-dimensional display device 11 by a certain coordinate distance (NO in step S13), the instruction presence / absence detection unit 101 sets the operation state to be detected to “point”. (Step S15).

  Thereafter, if the OFF state has elapsed for a certain time or longer (YES in step S16), the instruction presence / absence detection unit 101 sets the detected operation state to “leave” (step S11).

  Returning to FIG. 2, the position specifying unit 102 is positioned on the two-dimensional display device 11 when the instruction presence / absence detection unit 101 detects that the teacher's postcard operation state is “writing” or “pointing”. The coordinate information with which the designation device 13 is in contact is specified. When the “writing” operation is detected by the instruction presence / absence detecting unit 101, the position specifying unit 102 specifies continuous coordinate information collectively. If the “indication” operation is detected by the instruction presence / absence detection unit 101, the coordinate information of one point is specified.

  The area specifying unit 103 records the continuous coordinate information that can be acquired from the position specifying unit 102 by the “write” operation as an object that is a single line in the display area set. By repeating such “writing” operation, the board writing accumulates a plurality of objects. Here, if the instruction presence / absence detecting unit 101 detects a “write” operation, the region specifying unit 103 collects a plurality of objects as a collection and creates a region for specifying using the operation as a trigger. Then, it is assumed that the operation until the instruction presence / absence detection unit 101 detects the “leaving” operation is a series of operations on the board, and the region created until the “leaving” operation is detected is defined as one region. Identify. This area is a rectangle surrounded by the coordinates of the minimum position and the coordinates of the maximum position on the xy axis among the plurality of objects.

  Further, the area specifying unit 103 changes the area to be controlled by detecting the “write” or “point” operation from the instruction presence / absence detection unit 101. Here, when the coordinate information acquired from the position specifying unit 102 is included in the range of the existing area, control is transferred to the existing area. On the other hand, if the coordinate information acquired from the position specifying unit 102 is not included in the range of the existing area, a new area is created or added to the existing area under control to expand the area. Therefore, the area specifying unit 103 holds addition permission information for areas other than the area under control in order to determine these.

  FIG. 4 is a flowchart showing the flow of the area specifying process executed by the area specifying unit 103 of the data recording apparatus 100 according to the embodiment of the present invention.

  First, when the lesson is started, the region specifying unit 103 has no controlled region, and it is assumed that only the non-region portion is spread on the two-dimensional display device 11. Therefore, since it is possible to create an area in any part, permission to add to another area is issued (step S21). Here, it is determined what the writing / writing state is detected by the instruction presence / absence detection unit 101 (step S22).

  In step S22, when the operation state detected by the instruction presence / absence detecting unit 101 is “leaving”, the process returns to step S21, and permission to add to another area is given.

  In step S22, when the operation state detected by the instruction presence / absence detection unit 101 is “write”, it is first determined whether or not permission to add to another area is given (step S23). If it is determined that writing to another area is permitted (YES in step S23), whether or not the “writing” operation is performed in the unarea range from the coordinate position specified by the position specifying unit 102. Is determined (step S24). If it is determined that the area has not yet been set (YES in step S24), a new area is created (step S25). Then, control is transferred to the area (step S26), and writing to another area is disabled (step S27). The contents of the board written in the area are recorded in the display area set (step S28). Thereafter, the process returns to S22, and it is determined what the operation state detected by the instruction presence / absence detection unit 101 is.

  In step S24, if it is determined from the coordinate position specified by position specifying unit 102 that the “writing” operation has been performed on the existing area (NO in step S24), control is transferred to the existing area (step S24). S26). Similarly, writing to another area is disabled (step S27), and the contents of the board written to the area are recorded in the display area set (step S28).

  If it is determined in step S23 that writing to areas other than the area under control is impossible (NO in step S23), the “writing” operation is performed in the area under control from the coordinate position specified by the position specifying unit 102. It is determined whether or not there is (step S29). If it is determined that the area is an existing area other than the area under control (YES in step S29), the control shifts to the area where the “write” operation has been performed (step S26). Similarly, writing to another area is disabled (step S27), and the contents of the board written to the area are recorded in the display area set (step S28). On the other hand, if it is determined that the “writing” operation is performed in the controlled area (NO in step S29), the contents of the board written in the area are recorded in the display area set (step S28). Thereafter, the process returns to S22, and it is determined what the operation state detected by the instruction presence / absence detection unit 101 is.

  In S22, when the operation state detected by the instruction presence / absence detection unit 101 is “point”, whether or not the “pointing” operation is in the existing area from the coordinate position specified by the position specifying unit 102. Is determined (step S30). If it is determined that the area is an existing area (YES in step S30), control is transferred to the existing area (step S31), and writing to another area is disabled (step S32). On the other hand, if it is determined that the “pointing” operation is in the non-region range (NO in step S30), the process returns to step S22 to determine the next operation state.

  FIG. 5 is a diagram for explaining an example of processing executed by the area specifying unit of the data recording apparatus according to the embodiment of the present invention.

  In FIG. 5, among the data indicating the ON / OFF state of the position specifying device 13, a fixed elapsed time for determining that the operation state is “leaving” after the instruction presence / absence detecting unit 101 detects ON is indicated by a dotted line. Show.

  Further, the hatched portion of the frame of the control area representing the transition of the area controlled by the area specifying unit 103 indicates a state where permission to add to areas other than the area being controlled is given. That is, a new area cannot be created unless the state is shifted to this state.

  Further, in the figure, the solid-line arrows are “writing” operations for starting a series of board writing. The dotted arrow is the “pointing” action.

  With reference to FIG. 5, how the region controlled by region specifying unit 103 transitions by the operations (I) to (X) will be described with reference to the flowchart shown in FIG. 4.

  In the present embodiment, it is assumed that the 2D display device 11 is in a blank state at the start of the lesson. Therefore, there is no area controlled by the area specifying unit 103 at the start of the lesson.

  (I) Even if the “pointing” operation is performed in a state where all are in the unarea range, there is no area that can be controlled by the area specifying unit 103 (NO in step S30). Since there is no area under control, permission to add to areas other than the area under control is issued.

  (II) If the write permission is given to the other area when the “write” operation is performed in the non-area range, this area is created as a new area A (step S25). At the same time, writing to areas other than the area under control is disabled (step S27). Then, the board writing content is recorded in the area A (step S28). Thereafter, when an operation of “leaving” is detected, permission to add to an area other than the area under control is given (step S21).

  (III) Since permission to add to areas other than the area under control has been issued, when the “write” operation is started in the non-area area, this area is created as a new area B (step S25). At the same time, writing to areas other than the area under control is disabled (step S27). Then, the board writing content is recorded in the area B (step S28).

  (IV) When the operation of “pointing” to another existing area A is performed while the area B is controlled, the area specifying unit 103 shifts the control to the area A (step S31). At the same time, writing to areas other than the area under control is disabled (step S32). Thereafter, when the “leaving” operation is detected, permission to add to a region other than the controlled region is given (step S21).

  (V) Since permission to add to areas other than the area under control has been issued, when the “write” operation is started in the non-area area, this area is created as a new area C (step S25). At the same time, writing to areas other than the area under control is disabled (step S27). Then, the board writing content is recorded in the area B (step S28).

  (VI) Even if the operation of “pointing” to the non-area range is performed in the state where the area C is controlled, the area controlled by the area specifying unit 103 does not change (NO in step S30). Thereafter, when the “leaving” operation is detected, permission to add to a region other than the controlled region is given (step S21).

  (VII) Here, when it is desired to add to the area B under control, a new area is created when the “write” operation is started in the non-area area. Therefore, first, an operation of “pointing” to the existing area B is performed, and control is transferred to the area B (step S31). At the same time, writing to areas other than the area under control is disabled (step S32). Thereafter, by starting the “write” operation in the area B (NO in step S29), it is possible to add to the existing area B (step S28).

  (VIII) When the “write” operation is started within the range of the existing area A that is not under control in a state where the permission to add to areas other than the area under control has not been issued (YES in step S29), the area specifying unit 103 Transfers control to region A (step S26). Then, writing to areas other than the controlled area is disabled (step S27), and the contents of the board are added to the existing area A (step S28). Thereafter, when the “leaving” operation is detected, permission to add to a region other than the controlled region is given (step S21).

  (IX) Here, when it is desired to add to the area A being controlled, a new area is created when the “write” operation is started in the non-area range. Therefore, first, an operation of “pointing” to the area A being controlled is performed, and the area A is designated again (step S31). Then, writing to areas other than the area under control is disabled (step S32). Thereafter, by starting the “write” operation in the area A (NO in step S29), it is possible to add to the area A being controlled (step S28).

  (X) In a state where permission to add to areas other than the controlled area is given (YES in step S23), a “write” operation is started within the range of the controlled area A (NO in step S24). The area A can be added (steps S26 to S28).

  As described above, the area specifying unit 103 can determine whether to create a new area or add to an existing area depending on the situation by holding permission to add to another area.

  If an existing area other than the area under control is entered during the “write” operation, it may not be possible to write in the intrusion part, and the intrusion part is overwritten and processed as an overlapping area. May be.

  Referring to FIGS. 1 and 2 again, the stream data input unit 104 acquires, via the stream data input device 12, information related to audio, such as teacher explanations and student questions about the board, as stream data. This stream data is not limited to audio-only data, and may be video data including audio.

  Next, the utterance position specifying unit 105 specifies the boundary between the silent part and the sound part in the acquired stream data. However, in class, the utterance is considered to be interrupted. Therefore, a state where a sound with a certain volume or lower continues for a certain time or more is determined as a silent part. In this way, by specifying the boundary between the silent part and the voiced part, the stream data is segmented using only the voiced part as the uttered part and recorded as a stream data set.

  The link correspondence specifying unit 106 includes a display area set recorded by the area specifying unit 103 and a stream data set recorded by the utterance position specifying unit 105 when the area specifying unit 103 and the utterance position specifying unit 105 perform parallel processing. A method of associating with will be described.

  The link correspondence specifying unit 106 associates the display area set recorded by the area specifying unit 103 and the stream data set recorded by the utterance position specifying unit 105 with a temporal relationship. As described above, an operation for creating and controlling a region is set as a trigger for a region divided for each series of board writing operations. Then, each region and each utterance part are associated with each other by searching for the time when this trigger occurs and the segment position (hereinafter referred to as “utterance start position”) that is the utterance start time of each utterance part. The area specifying unit 103 summarizes the associations and creates a link correspondence table.

  The link correspondence specifying process executed by the area specifying unit 103 is performed as follows.

  First, if it is uttered at the time of trigger occurrence, it returns at the start of utterance of this utterance part, and associates the area with the utterance part. If it is not uttered when the trigger is generated, the utterance start time of the utterance part to be uttered thereafter is found, and the area and the utterance part are associated with each other.

  FIG. 6 is a flowchart showing the flow of processing executed by the link correspondence specifying unit 106 of the data recording apparatus in the embodiment of the present invention.

  First, it is determined whether or not an operation for creating and controlling a region (hereinafter referred to as “region trigger”) has been detected (step S41). Wait until a region trigger is detected (NO in step S41). When the area trigger is detected (YES in step S41), it is determined whether or not the utterance part is detected when the area trigger occurs (step S42).

  When the utterance part is detected when the region trigger occurs (YES in step S42), the utterance start position of the utterance part is detected by going back in time (step S43). Then, it is associated with the area under control (step S44).

  After the process of step S44, it is determined whether or not there is an utterance that is not associated with any area (hereinafter referred to as “unlinked utterance”) (step S45). If there is no unlinked utterance (NO in step S45), the process proceeds to S49. On the other hand, if it is determined that there is an unlinked utterance, it is determined whether there is a control area immediately before (step S46). If there is a control area immediately before (YES in step S46), the detected unlinked utterance is associated with the immediately previous control area (step S47). On the other hand, if there is no control area immediately before (NO in step S46), unlinked utterance is processed as having no associated area (step S48).

  After that, if the utterance portion is detected when the region trigger occurs (YES in step S49), it is determined whether or not the utterance portion associated with the region being controlled has been linked to another region (step S50). . If it is determined that the utterance part has been linked to another area (YES in step S50), these areas are considered to be related, and are processed as one area (step S51). Then, it returns to step S41 and waits until the next area trigger is detected. On the other hand, if it is determined that the utterance part has not been linked to another area (NO in step S50), the process returns to step S41 and waits until the next area trigger is detected.

  If it is determined in step S42 that there is no utterance when the region trigger occurs (NO in step S42), it is determined whether there is an unlinked utterance and the same processing as above is performed (steps S45, S46, S47, S48). After that, since the utterance part is not detected when the region trigger occurs (NO in step S49), the time is advanced until the next region trigger is detected (NO in step S52), and the next utterance start position is detected. (Step S53). If the utterance start position is detected before the next area trigger is detected (YES in step S53), the utterance part to which the detected utterance start position belongs is associated with the control area (step S54), and step 41 is performed. Return to.

  On the other hand, when the next utterance start position cannot be detected (NO in step S53) and the area trigger is detected (YES in step S52), the control area is processed as having no utterance part to be associated. (Step S55), the process returns to Step S42.

  FIG. 7 is a diagram for explaining a specific example of the process executed by the link correspondence specifying unit 106 of the data recording apparatus according to the embodiment of the present invention.

  With reference to the flowchart shown in FIG. 6, the link correspondence specifying method according to the situation will be specifically described.

  FIG. 7A shows a case where the voice is uttered when the area trigger occurs (YES in step S42).

  The link correspondence specifying unit 106 detects an operation of creating the region A as a trigger (step S41). Returning to the utterance start position of the utterance S2 uttered when the area trigger occurs (step S43), the utterance S2 is associated with the area A using the utterance start position as the link start position (step S44). Here, there is an unlinked utterance S1 before the generation of the region trigger for creating the region A (YES in step S45). Since there is no controlled area before area A (NO in step S46), utterance S1 is recorded in the link correspondence table on the assumption that there is no associated area (step S48).

  Next, using the operation of creating the region B as a trigger (step S41), the utterance S4 is similarly associated with the region B (step S44). The link correspondence specifying unit 106 associates the utterance S3 existing between the utterance S2 associated with the area A and the utterance S4 associated with the area B with the area A (step S47).

  FIG. 7B shows a case where the voice is not uttered when the region trigger occurs (NO in step S42).

  The link correspondence specifying unit 106 detects an operation of creating the region A as a trigger (step S41). Here, unlinked utterance S1 and utterance S2 exist before the area trigger for creating area A is generated (YES in step S45). Since there is no controlled area before area A (NO in step S46), utterance S1 and utterance S2 are recorded in the link correspondence table on the assumption that there is no associated area (step S48). Since the voice is not uttered when the area trigger occurs, the utterance start position uttered after the area trigger occurs is found (YES in step S53), and the utterance S3 is associated with the area A using the utterance start position as the link start position (step S54). ).

  Next, using the operation for creating the region B as a trigger (step S41), the utterance S4 existing between the utterance S3 and the region trigger for creating the region B is associated with the region A (step S47). Then, the utterance S5 is associated with the region B (step S54).

  FIG. 7C shows a case where the state of the above (a) extends over a plurality of regions (YES in step S50).

  Similar to (a) above, the region A and the utterance S2 are associated (step S44). Similarly to (a) above, the utterance S1 is recorded in the link correspondence table on the assumption that there is no associated area (step S48). Next, when a region trigger for creating the region B is detected (step S41), since the utterance S2 is uttered even when the region trigger occurs, the utterance S2 is also associated with the region B (step S44). However, in this case, the regions A and B are associated with the same utterance S2. In such a case, the regions A and B are considered to be closely related to each other. Thus, when the link start positions overlap in a plurality of areas as described above, the plurality of areas are recognized as one area. In this case, information regarding the area of the display area set 109 is rewritten (step S51).

  FIG. 7D shows a case where the states (b) and (a) extend over a plurality of regions (YES in step S50).

  Similar to (b) above, the region A and the utterance S2 are associated (step S54). Similarly to (a) above, the utterance S1 is recorded in the link correspondence table on the assumption that there is no associated area (step S48). Next, when a region trigger for creating the region B is detected (step S41), since the utterance S2 is uttered even when this region trigger occurs, the utterance S2 is also associated with the region B as in (a) above ( Step S44). However, in this case, both the areas A and B are associated with the same utterance S2. In such a case, the regions A and B are considered to be closely related to each other. Thus, when the link start positions overlap in a plurality of areas as described above, the plurality of areas are recognized as one area. In this case, information regarding the area of the display area set 109 is rewritten (step S51).

  FIG. 7E shows a case where the utterance explanation part cannot be found (NO in step S53).

  The link correspondence specifying unit 106 detects an operation of creating the region A as a trigger (step S41). Since the voice is not uttered when the area trigger occurs, the utterance S1 is recorded in the link correspondence table on the assumption that there is no associated area (step S48), and uttered after the area trigger occurs, as in (b) above. A start position is searched (step S53). However, if a region trigger for creating region B occurs without finding the utterance start position (YES in step S52), control is transferred to region B. As described above, when the utterance part is not found between the area triggers, it is processed that there is no utterance part associated with the area A (step S55).

  When the utterance start position of the utterance S2 is detected after the area trigger of the area B is generated (YES in step S53), the utterance S2 is associated with the area B (step S54).

  FIG. 7F is a diagram similar to FIG. 7A, but the case where there is no region to be controlled (NO in step S46) will be specifically described.

  At the start of the lesson, the board written part is in a blank state, and there is no area controlled by the area specifying unit 103. In this state, the utterance starts and the ended utterance S1 has no area to be linked. However, since the utterance S1 is considered to be an utterance explanation part necessary as a prelude part of the lesson, it is necessary to record it. Therefore, the utterance S1 is recorded in the link correspondence table as having no associated area (step S48).

  In addition, as long as there is no trouble in associating with the front and back areas, it may be associated with either area.

  Referring to FIGS. 1 and 2, data recording unit 107 records a display area set, a stream data set, and a link correspondence table created through the above processing. This recorded data is the basis for the reproduction of the lesson by a later data reproducing apparatus.

  In the embodiment of the present invention, the data recording device detects the states of “writing”, “pointing”, and “leaving” by specifying the coordinate position instructed by the position specifying device. Thereby, a new area can be created or added to an existing area according to the actual situation.

  Further, since the division position of the stream data to be associated automatically is detected when the control of the area shifts, it can be edited in real time. Therefore, the trouble of editing later can be saved.

  Next, the data reproducing apparatus according to the embodiment of the present invention will be described with reference to FIGS.

  FIG. 8 is a functional block diagram of the data reproducing apparatus 200 according to the embodiment of the present invention.

  Referring to FIG. 8, a two-dimensional display device 11 for reproducing and displaying the class written part of the lesson, a position specifying unit 13 capable of electronically indicating a divided area of the class part, and the utterance explanation of the class A stream data output device 14 for reproducing the portion is provided.

  Note that the data reproducing apparatus 200 does not have to be used in the same equipment environment as the lesson. A PC, a mobile terminal, etc. may be sufficient if the apparatus which can reproduce | regenerate the data which recorded the lesson is provided.

  In order to reproduce the lesson contents using these devices, the data reproduction device 200 includes a data recording unit 201, a two-dimensional display reproduction unit 202, a position designation unit 203, a link correspondence analysis unit 204, and a stream data reproduction unit 205. .

  Review lessons in an environment with the above configuration. The practitioner acquires data in which the content of the lesson is recorded by the data recording device 100 and stores it in the data recording unit 201. Then, the display area set, the stream data set, and the link correspondence table are acquired from the data recording unit 201 and displayed and reproduced.

  First, the two-dimensional display reproduction unit 202 transmits a signal indicating the display area set stored in the data recording unit 201 to the two-dimensional display device 11 so as to reproduce and display the board part of the lesson. The display area set is a set of continuous coordinate information recorded together for each area as an object that is a single line.

  At the same time, based on the link correspondence table, the presence of stream data, which is the utterance portion associated with each area, is displayed. This image is shown in FIG. Here, in order to display the range in which the board part is divided into regions, the region is clarified by surrounding the region with a line or coloring the region range (702). Also, in order to indicate the presence of stream data, icons for identifying each utterance part are displayed by the number of associated utterance parts (703), or a list of utterance parts is displayed. (704). In addition, regarding the utterance part that is not associated with the region, the existence is clarified by displaying a special icon (701).

  Next, in order to output stream data linked to each area, the position designating unit 203 points to each area (702) displayed by the two-dimensional display reproduction unit 202 by the position designating device 13. Specify an area. Alternatively, an icon or the like (701, 703, 704) indicating the presence of the utterance portion is selected, and each utterance portion is designated. The position specifying unit 203 acquires the specified coordinate information, and specifies an area from this coordinate information. Similarly, an icon is specified.

  Next, the link correspondence analysis unit 204 finds out and specifies the utterance part linked to the area (702) specified by the position specifying unit 203 from the stream data set based on the link correspondence table. At this time, when there are a plurality of utterance parts, all the utterance parts to be linked are specified. If the position designation unit 203 detects that an icon (703) representing a specific utterance portion is designated, the utterance portion is specified.

  Next, the stream data reproduction unit 205 acquires the stream data corresponding to the utterance part specified by the link correspondence analysis unit 204 from the stream data set stored in the data recording unit 201. Then, the acquired stream data is reproduced by being transmitted to the stream data output device 14. If there are a plurality of utterance parts, they are reproduced in the order recorded in the stream data set stored in the data recording unit 201.

  As described above, by designating the area (701) displayed on the two-dimensional display device 11, the related utterance explanation part can be easily heard. Further, when there is an utterance portion to be repeatedly heard, by selecting an icon (701, 703) displayed on the two-dimensional display device 11, a necessary portion can be extracted and heard.

  In the present embodiment, the area is a rectangle surrounded by the coordinates of the minimum position and the coordinates of the maximum position on the xy axis among the plurality of objects representing the coordinate position designated by the “write” operation. If it can be associated with, it is not limited to this form.

  In addition, the link correspondence specifying unit may specify the utterance position of the stream data to be associated with the time when the region specified by the region specifying unit is switched.

  In addition, although the segmentation position of the associated stream data is the utterance start position, the segmentation position is not limited to the utterance start position as long as the stream data can be segmented.

  Moreover, although the link correspondence specific | specification part decided to link by searching the division position on stream data, you may link by searching the utterance position on stream data.

  The embodiment disclosed this time should be considered as illustrative in all points and not restrictive. The scope of the present invention is defined by the terms of the claims, rather than the description above, and is intended to include any modifications within the scope and meaning equivalent to the terms of the claims.

1 is a functional block diagram of a data recording apparatus in an embodiment of the present invention. 1 is a conceptual diagram of a data recording apparatus in an embodiment of the present invention. It is a flowchart which shows the flow of the board writing operation | movement state detection process which the instruction | indication presence / absence detection part of the data recording device in embodiment of this invention performs. It is a flowchart which shows the flow of the area | region identification process which the area | region identification part of the data recording device in embodiment of this invention performs. It is a figure for demonstrating an example of the process which the area | region specific part of the data recording device in embodiment of this invention performs. It is a flowchart which shows the flow of the process which the link corresponding | compatible identification part of the data recording device in embodiment of this invention performs. It is a figure for demonstrating the specific example of the process which the link corresponding | specification specific part of the data recording device in embodiment of this invention performs. It is a functional block diagram of the data reproducing device in the embodiment of the present invention. It is a figure which shows an example of the content which the two-dimensional display reproduction part of the data reproduction apparatus in embodiment of this invention displays on a two-dimensional display apparatus.

Explanation of symbols

  11 Two-dimensional display device, 12 Stream data input device, 13 Position designation device, 14 Stream data output device, 100 Data recording device, 101 Instruction presence / absence detection unit, 102 Location specifying unit, 103 Region specifying unit, 104 Stream data input unit, 105 utterance position specifying unit, 106 link correspondence specifying unit, 107 data recording unit, 109 display area set, 200 data reproducing device, 201 data recording unit, 202 two-dimensional display reproducing unit, 203 position specifying unit, 204 link correspondence analyzing unit, 205 Stream data reproduction unit.

Claims (12)

An instruction presence / absence detecting means for detecting the presence / absence of an instruction for two-dimensional display;
Position specifying means for specifying a coordinate position on the two-dimensional display of the instruction detected by the instruction presence / absence detecting means;
An area specifying means for specifying an area including the specified coordinate position based on the coordinate position specified by the position specifying means;
Stream data input means for inputting stream data including audio;
Utterance position specifying means for detecting the presence or absence of sound on the stream data input by the stream data input means, and specifying a sound part as a utterance position;
The utterance position specifying means can specify a segment position on the stream data,
Link correspondence specifying means for associating the area specified by the area specifying means with the segment position specified by the utterance position specifying means;
The plurality of specified areas, the coordinate positions associated with each of the identified areas, the stream data, and the link correspondence information between the areas and the segment positions associated by the link correspondence identifying means. A data recording apparatus further comprising recording means for recording. The instruction presence / absence detecting means detects the instruction, and when the distance the instruction has moved on the two-dimensional display is a predetermined value or more, the first state, and when the distance is less than the predetermined value, the second state. A state where the instruction is not detected is determined as a third state,
The data recording apparatus according to claim 1, wherein the recording unit controls a recording process of the specified area based on the first to third states. When the instruction presence / absence detecting unit determines that the instruction is in the first state, and the coordinate position of the instruction specified by the position specifying unit is not included in the existing area,
The data recording apparatus according to claim 1, wherein the area specifying unit creates a new area. When the instruction presence / absence detecting means determines that the instruction is in the first or second state, and the coordinate position of the instruction specified by the position specifying means is included in an existing area,
The data recording apparatus according to claim 1, wherein the area specifying unit selects and controls the existing area.   The link correspondence specifying unit specifies an instruction start time at which an instruction of the first or second state is first detected for the region specified by the region specifying unit, and the utterance at the instruction start time The data recording apparatus according to claim 1 or 2, wherein a segment position to be associated with the specified area is searched based on whether or not the utterance position specified by the position specifying means is detected.   The link correspondence specifying unit specifies an instruction start time at which an instruction of the first or second state is first detected for the region specified by the region specifying unit, and the utterance at the instruction start time The data recording apparatus according to claim 1 or 2, wherein when the utterance position specified by the position specifying means is detected, the section position for associating with the specified area is searched backwards in time.   The link correspondence specifying unit specifies an instruction start time at which an instruction of the first or second state is first detected for the region specified by the region specifying unit, and the utterance at the instruction start time 3. The data recording apparatus according to claim 1, wherein when the utterance position specified by the position specifying means is not detected, the time is advanced to search for a segment position to be associated with the specified area.   The link correspondence specifying means includes a partition position associated with the first area specified by the area specifying means, and a partition position associated with the second area specified temporally by the area specifying means. The data recording apparatus according to claim 1, wherein a utterance position existing between the first and second areas is associated with the first area.   2. The link correspondence specifying unit according to claim 1, wherein when the division position associated with the region specified by the region specifying unit is already associated with an existing region, the two regions are combined into one region. Data recording device.   The link correspondence specifying means, when the area specifying means specifies another area without detecting the area to be associated with the area specified by the area specifying means, the former area includes an associated area. The data recording apparatus according to claim 1, wherein no data exists.   The data recording apparatus according to claim 1, wherein the link correspondence specifying unit does not have an associated area for a remaining utterance position that cannot be associated with any area. A plurality of areas on a two-dimensional display, coordinate positions associated with each of the areas, stream data including audio, and link correspondence information in which the areas and segment positions on the stream data are associated are stored. Recording means for
Two-dimensional display reproduction means for transmitting a signal for displaying the plurality of areas stored in the recording means and coordinate positions associated with each area;
Position designation means for receiving a signal designating a coordinate position on a two-dimensional display and identifying an area including the coordinate position;
Link correspondence analyzing means for analyzing the link correspondence information stored in the recording means,
The link correspondence analyzing unit specifies a segment position on the stream data associated with the area specified by the position specifying unit,
Further comprising stream data reproduction means for transmitting a signal for outputting the stream data stored in the recording means,
The data reproduction apparatus, wherein the stream data reproduction unit transmits a signal for outputting the corresponding stream data based on the division position specified by the link correspondence analysis unit.

Images