KR102347068B1 - Method and device for replaying content - Google Patents

Abstract

a display unit for displaying handwritten content created based on a user's analog handwriting input; a user input unit for receiving a user input for selecting a partial region of the handwritten content displayed on the display unit; and a control unit for reproducing a multimedia content section corresponding to a selected partial area of multimedia content that is synchronized with the handwritten content and can be reproduced together.

Description

Method and device for playing content {METHOD AND DEVICE FOR REPLAYING CONTENT}

The present invention relates to a method for reproducing some handwritten content selected by a user and a device therefor.

The mobile terminal may be configured to perform various functions. Examples of such various functions include a data and voice communication function, a function to take a picture or video through a camera, a voice storage function, a function to play a music file through a speaker system, a function to display an image or video, and the like.

Some mobile terminals include an additional function capable of running a game, and some other terminals are implemented as multimedia devices. Moreover, as the development of smart phones accelerates, the development and use of various applications are increasing.

An object of the present invention is to provide a method of reproducing partial handwritten content using time information of strokes included in the partial handwritten content. Another object of the present invention is to provide a method of reproducing a partial section of multimedia content synchronized with some handwritten content.

A device according to an embodiment of the present invention includes: a display unit for displaying handwritten content on a screen; a user input unit receiving a user input for selecting some handwritten content from among the displayed handwritten content; and a control unit configured to reproduce the partial handwritten content by using time information of strokes included in the selected partial handwritten content.

The controller of the device according to an embodiment of the present invention may generate handwritten content by acquiring time stamp information and coordinate information corresponding to each of a plurality of strokes.

The controller of the device according to an embodiment of the present invention may generate the handwritten content during reproduction of the multimedia content, synchronize the handwritten content with the multimedia content, and store it in a memory.

The user input unit of the device according to an embodiment of the present invention receives an input specifying a region of interest on handwritten content displayed on a screen, and the controller of the device according to an embodiment of the present invention corresponds to the region of interest You can select some handwritten content that you want to use.

The controller of the device according to an embodiment of the present invention may select some handwritten content corresponding to the ROI based on a stroke unit or a coordinate unit.

When some handwritten content is selected, the display unit of the device according to an embodiment of the present invention displays a control panel including a play function button within a predetermined distance from the partial handwritten content, and the device according to an embodiment of the present invention The user input unit may receive an input for selecting a play function button through the control panel.

The controller of the device according to an embodiment of the present invention may extract strokes included in some handwritten content and determine an order between strokes by using time stamp information corresponding to the strokes.

The controller of the device according to an embodiment of the present invention determines a stroke input first among strokes as a start position, and determines a stroke input last among strokes as an end position. can

The controller of the device according to an embodiment of the present invention may extract video content or audio content corresponding to some handwritten content, and provide the extracted video content or audio content.

When a plurality of partial handwritten contents are selected, the controller of the device according to an embodiment of the present invention may provide a list of a plurality of video contents corresponding to the plurality of partial handwritten contents.

The controller of the device according to an embodiment of the present invention may edit the partial handwritten content by using time information of strokes included in the partial handwritten content.

According to an embodiment of the present invention, a method for reproducing some handwritten content includes: displaying the handwritten content on a screen; receiving a user input for selecting some handwritten content from among the displayed handwritten content; and reproducing the partial handwritten content by using time information of strokes included in the selected partial handwritten content.

The displaying of the handwritten content on the screen according to an embodiment of the present invention may include generating the handwritten content by acquiring time stamp information and coordinate information corresponding to each of a plurality of strokes.

Receiving a user input according to an embodiment of the present invention may include: receiving an input specifying a region of interest on handwritten content displayed on a screen; and selecting some handwritten content corresponding to the region of interest.

Receiving a user input according to an embodiment of the present invention may include: when some handwritten content is selected, displaying a control panel including a play function button within a predetermined distance from the handwritten content; and receiving an input for selecting a play function button through the control panel.

The reproducing of the partial handwritten content according to an embodiment of the present invention may include: extracting strokes included in the partial handwritten content; and determining an order between the strokes by using time stamp information corresponding to the strokes.

The step of determining the sequence of strokes according to an embodiment of the present invention includes determining a first inputted stroke among strokes as a start position and ending a last inputted stroke among strokes. It may include the step of determining the position). .

A method according to an embodiment of the present invention includes extracting video content or audio content corresponding to some handwritten content; and providing the extracted video content or audio content.

The step of extracting the video content or the audio content according to an embodiment of the present invention may include providing a list of a plurality of video contents corresponding to the plurality of partial handwritten contents when a plurality of partial handwritten contents are selected. have.

A device according to an embodiment of the present invention includes a display unit for displaying handwriting content on a screen; and a user input unit for receiving a user input for selecting a part of handwriting displayed on the display unit; and a controller for controlling the display unit to reproduce the selected part of the handwritten content in the same temporal manner as the handwritten content.

A method of reproducing handwritten content according to an embodiment of the present invention includes: displaying the handwritten content on a screen of a device; Receiving a user input for selecting a part of the handwritten content displayed on the screen; and playing the selected portion of the handwritten content in the same temporal manner as it was handwritten.

A device according to an embodiment of the present invention includes: a display unit for displaying handwritten content created based on a user's analog handwriting input; a user input unit for receiving a user input for selecting a partial region of the handwritten content displayed on the display unit; and a controller for playing a multimedia content section corresponding to a selected partial area among multimedia content that is synchronized with the handwritten content and can be reproduced together.

The control unit of the device according to an embodiment of the present invention may obtain time information of the multimedia content based on coordinate information of the selected handwritten content so that the multimedia content section is reproduced.

The controller of the device according to an embodiment of the present invention may reproduce the handwritten content of the selected partial area while the multimedia content section is being reproduced.

The controller of the device according to an embodiment of the present invention may reproduce the handwritten content of the selected partial area in the same temporal manner as that in which it was written.

The controller of the device according to an embodiment of the present invention may extract strokes included in the selected partial region and determine an order between the strokes by using time stamp information corresponding to the strokes.

The controller of the device according to an embodiment of the present invention determines a first stroke input first among strokes as a start position, and sets a last stroke input last among strokes to an end position. can be decided with

When a plurality of areas of the handwritten content displayed on the display unit are selected, the controller of the device according to an embodiment of the present invention may reproduce a plurality of multimedia content sections corresponding to the plurality of selected areas.

The controller of the device according to an embodiment of the present invention controls the display unit to display a control panel including a play function button within a predetermined distance from the partial region when a partial region of the handwritten content is selected, and the user input unit includes: The panel can receive input to select a play function button.

The controller of the device according to an embodiment of the present invention may edit the handwritten content by using time information of strokes included in the selected partial region.

According to an embodiment of the present invention, there is provided a method for reproducing content in a device, the method comprising: displaying handwritten content created based on a user's analog handwriting input; receiving a user input for selecting a partial area of the handwritten content; and reproducing a multimedia content section corresponding to a selected partial region among multimedia content that is synchronized with the handwritten content and can be reproduced together.

The step of reproducing the multimedia content section according to an embodiment of the present invention may include acquiring time information of the multimedia content based on coordinate information of the selected handwritten content and playing the multimedia content section.

The content reproducing method according to an embodiment of the present invention may further include reproducing the handwritten content of the selected partial area while the multimedia content section is being reproduced.

The step of reproducing the handwritten content of the selected partial area according to an embodiment of the present invention may include reproducing the handwritten content of the selected partial area in the same temporal manner as that in which it was written.

According to an embodiment of the present invention, the reproducing of the handwritten content in the selected partial region may include extracting strokes included in the selected partial region; and determining an order between the strokes by using time stamp information corresponding to the strokes.

The step of determining the order between strokes according to an embodiment of the present invention includes determining a first stroke inputted first among strokes as a start position, and determining a last stroke inputted last among strokes. determining the stroke to an end position.

The step of reproducing the multimedia content section according to an embodiment of the present invention may include, when a plurality of areas of the handwritten content displayed on the display unit are selected, reproducing a plurality of multimedia content sections corresponding to the plurality of selected areas. can

Receiving a user input according to an embodiment of the present invention may include: when a partial region of the handwritten content is selected, displaying a control panel including a play function button within a predetermined distance from the partial region; and receiving an input for selecting a play function button through the control panel.

The content reproduction method according to an embodiment of the present invention may further include editing the handwritten content by using time information of strokes included in the selected partial region.

1 is a diagram for describing a device that provides a function of reproducing some handwritten content according to an embodiment of the present invention.
2 is a flowchart illustrating a method for a device to reproduce some handwritten content according to an embodiment of the present invention.
3 is a flowchart illustrating a method for a device to generate handwritten content according to an embodiment of the present invention.
4A to 4C are diagrams for explaining an example of generating handwritten content in a device.
5 is a diagram for explaining an example of a storage format of handwritten content.
6 is a flowchart illustrating a method for a device to synchronize and store handwritten content and audio content, according to an embodiment of the present invention.
7A to 7E are diagrams for explaining an example in which a device synchronizes and stores handwritten content and audio content.
8 is a diagram for explaining an example of a storage format of handwritten content and audio content.
9 is a flowchart illustrating a method for a device to synchronize and store handwritten content and video content, according to an embodiment of the present invention.
10A and 10B are diagrams for explaining an example in which a device synchronizes and stores handwritten content and video content.
11A is a diagram for explaining an example of a storage format of handwritten content and video content.
11B is a diagram for explaining connection information connecting a stroke of handwritten content and a frame of moving image content.
12, 13A, and 13B are diagrams for explaining a method of storing handwritten content, audio content, and video content in a device.
14 is a diagram for explaining a method for a device to store handwritten content in synchronization with emotion information, according to an embodiment of the present invention.
15 is a diagram for describing an example in which a device acquires emotion information while generating handwritten content.
16 is a flowchart illustrating a method for a device to select some handwritten content corresponding to a region of interest according to an embodiment of the present invention.
17 is a diagram for explaining an example of an input for designating a region of interest.
18, 19, 20A, and 20B are diagrams for explaining an example of a user interface for designating a region of interest.
21 and 22 are diagrams illustrating an example of a graphical user interface (GUI) for determining some handwritten content corresponding to a region of interest.
23A is a diagram illustrating an example in which a device displays a GUI for determining a stroke corresponding to a region of interest in a pop-up window.
23B is a diagram illustrating an example in which a device displays a guide image for guiding designation of a region of interest.
24 is a diagram illustrating an example of a user interface for selecting some handwritten content in units of segments.
25 is a diagram illustrating an example in which a device identifies and displays selected strokes to distinguish them from other strokes.
26 is a flowchart illustrating a method for a device to receive a play request through a control panel according to an embodiment of the present invention.
27A and 27B are diagrams for explaining an example in which the device provides a control panel including a playback function button or an edit function button.
28 is a flowchart illustrating a method for a device to display some handwritten content selected by a user on a separate layer, according to an embodiment of the present invention.
29 is a diagram illustrating an example in which a device displays some handwritten content selected by a user on a separate layer.
30 is a flowchart illustrating a method for a device to reproduce some handwritten content by using time information of strokes in an ROI, according to an embodiment of the present invention.
31 is a diagram illustrating an example in which a device selects strokes within a region of interest.
32A is a diagram illustrating an example in which a device extracts information on strokes in an ROI.
32B is a diagram illustrating an example in which a device generates a stack by using information on strokes in an ROI.
33A to 33D are diagrams illustrating an example in which a device reproduces some handwritten content using information on strokes included in a stack.
34 is a diagram illustrating an example in which a device reproduces some handwritten content in a separate window using information on strokes included in a stack.
35 is a flowchart illustrating a method for a device to edit some handwritten content according to an embodiment of the present invention.
36 is a diagram for describing a method of a device performing an undo or redo function for some handwritten content, according to an embodiment of the present invention.
37 is a diagram illustrating a list of a plurality of strokes included in handwritten content according to an embodiment of the present invention.
38 is a diagram illustrating an example of a first stack including information on strokes in a region of interest and a second stack including information on deleted strokes, according to an embodiment of the present invention.
39A and 39B are diagrams illustrating a first stack and a second stack that change according to a user's undo input.
40A, 40B, and 40C are diagrams for explaining an example in which a device performs an undo function and a redo function for some handwritten content.
41A and 41B are diagrams for explaining an example in which a device performs an undo function and a stroke addition function for some handwritten content.
42 is a diagram for explaining a method for a device to replace some handwritten content with external handwritten content, according to an embodiment of the present invention.
43A, 43B, 43C, and 43D are diagrams for explaining an example in which a device replaces some handwritten content with external handwritten content.
44 is a flowchart illustrating a method for a device to reproduce the entire handwritten content by reflecting editing information for some handwritten content, according to an embodiment of the present invention.
45A, 45B, and 45C are diagrams for explaining an example in which a device reproduces the entire handwritten content by reflecting editing information for some handwritten content.
46 is a flowchart illustrating a method for a device to perform character recognition for some handwritten content, according to an embodiment of the present invention.
47A and 47B are diagrams for explaining an example in which a device performs character recognition for some handwritten content.
48 is a flowchart illustrating a method for a device to add a background image, an icon, or background music by using emotional information corresponding to some handwritten content, according to an embodiment of the present invention.
49 is a diagram illustrating an example in which a device displays pulse information corresponding to some handwritten content.
50 is a flowchart illustrating a method for a device to reproduce a plurality of partial handwritten content according to an embodiment of the present invention.
51A is a diagram for explaining an example in which a device reproduces a plurality of partial handwritten content.
51B and 51C are diagrams for explaining an example in which a device exchanges positions of a plurality of partial handwritten contents.
52 is a flowchart illustrating a method for a device to reproduce music by using information on strokes included in some handwritten content, according to an embodiment of the present invention.
53A, 53B, and 53C are diagrams for explaining an example in which a device reproduces music by using information on strokes included in some handwritten content.
54 is a flowchart for explaining a method of extracting, by a device, video content or audio content corresponding to some handwritten content, according to an embodiment of the present invention.
55A, 55B, and 55C are diagrams for explaining an example in which a device extracts video content corresponding to some handwritten content.
56 is a flowchart illustrating a method for a device to provide a list of a plurality of video content corresponding to a plurality of partial handwritten content, according to an embodiment of the present invention.
57 is a diagram illustrating a list of a plurality of video contents corresponding to a plurality of partial handwritten contents according to an embodiment of the present invention.
58 is a flowchart illustrating a method for a device to provide a plurality of handwritten content received from a plurality of external devices, according to an embodiment of the present invention.
59 is a diagram illustrating an example in which a device provides a plurality of pieces of handwritten content received from a plurality of external devices.
60A and 60B are diagrams for explaining an example in which a device transmits a plurality of handwritten contents to each of a plurality of external devices.
61 is a diagram for explaining a method for a first device to receive handwritten content edited by a second device, according to an embodiment of the present invention.
62A is a diagram illustrating an example in which a second device reproduces handwritten content received from a first device.
62B is a diagram illustrating an example in which the second device edits some handwritten content.
62C is a diagram illustrating an example in which the first device reproduces the entire handwritten content by reflecting editing information for some handwritten content.
63 is a flowchart illustrating a method of reproducing a part of handwritten content selected in a first device in a second device, according to an embodiment of the present invention.
64 is a diagram illustrating an example of reproducing a part of content selected by a device on an external display device.
65 is a diagram illustrating an example in which a part of handwritten content selected in the first device is reproduced in the same temporal manner as that in which it was written.
66 is a diagram illustrating an example of reproducing a part of handwritten content selected by the first device in a plurality of external devices.
67 is a flowchart for explaining a method of respectively reproducing a part of handwritten content selected by the first device by the first device and the second device;
68 is a diagram illustrating an example in which the first device and the second device respectively reproduce a part of the handwritten content selected by the first device in the same temporal manner as when the handwritten content was written.
69 is a flowchart for explaining a method of providing, by a device, multimedia content of a partial section corresponding to a selected transcript, according to an embodiment of the present invention.
70 is a diagram illustrating an example in which a device provides a partial video section corresponding to a selected transcript according to an embodiment of the present invention.
71 is a diagram illustrating an example in which a device searches for a partial video section based on a search keyword.
72A and 72B are diagrams illustrating an example in which a device provides a partial audio section corresponding to a selected word according to an embodiment of the present invention.
73 and 74 are block diagrams for explaining the configuration of a device according to an embodiment of the present invention.

Terms used in this specification will be briefly described, and the present invention will be described in detail.

The terms used in the present invention have been selected as currently widely used general terms as possible while considering the functions in the present invention, but these may vary depending on the intention or precedent of a person skilled in the art, the emergence of new technology, and the like. In addition, in a specific case, there is a term arbitrarily selected by the applicant, and in this case, the meaning will be described in detail in the description of the corresponding invention. Therefore, the term used in the present invention should be defined based on the meaning of the term and the overall content of the present invention, rather than the name of a simple term.

In the entire specification, when a part "includes" a certain element, this means that other elements may be further included, rather than excluding other elements, unless otherwise stated. In addition, terms such as "...unit" and "module" described in the specification mean a unit that processes at least one function or operation, which may be implemented as hardware or software, or a combination of hardware and software. .

Throughout the specification, “touch input” refers to a gesture, etc., performed by a user on a touch screen to control a device. For example, the touch input described herein may include a tap, a touch & hold, a double tap, a drag, a panning, a flick, a drag and drop, and the like.

"Tap" refers to an action of a user touching a screen using a finger or a touch tool (eg, an electronic pen) and then immediately lifting it off the screen without moving.

"Touch & hold" refers to an operation in which a user touches a screen using a finger or a touch tool (eg, an electronic pen) and then maintains a touch input for a threshold time (eg, 2 seconds) or longer. For example, it means a case in which the time difference between the touch-in time and the touch-out time is equal to or greater than a threshold time (eg, 2 seconds). In order to make the user recognize whether the touch input is a tap or touch & hold, if the touch input is maintained for a threshold time or longer, a feedback signal may be provided visually, auditory or tactile. The threshold time may be changed according to an implementation example.

"Double tap" refers to an operation in which a user touches a screen twice using a finger or a touch tool (eg, an electronic pen).

“Drag” refers to an operation in which a user touches a finger or a touch tool on the screen and then moves the finger or the touch tool to another location on the screen while maintaining the touch. An object is moved due to the drag operation or a panning operation, which will be described later, is performed.

“Panning” refers to a case in which a user performs a drag operation without selecting an object. Because panning does not select a specific object, the page itself moves within the screen rather than the object moving within the page, or a group of objects moves within the page.

A “flick” refers to an action in which a user drags with a finger or a touch tool at a threshold speed (eg, 100 pixel/s) or more. Drag (or panning) and flick may be distinguished based on whether the moving speed of the finger or the touch tool is equal to or greater than a threshold speed (eg, 100 pixel/s).

"Drag & drop" refers to an action in which a user drags an object to a predetermined position on the screen using a finger or a touch tool and then drops the object.

“Pinch” refers to a motion in which the user moves in different directions while touching the screen with two fingers. It is a gesture to enlarge (Pinch Open) or Reduce (Pinch Close) an object or page, and the enlargement or reduction value is determined according to the distance between two fingers.

A “swipe” is an action of moving a predetermined distance in a horizontal or vertical direction while touching an object on the screen with a finger or a touch tool. A movement in an oblique direction may not be recognized as a swipe event.

Hereinafter, with reference to the accompanying drawings, the embodiments of the present invention will be described in detail so that those of ordinary skill in the art can easily implement them. However, the present invention may be embodied in several different forms and is not limited to the embodiments described herein. And in order to clearly explain the present invention in the drawings, parts irrelevant to the description are omitted, and similar reference numerals are attached to similar parts throughout the specification.

1 is a diagram for explaining a content reproduction system according to an embodiment of the present invention.

1 , according to an embodiment of the present invention, a content reproduction system may include a device 100 and an input tool. Hereinafter, each configuration will be described in detail.

According to an embodiment of the present invention, the device 100 may be a display device that displays handwritten content. For example, the device 100 described herein is a mobile phone, a smart phone, a laptop computer, a tablet PC, an e-book terminal, a digital broadcasting terminal, a PDA (Personal Digital Assistants), a PMP ( Portable Multimedia Player), navigation, MP3 player, digital camera, wearable device (eg, glasses, wrist watch), etc. may be included, but is not limited thereto.

Throughout the specification, “handwriting content” may refer to content obtained by converting a user's analog handwriting input into digital information. For example, the handwritten content may include, but is not limited to, a handwritten image drawn on a digital device, handwritten text, and the like.

The handwritten content 10 according to an embodiment of the present invention may be a set of strokes input in a predetermined time order. A “stroke” may refer to a single trajectory of drawing while maintaining a touch from a point in time when the input tool touches the device 100 . For example, in ‘3x + 6y = 5’, if the user draws each of 3, x, 6, and y at once while attracting a touch, each of 3, x, 6, and y may constitute one stroke. In the case of ‘+’, since the user draws ‘-’ first and then ‘ㅣ’, ‘-’ and ‘ㅣ’ can constitute one stroke, respectively. According to an embodiment of the present invention, one stroke may constitute one character or symbol or a combination of a plurality of strokes may constitute one character or symbol.

Meanwhile, each of the strokes included in the handwritten content 10 is associated with time information. In this case, the time information of the stroke may correspond to a time when the stroke is generated or changed. In addition, the stroke time information may be recorded according to a time stamp format indicating a specific time.

According to an embodiment of the present invention, the device 100 may receive a user's touch input. For example, when the device 100 displays the first handwritten content 10 including the mathematical problem solving process on the screen, the device 100 displays the region of interest in the first handwritten content 10 through the input tool. A touch input designating (RoI, Region of Interest) 20 may be received. The region of interest 20 may be a region including an object of interest to the user (eg, some handwritten content or at least one stroke).

According to an embodiment of the present invention, the input tool may be a tool for the user to input specific information into the device 100 . For example, the input tool may include a finger, an electronic pen (eg, a stylus pen), but is not limited thereto. Hereinafter, for convenience of description, the case where the input tool is the electronic pen 200 will be described as an example.

According to an embodiment of the present invention, the device 100 may display a virtual control panel 30 including a predetermined button on the screen. For example, the device 100 may include at least one button (eg, a button that can control playback of some handwritten content (eg, -3x + 9y = 0, 0x + 15y = 0) included in the region of interest 20 ). , forward button, rewind button, playback button, pause button, playback speed control button, etc.) may be displayed on the control panel 30 .

Hereinafter, a method in which the device 100 reproduces some handwritten content will be described in more detail with reference to FIG. 2 .

2 is a flowchart illustrating a method for a device to reproduce some handwritten content according to an embodiment of the present invention.

In operation S210, the device 100 may display the handwritten content on the screen.

According to an embodiment of the present invention, the handwritten content may include a plurality of strokes input in a predetermined order. Each of the plurality of strokes included in the handwritten content may have various shapes. For example, the thickness, color, brightness, saturation, transparency, etc. of each of the plurality of strokes may be different.

According to an embodiment of the present invention, the handwritten content displayed on the device 100 may be generated by the device 100 or received from an external device (eg, the electronic pen 200, a server, or an external device). it may be For example, the device 100 may generate handwritten content including a plurality of strokes for a predetermined time based on a user request. An operation of generating the handwritten content in the device 100 will be described later in detail with reference to FIGS. 3 to 5 . Also, the device 100 may receive the handwritten content generated in the external device from the external device.

According to an embodiment of the present invention, the device 100 may display a list of pre-stored handwritten content on the screen. The device 100 may receive a user input for selecting one handwriting content from a list of pre-stored handwriting content. For example, the user may select one handwriting content from a list of pre-stored handwriting content by using an input tool (eg, a finger or an electronic pen). The device 100 may extract the selected handwritten content from the memory and display it on the screen.

According to an embodiment of the present invention, the device 100 may display the handwritten content in the execution window of a specific application executed in the device 100 . For example, the specific application may be a handwritten note application, a memo application, a diary application, a calendar application, an e-book application, a household account application, a letter writing application, an online learning application, and the like, but is not limited thereto.

Meanwhile, according to an embodiment of the present invention, the handwritten content may be synchronized with the moving picture content or the audio content. In this case, the device 100 may display video content or audio content together with the handwritten content on the screen.

In operation S220 , the device 100 may receive a user input requesting reproduction of some handwritten content among the handwritten content displayed on the screen. Some handwritten content may be part of the handwritten content. According to an embodiment of the present invention, a user input requesting reproduction of some handwritten content may be varied.

For example, the device 100 may receive a user input for selecting some handwritten content from among handwritten content. The selection of some handwritten content may include highlighting some handwritten content, encircling some handwritten content, bracketing some handwritten content, or the behavior of the application in which the handwritten content is displayed. It can be implemented by selecting some handwritten content according to In this case, the device 100 may display a control panel including a playback function button. The device 100 may receive a user input for selecting a play function button through the control panel, and may play some handwritten content.

Also, the device 100 may receive a preset gesture corresponding to the playback command. For example, when the user double-taps the selected handwritten content with two fingers after selecting some handwritten content by a drag input, the device 100 may determine that a reproduction request for the partial handwritten content has been received.

Meanwhile, the user may input a reproduction command for some handwritten content to the device 100 by using a touch input and a voice input. For example, the user may select some handwritten content through a touch input and input a command for requesting reproduction of some handwritten content into the device 100 by voice.

In operation S230 , the device 100 may reproduce the partial handwritten content by using time information of strokes included in the partial handwritten content. In this specification, reproducing some handwritten content may mean reproducing a drawing process of strokes included in some handwritten content.

According to an embodiment of the present invention, the time information of the strokes may include at least one of relative time order information between the strokes, and time stamp information corresponding to each of the strokes. Meanwhile, the time stamp information may be absolute time information (eg, year, month, day, hour, minute, second, 2014.05.18.08.53.36) of the device 100 when the stroke is generated, and may be the first stroke of the first stroke. It may be an arbitrarily assigned number (eg, 0001, 0002, 0003, etc.) indicating an offset from the reference time, such as 1 timestamp.

According to an embodiment of the present invention, the device 100 may extract strokes included in some handwritten content. The device 100 may determine an order between the extracted strokes by using time stamp information of each of the extracted strokes. For example, a first stroke input first among extracted strokes is determined as a start position, and an n-th stroke (n is an integer greater than 1) input last among the extracted strokes ends. It can be determined by the end position. The device 100 may reproduce from the first stroke determined as the start position to the nth stroke determined as the end position by using time stamp information and coordinate information of the extracted strokes.

A method for the device 100 to reproduce some handwritten content will be described later in more detail with reference to FIG. 30 . Hereinafter, an operation in which the device 100 directly generates handwritten content will be described with reference to FIG. 3 .

3 is a flowchart illustrating a method for a device to generate handwritten content according to an embodiment of the present invention.

In operation S310, the device 100 may receive a handwritten content creation request. For example, the device 100 may receive an input for selecting a predetermined button corresponding to the handwritten content creation request. Also, the device 100 may recognize a predetermined voice or a predetermined motion corresponding to the handwritten content creation request.

In operation S320 , the device 100 may obtain time stamp information and coordinate information corresponding to each of a plurality of strokes input for a predetermined time. For example, the device 100 obtains time stamp information and coordinate information corresponding to each of a plurality of strokes by tracking coordinate values at a predetermined time interval (eg, 0.01 second) for each stroke drawn on the device 100 . can do. The coordinate values may be coordinate values referring to a physical location on the display of the device 100 , such as an origin on the display or a virtual location associated with a note application.

In operation S330 , the device 100 may generate or store the handwritten content using time stamp information and coordinate information corresponding to each of the plurality of strokes. An example of generating handwritten content in the device 100 will be described in detail with reference to FIGS. 4A to 4C .

4A to 4C are diagrams for explaining an example of generating handwritten content in a device.

Referring to FIG. 4A , the device 100 may execute a note application and display a note application execution window. In this case, the device 100 may receive a user input for selecting the sketch recording button 410 included in the note application execution window. The device 100 may start sketch recording in response to a user input of selecting the sketch recording button 410 . For example, the device 100 may record time stamp information and coordinate information of strokes input by the user after the sketch recording button 410 is selected.

Referring to FIG. 4B , the device 100 may receive an input for drawing a sheet music 420 from a user. For example, the device 100 may sequentially receive an input for drawing a staff, an input for drawing a treble clef, and an input for drawing a musical note. In this case, the device 100 may sequentially record time stamp information and coordinate information for each of the staff, the treble clef, and the note. Also, in response to the startup of the note application, the note application may automatically record timestamp information and coordinate information.

Referring to FIG. 4C , the user may select the sketch record button 410 again after drawing the sheet music 420 . In this case, the device 100 may end sketch recording. When the user selects the save button 430, the device 100 may generate handwritten content corresponding to the sheet music by storing time stamp information and coordinate information for each of the staff, treble clef, and notes. Also, the device 100 may automatically record strokes, and the record button 410 and the save button 430 may be optional.

Meanwhile, although an example in which the device 100 generates handwritten content has been described in FIGS. 4A to 4C , the present invention is not limited thereto. According to an embodiment of the present invention, the electronic pen 200 may generate handwritten content by recording time stamp information and coordinate information of strokes. In this case, the electronic pen 200 may transmit the handwriting content to the device 100 through the communication link.

5 is a diagram for explaining an example of a storage format of handwritten content.

In FIG. 5 , a case in which the device 100 generates the handwritten content 500 drawn in the order of the staff, the treble clef, and the notes will be described as an example.

When a stroke input is received, the device 100 may store a table in which the stroke identification information 510 and the first time stamp information 520 are mapped. For example, the device 100 gives a first time stamp (eg, 002525) to the first stroke (S1) according to the input order, and a second time stamp (eg, 002688) to the second stroke (S2) ), a third time stamp (eg, 003900) may be given to the third stroke S3, and a fourth time stamp (eg, 004566) may be given to the fourth stroke S4.

Also, the device 100 may record the coordinate information 530 for each stroke. In this case, the device 100 may store a table in which coordinate values included in the coordinate information 530 are mapped with the second time stamp information 540 . For example, the device 100 maps the first coordinate values (eg, x0, y0) of the time point when the input of the first stroke S1 is generated and the 1-1 timestamp (eg, '000000') and the second coordinate value (eg, x1, y1) of the first stroke S1 and the 1-2 timestamp (eg, '000003) are mapped, and the third coordinate value (eg, '000003) of the first stroke S1 , x2, y2) and the 1-3 timestamp (eg, '000004) may be mapped.

According to an embodiment of the present invention, the device 100 may calculate the drawing speed of the strokes by using the coordinate information 530 and the second time stamp information 540 . For example, the device 100 corresponds to a distance between the first coordinate value (eg, x0, y0) and the second coordinate value (eg, x1, y1) and the first coordinate value (eg, x0, y0) Using the 1-1 timestamp (eg, '000000') and the second coordinate value (eg, x1, y1) corresponding to the 1-2 timestamp (eg, '000003), the first coordinate value ( For example, the drawing speed from x0, y0) to the second coordinate value (eg, x1, y1) may be calculated.

6 is a flowchart illustrating a method for a device to synchronize and store handwritten content and audio content, according to an embodiment of the present invention.

In operation S610, the device 100 may receive a handwritten content creation request.

For example, the device 100 may receive an input for selecting a predetermined button corresponding to the handwritten content creation request. Also, the device 100 may recognize a predetermined voice or a predetermined motion corresponding to the handwritten content creation request.

According to an embodiment of the present invention, the device 100 may obtain time stamp information and coordinate information corresponding to each of a plurality of strokes input for a predetermined time in response to a request for generating handwritten content.

In operation S620, the device 100 may acquire the audio content while generating the handwritten content. According to an embodiment of the present invention, the audio content may be a voice, music, or the like, but is not limited thereto. In addition, the voice may be the voice of a user who inputs handwriting or the voice of another user (eg, a lecturer or a professor).

For example, the device 100 may sense an external sound through a microphone while sensing a user's handwriting input through the touch screen.

In operation S630, the device 100 may store the handwritten content in synchronization with the audio content. For example, when the input of the first stroke and the input of the first voice are simultaneously performed at the first time point, the device 100 performs the first stroke information (eg, identification information of the first stroke and coordinate information) and the first A table to which voice information is mapped can be saved. Referring to FIGS. 7A to 7E , it will be described in more detail.

7A to 7E are diagrams for explaining an example in which a device synchronizes and stores handwritten content and audio content.

7A to 7E , a case in which the device 100 generates the handwriting content 700 for drawing the Pythagorean theorem and a formula for calculating the area of a figure will be described as an example.

Referring to FIG. 7A , the device 100 may execute a note application and display a note application execution window. In this case, the device 100 may receive a user input for selecting the sketch record button 710 included in the note application execution window. The device 100 may start sketch recording in response to a user input of selecting the sketch recording button 710 .

Referring to FIG. 7B , the device 100 may receive a user input for selecting the record button 720 . The device 100 may activate the microphone according to a user input for selecting the record button 720 . The device 100 may sense an external sound through the activated microphone.

In addition, the device 100 may provide a palette tool 730 . In this case, the user may select a pencil type, color, thickness, etc. through the palette tool 730 .

Referring to FIGS. 7C and 7D , the device 100 uses an input for drawing a right triangle from the user, an input for writing the Pythagorean theorem (eg, a2+b2=c2), an input for drawing a rectangle, and a formula for calculating the area of the rectangle. An input to take notes (eg, A = bh) may be sequentially received. In this case, the device 100 may sequentially record time stamp information and coordinate information for each of a right triangle, the Pythagorean theorem, a rectangle, and a formula for calculating the area of a rectangle.

Meanwhile, the device 100 sequentially receives an input for drawing a right triangle and an input for writing down the Pythagorean theorem (eg, a2+b2=c2) from a user while receiving an external voice 740 (eg, Pythagoras') theorem can be written as an equation relating the lengths of the sides a, b, and c, …).

In this case, the device 100 may synchronize and record information on the voice 740 with information on strokes corresponding to a right triangle and the Pythagorean theorem (eg, time stamp information and coordinate information).

Referring to FIG. 7E , the user may select the sketch record button 710 again after completing the handwriting input. In this case, the device 100 may end sketch recording. When the user selects the save button 750, the device 100 is a right triangle, the Pythagorean theorem, a rectangle, a formula for finding the area of a rectangle, a sphere, a formula for finding the volume of a sphere, a circle, a formula for finding the area of a circle, parallel By storing time stamp information and coordinate information for each of the formulas for calculating the area of a quadrilateral and a parallelogram, the handwritten content 700 may be generated.

Also, the device 100 may generate audio content for a voice input from the outside through a microphone. In addition, the device 100 may synchronize the audio content and the handwriting content by comparing the input timing of the handwriting input and the voice input.

Meanwhile, although an example in which the device 100 generates handwritten content has been described in FIGS. 7A to 7E , the present invention is not limited thereto. According to an embodiment of the present invention, the electronic pen 200 may generate handwritten content by recording time stamp information and coordinate information of strokes. In addition, the electronic pen 200 may detect an external voice when generating the handwritten content, and store the handwritten content and audio content in synchronization. The electronic pen 200 may transmit the handwriting content synchronized with the audio content to the device 100 through the communication link.

8 is a diagram for explaining an example of a storage format of handwritten content and audio content.

Referring to FIG. 8 , when the audio content 810 is acquired while the handwritten content 820 is being generated, the device 100 may map information of the audio content 810 and information of the handwritten content 820 and store it. For example, the device 100 obtains identification information 811 of the first audio content (eg, 01.mp3), the timeline information 812, and identification information 813 of strokes included in the handwriting content 820. It can be mapped and saved.

When a stroke input is received, the device 100 may store a table in which identification information 813 of strokes and first time stamp information 814 are mapped. The device 100 may record coordinate information 815 for each stroke. In this case, the device 100 may store a table in which coordinate values included in the coordinate information 815 are mapped with the second time stamp information 816 . Since the description of the storage format of the handwritten content 820 overlaps with the description of FIG. 5 , a detailed description thereof will be omitted.

9 is a flowchart illustrating a method for a device to synchronize and store handwritten content and video content, according to an embodiment of the present invention.

In step S910 , the device 100 may reproduce video content. There may be various types of video content. For example, the video content includes an educational video (eg, a cooking class video, a math lecture video, etc.), an exercise-related video (eg, a yoga video, a stretching video), a game video, and a user-created content (UCC: User Created Contents) video. , a presentation image, a conference image, a medical image, etc., but is not limited thereto.

In operation S920, the device 100 may receive a handwritten content creation request during video playback. For example, the device 100 may receive an input requesting execution of an application for handwriting input. The device 100 may display an execution window of an application for a handwriting input (eg, a memo application) and receive the handwriting input through the execution window. The device 100 may display an execution window of an application for handwriting input on the video content being reproduced, or may display an execution window of an application for handwriting input so as not to overlap with the video content being reproduced.

In operation S930, the device 100 may generate handwritten content including a plurality of strokes during reproduction of the moving image content in response to the request for generating the handwritten content. For example, the device 100 may receive the input of the first stroke, the second stroke, and the third stroke during reproduction of the first section of the first moving picture content. In this case, the device 100 may generate handwritten content including the first stroke, the second stroke, and the third stroke.

In operation S940, the device 100 may store the handwritten content in synchronization with the moving image content. For example, when the input of the first stroke is received during reproduction of the first section of the first moving picture content, the device 100 displays the first stroke information (eg, identification information of the first stroke; coordinate information) can be mapped and stored. Referring to FIGS. 10A and 10B , a more detailed description will be given. Although it has been described above that audio content or video content can be synchronized with handwritten content, various types of content can be associated with handwritten content. For example, both audio content and moving picture content may be associated with the handwritten content, and audio content and still image content may be associated with the handwritten content.

10A and 10B are diagrams for explaining an example in which a device synchronizes and stores handwritten content and video content.

Referring to FIG. 10A , the device 100 may play a cooking lecture video 1000 . In this case, when the user touches the electronic pen 200 on the screen of the device 100 for a predetermined time or longer, the device 100 may provide the menu window 1010 . When the user selects the icon of the memo application in the menu window 1010 , the device 100 may execute the memo application. Also, the memo application may be enabled by default.

Referring to FIG. 10B , the device 100 may display the cooking course video and the memo application execution window together by adjusting the size of the cooking class video 1000 . In this case, the device 100 may receive a handwriting input from the user through the memo application execution window. For example, the device 100 may receive a handwriting input for recording a recipe through a memo application execution window.

11A is a diagram for explaining an example of a storage format of handwritten content and video content.

Referring to FIG. 11A , when a handwriting input is received while playing a video, the device 100 may map information of the video content 1110 and information of the handwriting content 1120 and store it. For example, the device 100 may include identification information 1111 of the first video content (eg, 01.avi), timeline information 1112 , and identification information 1113 of strokes included in the handwriting content 1120 . can be mapped and saved.

When a stroke input is received, the device 100 may store a table in which identification information 1113 of strokes and first time stamp information 1114 are mapped. The device 100 may record coordinate information 1115 for each stroke. In this case, the device 100 may store a table in which coordinate values included in the coordinate information 1115 are mapped with the second time stamp information 1116 . Since the description of the storage format of the handwritten content 1120 overlaps with the description of FIG. 5 , a detailed description thereof will be omitted.

11B is a diagram for explaining connection information connecting a stroke of handwritten content and a frame of moving image content.

Referring to FIG. 11B , the device 100 may store connection information connecting a stroke of the handwritten content 1120 and a frame of the video content 1110 .

For example, the device 100 may include a start coordinate value (eg, x0, y0) of a first stroke (eg, s1) and a first frame (eg, start frame) of the first video content (eg, 01.avi) , time: 000000) can be connected to store connection information. In addition, the device 100 determines the last coordinate value (eg, x3, y3) of the first stroke (eg, s1) and the second frame (eg, time: 000003) of the first video content (eg, 01.avi) The device 100 may store the connection information linking the start coordinate values (eg, x11, y11) of the fourth stroke (eg, s4) and the third frame of the first video content (eg, 01.avi) (eg, time: 000560) may be stored as connection information.

According to an embodiment of the present invention, the device 100 may connect the start (or end) coordinates of the specific stroke with the identification information of the specific frame. Accordingly, when the user selects a specific stroke, the device 100 may extract a specific frame connected to the specific stroke. Conversely, when the user selects a specific frame, the device 100 may extract a specific stroke connected to the specific frame.

12, 13A, and 13B are diagrams for explaining a method of storing handwritten content, audio content, and video content in a device.

12 , according to an embodiment of the present invention, the device 100 may store handwritten content in units of pages. In this case, the device 100 may generate one handwritten content file 1200 including the stroke information 1210 and the audio/video content information 1220 . According to an embodiment of the present invention, in the stroke information 1210, coordinate information of each of the strokes, a color value corresponding to each of the strokes, a thickness value, a style value (eg, highlighter, pencil type, brush type), etc. may be included.

According to an embodiment of the present invention, the device 100 may map all coordinate values constituting a stroke to time stamps (or frames) of the multimedia content, respectively. According to an embodiment of the present invention, the device 100 may map one coordinate value among the coordinate values constituting the stroke and identification information of the multimedia content. For example, the device 100 may map the start coordinate value (or the last coordinate value) of the stroke and identification information of the specific multimedia content.

According to an embodiment of the present invention, the device 100 may map the stroke identification information and the time stamp (or frame) of the multimedia content, and may map the stroke identification information and the multimedia content identification information. .

As shown in FIG. 13A , according to another embodiment of the present invention, the device 100 may separately generate a handwritten content file 1310 and an audio/video content file 1320 . In this case, the handwritten content file 1310 may include link address information 1312 of audio/video content in addition to the stroke information 1311 . According to an embodiment of the present invention, the link address information 1312 of the audio/video content may include identification information of the audio/video content and timeline information corresponding to each stroke.

13A illustrates an example in which link address information 1312 of audio/video content is included in the handwritten content file 1310, but is not limited thereto. For example, link address information of a stroke may be stored in the audio/video content file 1320 .

13B is a diagram for explaining link address information 1312 .

Referring to FIG. 13B , the device 100 may generate link address information 1312 by mapping a partial coordinate value of a stroke included in the handwritten content and identification information of at least one content. In this case, the link address information 1312 may include a matching table in which a partial coordinate value of a stroke included in the fill-key content is matched with identification information of the content.

For example, when the user adds a memo for the twelfth photo image, the device 100 sets the start coordinate values (eg, x0, y0) of the first stroke (eg, S1) included in the memo and the twelfth photo The image identification information (eg, photo image 12) may be mapped and stored.

In addition, when the user adds a memo to the device 100 in relation to the first application (or icon), the device 100 stores the identification information (eg, Application 1) of the first application (or icon) and the handwritten content. The start coordinate values (eg, x7, y7) of the included fourth stroke (eg, S4) may be mapped and stored.

On the other hand, when the user adds a memo to the device 100 in relation to the video content, the device 100 provides identification information of the video content (eg, Video 3) and the ninth stroke (eg, S9) included in the handwritten content. The starting coordinate values (eg, x20, y20) may be mapped and stored.

In FIG. 13B , the mapping between the start coordinate value of the stroke and the content identification information has been described as an example, but the present invention is not limited thereto. For example, the device 100 may map a stroke time stamp and content identification information. Also, the device 100 may map the identification information of the stroke and the identification information of the content.

Accordingly, according to an embodiment of the present invention, when the user selects a specific stroke, the device 100 may extract specific content (eg, video, audio, icon, application, photo, etc.) connected to the specific stroke. have. Conversely, when the user selects a specific content, the device 100 may extract a specific stroke connected to the specific content.

14 is a diagram for explaining a method for a device to store handwritten content in synchronization with emotion information, according to an embodiment of the present invention.

In operation S1410, the device 100 may receive a request for generating the handwritten content. For example, the device 100 may receive an input for selecting a predetermined button corresponding to the handwritten content creation request. Also, the device 100 may recognize a predetermined voice or a predetermined motion corresponding to the handwritten content creation request.

According to an embodiment of the present invention, the device 100 may obtain time stamp information and coordinate information corresponding to each of a plurality of strokes input for a predetermined time in response to a request for generating handwritten content.

In operation S1420, the device 100 may acquire the user's emotion information while generating the handwritten content. According to an embodiment of the present invention, the emotional information may include the user's mood state information (eg, joy, sadness), biometric information (eg, pulse information, electrocardiogram information, temperature information), facial expression information, and the like.

According to an embodiment of the present invention, the device 100 may receive pulse information, an electrocardiogram degree, temperature information, and the like from an external wearable device. For example, the device 100 may receive pulse information and temperature information from the smart watch. According to an embodiment of the present invention, the device 100 may receive pulse rate information from an external wearable device through short-range communication (eg, Bluetooth, NFC, Wi-Fi Direct, etc.).

Also, according to an embodiment of the present invention, the device 100 may analyze the user's facial expression using an image sensor. For example, the device 100 may directly monitor the user's facial expression using an image sensor, or monitor the user's facial expression through an external wearable device (eg, smart glasses).

According to an embodiment of the present invention, the device 100 may acquire the user's emotion information based on information input from the user. For example, the device 100 may analyze text or an image input from the user, or analyze the user's voice pattern. In this case, the device 100 may infer the user's emotional state by comparing the user's voice pattern information with a pre-stored voice pattern template.

In operation S1430, the device 100 may synchronize and store the emotion information and the handwritten content.

For example, when the input of the first stroke and the acquisition of the first emotion information are simultaneously performed at the first time point, the device 100 may include the first stroke information (eg, identification information of the first stroke, coordinate information) and the second stroke information. 1 A table to which emotion information is mapped can be stored. Referring to FIG. 15 , it will be described in more detail.

15 is a diagram for describing an example in which a device acquires emotion information while generating handwritten content. In FIG. 15 , a case in which the device 100 generates handwritten content for drawing a formula for obtaining the Pythagorean theorem and the area of a figure will be described as an example.

Referring to FIG. 15 , the device 100 may execute a note application and display a note application execution window. In this case, the device 100 may receive a user input for selecting a sketch recording button included in the note application execution window. The device 100 may start sketch recording in response to a user input of selecting a sketch recording button.

For example, the device 100 provides an input for drawing a right triangle, an input for writing the Pythagorean theorem (eg, a2+b2=c2), an input for drawing a rectangle, an input for writing a formula for calculating the area of the rectangle ( For example, A = bh) may be sequentially received. In this case, the device 100 may sequentially record time stamp information and coordinate information for each of a right triangle, the Pythagorean theorem, a rectangle, and a formula for calculating the area of a rectangle.

Meanwhile, the device 100 receives the user's biometric information from the external wearable device 1500 while sequentially receiving an input for drawing a right triangle and an input for writing the Pythagorean theorem (eg, a2+b2=c2) from the user can receive For example, the device 100 may receive pulse rate information measured by the wearable device 1500 from the wearable device 1500 .

In this case, the device 100 may synchronize and record information on strokes corresponding to a right triangle and the Pythagorean theorem (eg, time stamp information and coordinate information) and pulse rate information.

16 is a flowchart illustrating a method for a device to select some handwritten content corresponding to a region of interest according to an embodiment of the present invention.

In operation S1610, the device 100 may display the handwritten content. According to an embodiment of the present invention, the device 100 may display handwritten content on an execution window of a predetermined application. According to an embodiment of the present invention, the device 100 may receive a selection of the specific handwritten content from the user and display the selected specific handwritten content.

In operation S1620, the device 100 may receive an input specifying an ROI on the handwritten content displayed on the screen.

According to an embodiment of the present invention, a user input for selecting an ROI may be various. For example, the user input may be at least one of a key input, a touch input (eg, tap, double tap, touch and drag, flick, swipe, etc.), a voice input, a motion input, and a multi-input, but is not limited thereto. it is not

Meanwhile, according to an embodiment of the present invention, the shape of the ROI may vary. For example, the region of interest may have a circular shape, an oval shape, a quadrangle shape, a free curve shape, or the like, but is not limited thereto. Also, the color and pattern of the ROI may be varied.

In operation S1630, the device 100 may select some handwritten content corresponding to the ROI.

According to an embodiment of the present invention, the device 100 may select some handwritten content corresponding to the ROI based on a stroke unit or a coordinate unit. For example, when some handwritten content is selected based on a stroke unit, the device 100 may extract strokes located in the ROI or strokes overlapping the ROI. When some handwritten content is selected based on a coordinate unit, the device 100 may select coordinate values overlapping the ROI as some handwritten content. An input for designating an ROI will be described in detail with reference to FIGS. 17 to 20B .

17 is a diagram for explaining an example of an input for designating a region of interest.

Referring to FIG. 17 , the device 100 may designate a rectangular region 1710 based on two points as a region of interest. For example, when the user touches two points on the handwritten content, the device 100 may designate a rectangular area 1710 formed based on the two points as the ROI.

When the user draws a free curve on the handwritten content, the device 100 may designate the closed region 1720 formed by connecting the free curve as the region of interest. Also, the device 100 may designate the circle region 1730 based on the center point and the radius as the region of interest.

Meanwhile, when the user draws a line on the handwritten content, the device 100 may designate a region 1740 in which strokes intersecting the line are displayed as the region of interest.

18, 19, 20A, and 20B are diagrams for explaining an example of a user interface for designating a region of interest. 18, 19, 20A, and 20B, a case in which a user desires to designate a circle and an area including a formula for calculating the area of a circle as a region of interest will be described as an example.

As shown in FIG. 18 , the device 100 may provide a tool bar including a button 1800 for designating a region of interest. A user may select a button 1800 from the toolbar to designate a region of interest. In this case, the user may further select the shape of the region of interest (eg, a rectangle, a circle, a triangle, a free-form curve). For example, the user may select a circle as the shape of the region of interest.

The device 100 may designate, as a region of interest, a circle and a circular region 1820 including a formula for calculating the area of a circle, based on a user's touch input.

As shown in FIG. 19 , according to an embodiment of the present invention, a user draws a free curve to include a circle and a formula for calculating the area of a circle while pressing a specific button 201 attached to the electronic pen 200 . can draw In this case, the device 100 may select a lung region formed as a free curve as a region of interest.

As shown in FIG. 20A , according to an embodiment of the present invention, the user may touch the electronic pen 200 on the screen of the device 100 for a predetermined time or longer (eg, 2 seconds or longer). In this case, the device 100 may display a menu window 2010 including an icon 2011 capable of designating a region of interest within a predetermined distance from the touched point.

Referring to FIG. 20B , the user may select an icon 2011 included in the menu window 2010 using the electronic pen 200 . In this case, the device 100 may provide a rectangle 2020 in which an ROI can be set. The user may move the rectangle 2020 to include a circle and a formula for calculating the area of the circle. In addition, the user may enlarge or reduce the region of interest.

21 and 22 are diagrams illustrating an example of a graphical user interface (GUI) for determining some handwritten content corresponding to a region of interest. In FIG. 21 , a case in which ‘M’ constitutes the first stroke and ‘erry’ constitutes the second stroke will be described as an example.

According to an embodiment of the present invention, the device 100 may provide menus corresponding to a method for determining a stroke corresponding to the ROI.

For example, the device 100 may include a first menu 2110 for selecting strokes overlapping the region of interest as partial handwritten content corresponding to the region of interest, and select strokes that are entirely included in the region of interest as partial handwriting content corresponding to the region of interest. A second menu 2120 for selecting handwritten content and a third menu 2130 for selecting coordinate values existing in the ROI as some handwritten content corresponding to the ROI may be provided.

Referring to 2210 of FIG. 22 , the user may select the first menu 2110 and position the ROI 2200 to include only 'Merr' among all handwritten content (eg, Merry Christmas). In this case, the device 100 may select the first stroke 'M' and the second stroke 'erry' overlapping the region of interest 2200 as some handwritten content corresponding to the region of interest 2200 .

Referring to 2220 of FIG. 22 , the user may select the second menu 2120 and position the ROI 2200 to include only Merr among all handwritten content (eg, Merry Christmas). In this case, the device 100 may select only the first stroke 'M' including the entire stroke in the region of interest 2200 as some handwritten content corresponding to the region of interest 2200 .

Referring to 2230 of FIG. 22 , the user may select the third menu 2130 and position the ROI 2200 to include only Merr among all handwritten content (eg, Merry Christmas). In this case, the device 100 may select 'Merr' included in the ROI 2200 as some handwritten content corresponding to the ROI based on the coordinate information.

Also, the device 100 may select strokes based on a dictionary. For example, the device 100 may determine that 'Merr' corresponding to the ROI forms the word 'Merry'. Accordingly, the device 100 may select all of the corresponding strokes of the word 'Merry' even if the strokes of the word exist outside the region of interest. The device 100 may, therefore, refer to strokes stored before or after strokes in the ROI to determine whether the strokes in the selected ROI are part of a word.

Also, the device 100 may determine whether strokes in the ROI correspond to strokes of a phrase. Accordingly, the device 100 may select all strokes corresponding to the phrase. For example, when the region of interest of FIG. 22 is selected, the device 100 may select strokes corresponding to the phrase 'Merry Christmas'.

As described above, a portion of the handwritten content may be selected through the region of interest. Thus, a word, sentence, letter, phrase, paragraph, or group of letters may be selected.

23A is a diagram illustrating an example in which a device displays a GUI for determining a stroke corresponding to a region of interest in a pop-up window. In FIG. 23A , a case in which ‘M’ constitutes the first stroke and ‘erry’ constitutes the second stroke will be described as an example.

Referring to FIG. 23A , the device 100 may receive an input for locating the ROI 2310 to include only 'Merr' among all handwritten content (eg, Merry Christmas).

The device 100 provides a pop-up window 2320 including a first menu 2110 , a second menu 2120 , and a third menu 2130 in response to an input for locating the region of interest 2310 . can According to an embodiment of the present invention, the device 100 may display the pop-up window 2320 within a predetermined distance from the ROI 2310 .

For example, the user may position the region of interest 2310 to include only 'Merr' among all handwritten content (eg, Merry Christmas). In this case, the device 100 may display the first menu 2110 , the second menu 2120 , and the third menu 2130 around the ROI 2310 .

The user may select one of the first menu 2110 , the second menu 2120 , and the third menu 2130 from the pop-up window 2320 . For example, the user may select the second menu 2120 from the pop-up window 2320 . In this case, the device 100 may select only the first stroke 'M' including the entire stroke in the region of interest 2310 as some handwritten content corresponding to the region of interest 2310 .

23B is a diagram illustrating an example in which a device displays a guide image for guiding designation of a region of interest.

As shown in FIG. 23B , the device 100 may receive an input for locating the first region of interest 2330 to include only 'Merr' and 'Ch' among all handwritten content (eg, Merry Christmas). . In this case, the device 100 may guide the region 2340 that may include all strokes (eg, Merry Christmas) overlapping the first region of interest 2310 to be set as the region of interest. For example, the device 100 may display the guide image 2350 to increase the size of the first ROI 2330 .

24 is a diagram illustrating an example of a user interface for selecting some handwritten content in units of segments.

Referring to FIG. 24 , the device 100 may select a region of interest 2400 semi-automatically. According to an embodiment of the present invention, the device 100 may receive a selection for a specific point from the user. For example, the device 100 may receive an input of touching a portion in which ‘my’ is displayed in the entire handwritten content for a predetermined time or longer (eg, 2 seconds or longer).

According to an embodiment of the present invention, the device 100 parses the entire handwritten content, and a paragraph (eg, If I have lost confidence in myself, I have the universe against.) including a specific point (eg, my). can be extracted. For example, the device 100 may extract a paragraph including a specific point based on a paragraph mark (eg, comma, enter, etc.). The device 100 may designate the region of interest 2400 to include the entire extracted paragraph. Also, the device 100 may designate a word, a phrase, or a sentence.

According to another embodiment of the present invention, the device 100 may select the ROI 2400 of a predetermined size (eg, 10 pixels or 5 cm 2 ) based on a specific point selected by the user. The predetermined size may be preset by a user or may be preset by the device 100 .

25 is a diagram illustrating an example in which a device identifies and displays selected strokes to distinguish them from other strokes.

According to an embodiment of the present invention, the device 100 may identify and mark some handwritten content in order to identify some selected handwritten content from among all handwritten content.

For example, referring to 2510 of FIG. 25 , the device 100 may display a dotted line around a stroke included in some selected handwritten content.

Referring to 2520 of FIG. 25 , the device 100 may add a specific color around a stroke included in some selected handwritten content.

Referring to 2530 of FIG. 25 , the device 100 may change the color of a stroke included in some selected handwritten content and display it.

Referring to 2540 of FIG. 25 , the device 100 may change and display the thickness of a stroke included in some selected handwritten content.

The method of distinguishing the strokes is not limited to the above embodiment. Any method of distinguishing between unselected strokes and selected strokes may be used.

26 is a flowchart illustrating a method for a device to receive a play request through a control panel according to an embodiment of the present invention.

In operation S2610, the device 100 may display the handwritten content.

According to an embodiment of the present invention, the handwritten content may include a plurality of strokes input in a predetermined order. Each of the plurality of strokes included in the handwritten content may have various shapes. For example, the thickness, color, brightness, saturation, transparency, etc. of each of the plurality of strokes may be different.

According to an embodiment of the present invention, the device 100 may display handwritten content on an execution window of a predetermined application. According to an embodiment of the present invention, the device 100 may receive a selection of the specific handwritten content from the user and display the selected specific handwritten content.

In operation S2620, the device 100 may receive an input for selecting some handwritten content from among the handwritten content. For example, the device 100 may receive a user input designating a region of interest and select some handwritten content corresponding to the region of interest. Also, the device 100 may receive a user input of sequentially touching a plurality of strokes while pressing a predetermined button of the electronic pen 200 . In this case, the device 100 may select a plurality of strokes touched by the electronic pen 200 as some handwritten content.

In operation S2630, when some handwritten content is selected, the device 100 may display a control panel. According to an embodiment of the present invention, the device 100 may display the control panel within a predetermined distance from some handwritten content.

According to an embodiment of the present invention, the control panel may include at least one of a play function button and an edit function button. According to an embodiment of the present invention, the playback function button may include at least one of a playback button, a pause button, a forward button, a reverse button, and a playback speed control button.

Further, according to an embodiment of the present invention, the editing function button may include at least one of an undo button, a redo button, a character recognition button, an add button, and an exchange button.

Meanwhile, according to an embodiment of the present invention, the control panel may include a search bar for searching for a partial section of some handwritten content.

According to an embodiment of the present invention, the control panel may be displayed on the same layer as the layer on which the handwritten content is displayed, or may be displayed separately on a different layer.

In operation S2640, the device 100 may receive a button selection input from the user through the control panel. For example, the device 100 may receive a user input for selecting a play function button among the control panel. Hereinafter, the control panel will be described in more detail with reference to FIGS. 27A and 27B .

27A and 27B are diagrams for explaining an example in which the device provides a control panel including a playback function button or an edit function button.

As shown in FIG. 27A , the device 100 may display handwritten content 2710 including an image of a cat on the screen. In this case, the user may select the button 2700 for designating the region of interest 2720 from the tool bar using the electronic pen 200 . In addition, the user may use the electronic pen 200 to designate a region including the cat's head as the region of interest 2720 .

As shown in FIG. 27B , the device 100 may change the color, thickness, etc. of strokes 2740 included in the ROI 2720 based on a user input designating the ROI 2720 . . For example, the cat image may be entirely black, but only strokes 2740 of the head included in the region of interest 2720 may be displayed in blue.

According to an embodiment of the present invention, the device 100 may display a virtual control panel 2730 at the bottom of the ROI 2720 . The virtual control panel 2730 may include at least one of a play function button and an edit function button.

According to an embodiment of the present invention, when the user selects a predetermined button on the virtual control panel 2730 using the electronic pen 200 , the device 100 displays the head included in the region of interest 2720 . Reproduction or editing of the strokes 2740 of the portion may be performed.

According to another embodiment of the present invention, when the user touches the ROI 2720 for a predetermined time or longer, the device 100 may output a translucent window including an image of the cat's head. When the user moves the translucent window to the position of a specific button in the control panel 2730 while maintaining the touch input to the translucent window, the device 100 performs strokes of the head included in the region of interest 2720 ( 2740) can be played back or edited.

28 is a flowchart illustrating a method for a device to display some handwritten content selected by a user on a separate layer, according to an embodiment of the present invention.

In operation S2810, the device 100 may display the handwritten content on the first layer.

In operation S2820, the device 100 may receive an input for selecting some handwritten content from among the displayed handwritten content.

Steps S2810 and S2820 correspond to steps S2610 and S2620 of FIG. 26 , and thus a detailed description thereof will be omitted.

In operation S2830, the device 100 may display some handwritten content on a second layer that is distinct from the first layer on which the handwritten content is displayed. For example, the device 100 may display some handwritten content selected by the user from among the handwritten content in the pop-up window. The pop-up window may include a virtual control panel. For example, the pop-up window may include at least one of a playback function button, an editing function button, and a search bar. Reference will be made to FIG. 29 .

29 is a diagram illustrating an example in which a device displays some handwritten content selected by a user on a separate layer.

As shown in FIG. 29 , the device 100 may display handwritten content 2910 including an image of a cat on the screen. In this case, the user may select the button 2900 for designating the region of interest 2920 from the tool bar using the electronic pen 200 . In addition, the user may designate a region including the cat's head as the region of interest 2920 using the electronic pen 200 .

The device 100 may display strokes included in the region of interest 2920 in a separate pop-up window 2930 based on a user input designating the region of interest 2920 . The pop-up window 2930 may include at least one of a playback function button, an editing function button, and a search bar.

According to an embodiment of the present invention, when the user selects a predetermined button on the pop-up window 2930 using the electronic pen 200 , the device 100 displays the cat head included in the region of interest 2920 . Playback or editing may be performed on the strokes 2940 of .

30 is a flowchart illustrating a method for a device to reproduce some handwritten content by using time information of strokes in a region of interest, according to an embodiment of the present invention.

In operation S3005, the device 100 may receive an input for designating a region of interest with respect to the handwritten content. Since the operation of the device 100 for designating the region of interest based on the user input has been described in detail above, it will be omitted herein.

In operation S3010, the device 100 may receive an input requesting reproduction of some handwritten content in the ROI.

For example, the device 100 may display a virtual control panel including a play button in response to an input for designating a region of interest. The user may request playback of some handwritten content by selecting a play button on the virtual control panel.

In addition, the user may request playback of some handwritten content by selecting a physical playback button. For example, the device 100 may detect an input of pressing the home button more than a predetermined number of times (eg, twice or more) or an input of pressing the side button of the device 100 for a predetermined time or more (eg, 3 seconds or more). . In this case, the device 100 may determine the detected input as an input for requesting reproduction of some handwritten content.

Meanwhile, according to an embodiment of the present invention, a user may request reproduction of some handwritten content through a voice command.

In operation S3015, the device 100 may extract at least one stroke included in some handwritten content. For example, the device 100 may retrieve information on strokes included in some handwritten content (eg, stroke identification information, stroke time stamp information, and stroke coordinate information) from the memory.

In operation S3020 , the device 100 may generate a stack in which at least one stroke is inserted in chronological order. For example, the device 100 may determine the relative time order of the strokes by comparing the time stamp information between the extracted strokes. The device 100 may allow a stroke generated the longest among the extracted strokes to be inserted at the top of the stack, and the most recently generated stroke to be inserted as it goes down in the stack. In this regard, the order of the strokes may be determined.

In operation S3025 , the device 100 may extract the i-th coordinate value of the n-th stroke from the stack. According to an embodiment of the present invention, the device 100 may extract coordinate values from an upper layer to a lower layer of the stack. For example, the device 100 may first extract the first coordinate value of the first stroke from the stack. In this case, the first coordinate value of the first stroke may be the highest value in the stack, and may be a stroke in which the first stroke is generated first.

In operation S3030, the device 100 may output a point based on the i-th coordinate value. For example, the device 100 may output the first coordinate value of the first stroke, which is the highest value in the stack, as the first point.

In operation S3035 , the device 100 may determine whether the i-th coordinate value of the n-th stroke is the last value in the stack. For example, when the first coordinate value of the first stroke is the last value in the stack, the device 100 may output only a dot and end a reproduction operation for some handwritten content.

In operation S3040 , when the i-th coordinate value of the n-th stroke is not the last value in the stack, the device 100 may extract the i+1-th coordinate value of the n-th stroke from the stack. For example, the device 100 may extract the second coordinate value of the first stroke.

In operation S3045, the device 100 may determine whether the i+1th coordinate value of the nth stroke is a delimiter value. In the present specification, the separator value may be a value that distinguishes the nth stroke from the n+1th stroke. For example, the delimiter value may be the maximum value within the timestamp.

In operation S3050, when the i+1th coordinate value of the nth stroke is not a delimiter value, the device 100 may output a line connecting the i-th coordinate value and the i+1th coordinate value.

For example, when the second coordinate value of the first stroke is not a separator value, the device 100 may output a line connecting the first coordinate value of the first stroke and the second coordinate value of the first stroke have. According to an embodiment of the present invention, the device 100 uses a first coordinate value, a second coordinate value, time stamp information corresponding to the first coordinate value, and time stamp information corresponding to the second coordinate value, By interpolating between the first coordinate value and the second coordinate value, a line connecting the first coordinate value and the second coordinate value may be output. In this regard, strokes may be input and output in the same temporal manner as recorded.

In step S3055 , the device 100 may increase the value of i to i+1, and return to step S3035. For example, the device 100 may determine whether the i+1th coordinate value of the nth stroke is the last value in the stack. For example, it may be determined whether the second coordinate value of the first stroke is the last value in the stack.

When the i+1th coordinate value of the nth stroke is not the last value in the stack, the device 100 extracts the i+2th coordinate value of the nth stroke, and the i+2th coordinate value of the nth stroke is It can be determined whether it is a delimiter value. For example, when the second coordinate value of the first stroke is not the last value in the stack, the device 100 extracts the third coordinate value of the first stroke, and the third coordinate value of the first stroke is the delimiter value It can be judged whether

In step S3060 , when the i+1th coordinate value of the nth stroke is the delimiter value, the device 100 may increase the n value to n+1 and initialize the i value. For example, when the third coordinate value of the first stroke is the delimiter value, the device 100 may complete the output of the first stroke and sequentially extract the coordinate values of the second stroke from the first coordinate value .

Depending on the implementation, the order of steps S3005 to S3360 may be changed. For example, steps S3015 and S3020 may be performed before step S3010. Hereinafter, an operation in which the device 100 reproduces some handwritten content will be described in more detail with reference to FIGS. 31 to 34 .

As described above, strokes may be output in the same temporal manner as input, based on location information and timestamp information. Accordingly, the handwriting information may be output in the same temporal manner as the inputted handwriting information by combining a plurality of strokes. As a result, a virtual replay of the handwritten content may be output to the user on the device 100 .

31 is a diagram illustrating an example in which a device selects strokes within a region of interest.

31 , the device 100 may display handwritten content including a first stroke 3101 , a second stroke 3102 , a third stroke 3103 , and a fourth stroke 3104 . have.

The user may designate the region of interest 3100 to include the first stroke 3101 and the third stroke 3103 in the handwritten content.

32A is a diagram illustrating an example in which a device extracts information on strokes in an ROI.

Referring to FIG. 32A , the device 100 may extract time stamp information 3210 of strokes included in the ROI 3100 . For example, the device 100 may extract time stamp information (eg, 002525) of the first stroke 3101 and time stamp information (eg, 003900) of the third stroke 3103.

Also, the device 100 may extract coordinate information of strokes (eg, a coordinate value 3220 and time stamp information 3230 corresponding to the coordinate value). For example, the device 100 may extract coordinate information 3201 of the first stroke 3101 and coordinate information 3203 of the third stroke 3103 .

32B is a diagram illustrating an example in which a device generates a stack by using information on strokes in an ROI.

Referring to FIG. 32B , the device 100 may first compare time stamp information (eg, 002525) of the first stroke 3101 with time stamp information (eg, 003900) of the third stroke 3103. Since the timestamp information (eg, 002525) of the first stroke 3101 is earlier than the timestamp information (eg, 003900) of the third stroke 3103, the device 100, coordinate information of the first stroke 3101 The stack 3200 may be generated such that the 3201 is located on the upper layer than the coordinate information 3203 of the third stroke 3103 .

At this time, the device 100 determines that a value obtained by adding 1 to a time stamp value (eg, 001304) corresponding to the last coordinate value (eg, x30, y30) of the first stroke 3101 is the third stroke 3103 The time stamp information 3230 corresponding to the coordinate value of the third stroke 3103 may be entirely adjusted to be a time stamp value (eg, 001305) corresponding to one coordinate value (eg, x0, y0).

Meanwhile, the device 100 may insert a delimiter value 3240 between the coordinate information 3201 of the first stroke 3101 and the coordinate information 3203 of the third stroke 3103 . For example, the device 100 may use the maximum value (eg, 999999) of the time stamp as the delimiter value 3240 .

33A to 33D are diagrams illustrating an example in which a device reproduces some handwritten content using information on strokes included in a stack.

Referring to FIG. 33A , the device 100 may display handwritten content 3300 . In this case, the user may designate the region of interest 3100 to include the first stroke 3101 and the third stroke 3103 .

The device 100 may provide a virtual control panel 3310 in response to an input designating the region of interest 3100 .

Referring to FIG. 33B , the device 100 may receive an input for selecting the play button 3320 on the virtual control panel 3310 . In this case, the device 100 may generate the stack 3200 illustrated in FIG. 32B by extracting the information on the first stroke 3101 and the information on the third stroke 3103 .

The device 100 may erase the first stroke 3101 and the third stroke 3103 displayed on the region of interest 3100 based on an input of selecting the play button 3320 . In addition, the device 100 may regenerate the drawing process of the first stroke 3101 and the third stroke 3103 .

33C and 33D , the device 100 extracts a first coordinate value (eg, x0, y0) of the first stroke 3101 from the stack 3200, and the first coordinate value (eg, x0) , y0) can output a point. And the device 100 extracts the second coordinate value (eg, x1, y1) of the first stroke 3101, and the first coordinate value (eg, x0, y0) and the second coordinate value (eg, x1, The line connecting y1) can be output. Also, the device 100 may extract a third coordinate value (eg, x2, y2) to a 31st coordinate value (eg, x30, y30) of the first stroke 3101 , respectively. The device 100 connects the extracted third coordinate values (eg, x2, y2) to the 31st coordinate values (eg, x30, y30) with the previous coordinate values of the extracted first stroke 3101 to output a line. , the drawing process of the first stroke 3101 may be reproduced.

The device 100 may extract a coordinate value (eg, 0,0) following the 31st coordinate value (eg, x30, y30) of the first stroke 3101 . Since the time stamp corresponding to the coordinate value (eg, 0,0) following the 31st coordinate value (eg, x30, y30) of the first stroke 3101 is '999999', the device 100 3101), it can be identified that a coordinate value (eg, 0,0) following the 31st coordinate value (eg, x30, y30) is a delimiter value. In this case, the device 100 may confirm that reproduction of the drawing process of the first stroke 3101 is completed, and may continuously output the drawing process of the third stroke 3103 .

The device 100 extracts a first coordinate value (eg, x0, y0) of the third stroke 3103, and a position corresponding to the first coordinate value (eg, x0, y0) of the third stroke 3103 You can print a new point in . In addition, the device 100 may extract second coordinate values (eg, x1 , y1 ) to thirteenth coordinate values (eg, x12 , y12 ) of the third stroke 3103 , respectively. The device 100 connects the second coordinate values (eg, x1, y1) to the thirteenth coordinate values (eg, x12, y12) of the extracted third stroke 3103 with the previous coordinate values to output a line, The drawing process of the third stroke 3103 may be reproduced.

According to an embodiment of the present invention, the device 100 uses the time stamp information 3230 corresponding to each of the coordinate values, and uses some handwritten content (eg, the first stroke 3101 and the third stroke 3103 ). ) can be adjusted to correspond to the user's actual drawing speed.

According to an embodiment of the present invention, the device 100 allows the user to reproduce only the drawing process of some of the handwritten content of interest among the handwritten content in which the drawing process is recorded.

As described above, the device 100 may reproduce the handwritten content by drawing lines between the coordinates. Here, the device 100 may interpolate the positions of unrecorded coordinates from the positions of the coordinates. Accordingly, the reproduction speed of the handwritten content can be reduced.

In addition, device 100 may interpolate positions of coordinates to reduce memory storage requirements. In this case, the rate of capturing coordinates may be reduced.

34 is a diagram illustrating an example in which a device reproduces some handwritten content in a separate window using information on strokes included in a stack.

As shown in FIG. 34 , according to an embodiment of the present invention, the device 100 includes strokes 3401 included in the ROI 3410 , based on a user input designating the ROI 3410 , 3402 ) may be displayed in a separate pop-up window 3420 . The pop-up window 3420 may include at least one of a playback function button, an editing function button, and a search bar.

According to an embodiment of the present invention, when the user selects the play button 3421 on the pop-up window 3420 using the electronic pen 200 , the device 100 displays the strokes displayed on the pop-up window 3420 . The drawing process 3401 and 3402 may be reproduced in the pop-up window 3420 .

Meanwhile, in FIGS. 30 to 34 , the device 100 uses a stack to reproduce a drawing process of some handwritten content as an example, but the present invention is not limited thereto. For example, the device 100 may generate a table by collecting link information for accessing coordinate information of strokes. In addition, the device 100 may reproduce a drawing process of some handwritten content using a table including link information.

Hereinafter, an operation of editing some handwritten content based on a user input to the device 100 will be described with reference to FIG. 35 .

35 is a flowchart illustrating a method for a device to edit some handwritten content according to an embodiment of the present invention.

In operation S3510, the device 100 may display the handwritten content.

According to an embodiment of the present invention, the handwritten content may include a plurality of strokes drawn in a predetermined order. Each of the plurality of strokes included in the handwritten content may have various shapes. For example, the thickness, color, brightness, saturation, transparency, etc. of each of the plurality of strokes may be different.

According to an embodiment of the present invention, the device 100 may display handwritten content on an execution window of a predetermined application. According to an embodiment of the present invention, the device 100 may receive a selection of the specific handwritten content from the user and display the selected specific handwritten content.

In operation S3520, the device 100 may receive a user input requesting editing of some of the displayed handwritten content.

According to an embodiment of the present invention, a user input for requesting editing of some handwritten content may be various.

For example, the device 100 may receive a user input for selecting some handwritten content from among handwritten content. In this case, the device 100 may display a virtual control panel including an edit function button (eg, an undo button, a redo button, a character recognition button, an add button, an exchange button). . The device 100 may receive a user input for selecting an edit function button through the control panel.

Also, the device 100 may receive a preset gesture corresponding to the edit command. For example, when the user double-taps the selected handwritten content with two fingers after selecting some handwritten content by a drag input, the device 100 determines that an undo request for some handwritten content has been received can do.

Meanwhile, the user may input an editing command for some handwritten content to the device 100 by using a touch input and a voice input. For example, the user may select some handwritten content through a touch input and input a command for requesting editing of some handwritten content into the device 100 by voice.

In operation S3530, the device 100 may edit the partial handwritten content by using time information of strokes included in the partial handwritten content.

According to an embodiment of the present invention, the time information of the strokes may include at least one of relative time order information between the strokes, and time stamp information corresponding to each of the strokes. Meanwhile, the time stamp information may be absolute time information (eg, year, month, day, hour, minute, second, 2014.05.18.08.53.36) of the device 100 when generating a stroke, or an arbitrarily assigned number ( For example, 0001, 0002, 0003, etc.).

According to an embodiment of the present invention, the device 100 may extract strokes included in some handwritten content. The device 100 may determine an order between the extracted strokes by using time stamp information of each of the extracted strokes. For example, the device 100 determines a first stroke input first among the extracted strokes as a start position, and an n-th stroke input last among the extracted strokes (n is 1) larger integer) can be determined as the end position. The device 100 may edit the extracted strokes by using time stamp information and coordinate information of the extracted strokes. Hereinafter, an operation in which the device 100 edits some handwritten content will be described in more detail with reference to FIG. 36 .

36 is a diagram for describing a method of a device performing an undo or redo function for some handwritten content, according to an embodiment of the present invention.

In operation S3610, the device 100 may receive an input for designating an ROI with respect to the handwritten content displayed on the screen. Since the operation of the device 100 to designate the region of interest based on the user input has been described in detail above, it will be omitted herein. According to an embodiment of the present invention, the device 100 may select some handwritten content corresponding to the ROI.

In operation S3620, the device 100 may extract strokes included in some handwritten content corresponding to the ROI.

For example, the device 100 may retrieve information on strokes included in some handwritten content (eg, stroke identification information, stroke time stamp information, and stroke coordinate information) from the memory.

In operation S3630 , the device 100 may generate a stack in which strokes are inserted in chronological order.

For example, the device 100 may generate a first stack for storing information about strokes in the ROI and a second stack for storing information about deleted strokes. Hereinafter, for convenience of description, a stack for storing information about strokes included in some handwritten content is defined as a first stack, and a stack for storing information about deleted strokes is defined as a second stack. .

The device 100 may determine the relative time order of the strokes by comparing the time stamp information between the extracted strokes. The device 100 may insert a stroke generated the longest among the extracted strokes at the bottom of the first stack, and may insert a stroke that is recently generated as it moves upward in the first stack.

In operation S3640 , the device 100 may receive an undo input for some handwritten content. For example, the device 100 may receive a user input for selecting an undo button displayed on the control panel.

Also, the device 100 may receive a preset gesture corresponding to an undo command. For example, when the user double-taps some handwritten content with two fingers, the device 100 may determine that an undo request for some handwritten content has been received. Meanwhile, the device 100 may recognize a voice including an undo command for some handwritten content.

In operation S3650 , the device 100 may delete the last inserted stroke from the stack according to the undo input. For example, the device 100 may delete information on the first stroke located in the uppermost layer of the first stack from the first stack. In addition, the device 100 may insert information of the first stroke into the second stack. An operation in which the device 100 performs an undo function on some handwritten content will be described in more detail later with reference to FIGS. 37 to 39B .

In operation S3660, the device 100 may receive a redo input for some handwritten content. For example, the device 100 may receive a user input for selecting a redo button displayed on the virtual control panel.

Also, the device 100 may receive a preset gesture corresponding to the re-execution command. For example, when the user touches some handwritten content with two fingers for a predetermined time or longer (eg, for 2 seconds or longer), the device 100 may determine that a redo request for some handwritten content has been received. have. Meanwhile, the device 100 may recognize a voice including a re-execution command for some handwritten content.

In operation S3670 , the device 100 may re-insert the last deleted stroke into the stack based on the redo input.

For example, the device 100 may delete information on the first stroke located in the uppermost layer of the second stack from the second stack according to the re-execution input. In addition, the device 100 may insert the information of the first stroke into the uppermost layer of the first stack.

An operation of the device 100 to perform a re-execution function for some handwritten content will be described later in more detail with reference to FIGS. 40A to 40C .

In operation S3680, the device 100 may receive an input for canceling the designation of the ROI. For example, when editing of some handwritten content is completed, the user may release the designation of the ROI.

In operation S3690 , the device 100 may update information on a plurality of strokes included in the handwritten content based on the stack information at the time of canceling the designation of the region of interest. For example, the device 100 may modify the information list of strokes included in the pre-stored handwriting content based on the edited content (eg, information on deleted strokes and information about added strokes).

37 is a diagram illustrating a list of a plurality of strokes included in handwritten content according to an embodiment of the present invention.

37 , according to an embodiment of the present invention, the device 100 may store a list 3720 of a plurality of strokes included in the handwritten content 3710 when the handwritten content 3710 is generated. . In the list 3720 of the plurality of strokes, stroke identification information, time stamp information, and coordinate information may be mapped and stored.

The handwritten content 3710 may include a first stroke 3701 , a second stroke 3702 , a third stroke 3703 , a fourth stroke 3704 , and a fifth stroke 3705 .

38 is a diagram illustrating an example of a first stack including information on strokes in a region of interest and a second stack including information on deleted strokes, according to an embodiment of the present invention.

Referring to 3800-1 of FIG. 38 , the user may designate a region of interest 3800 to include a first stroke 3701 , a third stroke 3703 , and a fifth stroke 3705 among the handwritten content 3710 . .

In the list 3720 of a plurality of strokes included in the handwritten content 3710 , the device 100 determines whether the first stroke 3701 , the third stroke 3703 , and the fifth stroke 3705) can be extracted. The device 100 may generate a first stack 3800 - 2 including information regarding the extracted first stroke 3701 , the third stroke 3703 , and the fifth stroke 3705 . For example, the device 100 compares time stamp information between the extracted first stroke 3701 , the third stroke 3703 , and the fifth stroke 3705 , and the first stroke 3710 drawn first ) may be inserted into the bottom of the first stack 3800 - 2 , and a third stroke 3703 and a fifth stroke 3705 may be sequentially inserted thereon.

Meanwhile, according to an embodiment of the present invention, the device 100 may create a second stack 3800 - 3 for storing information about the deleted stroke.

According to an embodiment of the present invention, the device 100 may provide a virtual control panel 3810 including at least one edit function button.

A process in which the device 100 performs an undo function based on an input of selecting the undo button 3820 in the virtual control panel 3810 will be described in detail with reference to FIGS. 39A and 39B . .

39A and 39B are diagrams illustrating a first stack and a second stack that change according to a user's undo input. In FIGS. 39A and 39B , information on the first stroke (a), information on the third stroke (c), and information on the fifth stroke (e) in the first stack 3800 - 2 according to the designation of the region of interest 3800 are shown in FIGS. 39A and 39B . A case in which information is included will be described as an example.

In operation S3901 , the device 100 may receive an undo request for some handwritten content in the region of interest 3800 .

In step S3902 , the device 100 may delete the information of the fifth stroke e inserted in the upper layer of the first stack 3800 - 2 from the first stack 3800 - 2 in response to the undo request. have. In addition, the device 100 may insert information of the fifth stroke e into the second stack 3800 - 3 .

In operation S3903 , the device 100 may receive an input for canceling the designation of the region of interest 3800 .

In operation S3904 , the device 100 may update the list 3720 of a plurality of strokes included in the handwritten content 3710 according to an input for canceling the designation of the region of interest 3800 . For example, the device 100 may reflect the information of the first stack 3800 - 2 in the list 3720 of a plurality of strokes included in the handwritten content 3710 .

In operation S3905 , the device 100 may delete information on the fifth stroke e included in the second stack 3800 - 3 from the list 3720 of a plurality of strokes included in the handwritten content 3710 .

According to an embodiment of the present invention, the device 100 may delete a relatively recently drawn stroke from among a plurality of strokes included in the partial handwritten content, based on the undo request for the partial handwritten content.

40A, 40B, and 40C are diagrams for explaining an example in which a device performs an undo function and a redo function for some handwritten content. 40A, 40B, and 40C, according to the designation of the region of interest 3800, information on the first stroke (a), information on the third stroke (c), and the fifth stroke ( The case in which the information of e) is included will be described as an example.

In operation S4001 , the device 100 may receive the undo request for some handwritten content in the region of interest 3800 twice.

In step S4002 , in response to the first undo request, the device 100 transfers information of the fifth stroke e inserted into the upper layer of the first stack 3800 - 2 from the first stack 3800 - 2 . deleted, and the information of the fifth stroke (e) may be inserted into the second stack 3800 - 3 .

In response to the second undo request, the device 100 may delete information on the third stroke c inserted into the upper layer of the first stack 3800 - 2 from the first stack 3800 - 2 . In addition, the device 100 may insert information on the third stroke (c) onto the information on the fifth stroke (e) of the second stack 3800 - 3 .

In step S4003, the device 100 may receive a re-execution request.

In step S4004 , the device 100 deletes the information of the third stroke c inserted into the upper layer of the second stack 3800-3 from the second stack 3800-3 in response to the re-execution request, Information of the third stroke c may be inserted into the upper layer of the first stack 3800 - 2 .

In operation S4005 , the device 100 may receive an input for canceling the designation of the region of interest 3800 .

In operation S4006 , the device 100 may update the list 3720 of a plurality of strokes included in the handwritten content 3710 according to an input for canceling the designation of the region of interest 3800 . For example, the device 100 may reflect the information of the first stack 3800 - 2 in the list 3720 of a plurality of strokes included in the handwritten content 3710 .

In operation S4007 , the device 100 may delete information on the fifth stroke e included in the second stack 3800 - 3 from the list 3720 of a plurality of strokes included in the handwriting content 3710 .

41A and 41B are diagrams for explaining an example in which a device performs an undo function and a stroke addition function for some handwritten content.

In FIGS. 41A and 41B , information on the first stroke (a), information on the third stroke (c), and the information on the fifth stroke (e) in the first stack 3800 - 2 according to the designation of the region of interest 3800 are shown in FIGS. 41A and 41B . A case in which information is included will be described as an example.

In operation S4101 , the device 100 may receive the undo request for some handwritten content in the region of interest 3800 twice.

In step S4102 , the device 100 deletes information of the fifth stroke e inserted in the upper layer of the first stack 3800 - 2 from the first stack 3800 - 2 according to the first undo request and inserting the information of the fifth stroke (e) into the second stack 3800 - 3 .

The device 100 may delete information on the third stroke c inserted into the upper layer of the first stack 3800 - 2 from the first stack 3800 - 2 according to the second undo request. In addition, the device 100 may insert information on the third stroke (c) onto the information on the fifth stroke (e) of the second stack 3800 - 3 .

In step S4103, the device 100 may receive an input for adding the sixth stroke (f).

In operation S4104 , the device 100 may insert information on the sixth stroke f into the upper layer of the first stack 3800 - 2 according to an input for adding the sixth stroke f.

In operation S4105 , the device 100 may receive an input for releasing the designation of the region of interest 3800 .

In operation S4106 , the device 100 may update the list 3720 of a plurality of strokes included in the handwritten content 3710 according to an input for canceling the designation of the region of interest 3800 . For example, the device 100 may reflect the information of the first stack 3800 - 2 in the list 3720 of a plurality of strokes included in the handwritten content 3710 .

In step S4007 , the device 100 compares the information on the fifth stroke (e) and the information on the third stroke (c) included in the second stack 3800 - 3 of the plurality of strokes included in the handwriting content 3710 . It can be deleted from the list 3720 .

42 is a diagram for explaining a method for a device to replace some handwritten content with external handwritten content, according to an embodiment of the present invention.

In operation S4210, the device 100 may display the handwritten content.

In operation S4220, the device 100 may receive a user input requesting deletion of some handwritten content. For example, the device 100 may display a virtual control panel including an edit function button. The device 100 may receive a user input for selecting a delete button through the control panel.

In operation S4230, the device 100 may receive an input for adding external handwritten content. For example, the device 100 may receive an input for drawing a new stroke in an area where some handwritten content has been deleted.

In operation S4240, the device 100 may add a new stroke to an area where some handwritten content has been deleted.

According to an embodiment of the present invention, the device 100 may add information on a new stroke to a timeline corresponding to some handwritten content deleted from among the list 3720 of the plurality of strokes. According to another embodiment of the present invention, the device 100 may add new stroke information to the end of the list 3720 of the plurality of strokes.

According to an embodiment of the present invention, some handwritten content may be replaced with external handwritten content according to a user's input. It will be described in more detail with reference to FIGS. 43A to 43D.

43A, 43B, 43C, and 43D are diagrams for explaining an example in which a device replaces some handwritten content with external handwritten content. 43A to 43D, a case in which the handwritten content 4310 is synchronized with the moving picture content 4320 will be described as an example.

As shown in FIG. 43A , the device 100 may display the handwritten content 4310 in which the cake recipe is recorded on the screen. The user may designate a part (eg, blue) to be corrected in the cake recipe as the region of interest 4301 . The device 100 may display a virtual control panel 4302 in response to an input designating the region of interest 4301 .

According to an embodiment of the present invention, the device 100 may automatically provide a virtual control panel 4302 when the ROI 4301 is designated. According to another embodiment of the present invention, the device 100 may provide the control panel 4302 when a user input requesting the control panel 4302 is received.

As shown in FIG. 43B , the device 100 may receive a user input for selecting a delete button 4303 on the control panel 4302 .

The device 100 may delete strokes (eg, blue) included in the ROI 4301 based on a user input of selecting the delete button 4303 .

43C and 43D , the device 100 may receive an input of new strokes. For example, the user may newly input 'cran' in an area where 'blue' has been deleted.

According to an embodiment of the present invention, the device 100 may update the handwritten content by reflecting information on newly added strokes (eg, cran). For example, according to an embodiment of the present invention, the device 100 may add information on strokes corresponding to 'cran' to the timeline between sugar and berries.

According to another embodiment of the present invention, the device 100 adds a tag indicating that the strokes corresponding to 'blue' have been deleted, and 'cran' is added to the end of the list of a plurality of strokes included in the handwritten content. You may. At this time, the device 100 may indicate that 'blue' is replaced with 'cran' by adding link information of cran to information on strokes corresponding to 'blue'.

Meanwhile, according to an embodiment of the present invention, when some handwritten content is changed (eg, blue is changed to cran), the device 100 resynchronizes the moving picture content 4320 and the handwritten content 4310 can do.

44 is a flowchart illustrating a method for a device to reproduce the entire handwritten content by reflecting editing information for some handwritten content, according to an embodiment of the present invention.

In operation S4410, the device 100 may display the handwritten content. In operation S4420, the device 100 may receive a user input requesting editing of some handwritten content. In operation S4430, the device 100 may edit the partial handwritten content by using time information of strokes included in the partial handwritten content. Since steps S4410 to S4430 correspond to steps S3510 to S3530 of FIG. 35 , a detailed description thereof will be omitted.

In operation S4440, the device 100 may store editing information for some handwritten content. For example, the device 100 may store the edited handwriting content as one file by applying the editing information to the handwritten content. Also, the device 100 may store edit information separately from the handwritten content.

Meanwhile, according to an embodiment of the present invention, the device 100 may resynchronize the audio/video content with the handwritten content in which the editing information for some handwritten content is reflected.

In operation S4450, the device 100 may receive a reproduction request for the edited handwritten content. According to an embodiment of the present invention, the device 100 may display a virtual control panel including a play function button. The device 100 may receive a user input for selecting a play function button through a virtual control panel.

In addition, the device 100 may receive a preset touch input corresponding to the play command or may receive an input for selecting a physical button corresponding to the play command. Meanwhile, the user may input a command for requesting reproduction of the handwritten content to the device 100 by voice.

In operation S4460, the device 100 may reproduce the handwritten content by reflecting editing information for some handwritten content. For example, the device 100 may reproduce the drawing process of the handwritten content in a state in which the editing information is reflected.

According to an embodiment of the present invention, the device 100 may additionally reproduce the editing process after playing the drawing process of the original handwritten content. Also, the device 100 may provide a selection window for selecting whether to reflect edit information for some handwritten content. Each case will be described in detail with reference to FIGS. 45A, 45B, and 45C.

45A, 45B, and 45C are diagrams for explaining an example in which a device reproduces the entire handwritten content by reflecting editing information for some handwritten content.

Referring to FIG. 45A , the device 100 may receive a user input requesting reproduction of the handwritten content. For example, the device 100 may receive an input for selecting the play button 4510 from the tool bar.

Referring to FIG. 45B , the device 100 may reproduce the drawing process of the handwritten content by reflecting the editing information. For example, when 'blue'berries are changed to 'cran'berries, the device 100 performs a drawing process of a stroke corresponding to 'cran' at a time when a drawing process of a stroke corresponding to 'blue' is to be reproduced. can play

Referring to FIG. 45C , when the 'blue'berries are changed to 'cran'berries, the device 100 temporarily stops the reproduction of the handwritten content at a time when the drawing process of the stroke corresponding to 'blue' needs to be reproduced, and , a selection window 4530 for selecting whether to reflect the edit information may be provided.

When the user selects 'before correction', the device 100 may output a drawing process of a stroke corresponding to 'blue'. When the user selects 'after correction', the device 100 may output a drawing process of a stroke corresponding to 'cran'.

46 is a flowchart illustrating a method for a device to perform character recognition for some handwritten content, according to an embodiment of the present invention.

In operation S4610, the device 100 may display the handwritten content.

In operation S4620, the device 100 may receive a user input requesting character recognition for the displayed handwritten content. OCR (Optical Character Recognition) refers to a technology that converts Korean, English, and numeric fonts included in image documents into editable text.

According to an embodiment of the present invention, the device 100 may display a virtual control panel including a text recognition function button. The device 100 may receive a user input for selecting a character recognition function button through a virtual control panel.

Also, the device 100 may receive a preset touch input corresponding to a text recognition command, or may receive an input for selecting a physical button corresponding to a text recognition command. Meanwhile, the user may input a command for requesting character recognition for some handwritten content into the device 100 by voice.

In operation S4630, the device 100 may perform character recognition for some handwritten content. For example, the device 100 may perform character recognition for a plurality of strokes included in some handwritten content selected by the user.

In operation S4640, the device 100 may generate an index for the handwritten content by using the character recognition result. According to an embodiment of the present invention, the device 100 may store the index as a name for the handwritten content. According to an embodiment of the present invention, the user may use the index as a search term for searching for handwritten content.

47A and 47B are diagrams for explaining an example in which a device performs character recognition for some handwritten content.

Referring to FIG. 47A , the device 100 may display the handwritten content 4700 including the Christmas card on the screen. The user may designate a region of interest 4710 from among the handwritten content 4700 .

The device 100 may display the virtual control panel 4720 in response to an input designating the region of interest 4710 .

According to an embodiment of the present invention, the device 100 may automatically provide a virtual control panel 4720 when the region of interest 4710 is designated. According to another embodiment of the present invention, the device 100 may provide the control panel 4720 when a user input requesting the control panel 4720 is received.

According to an embodiment of the present invention, the device 100 may receive an input for selecting the character recognition function button 4730 from the control panel 4720 .

Referring to FIG. 47B , the device 100 displays a text for some handwritten content (eg, 'To. Yuna') included in the region of interest 4710 based on a user input for selecting the text recognition function button 4730 . It can perform a recognition function.

According to an embodiment of the present invention, the device 100 may display the character recognition result 4740 on the control panel 4720 . According to an embodiment of the present invention, the device 100 may store the character recognition result 4740 as an index for the handwritten content 4700 according to a user request.

48 is a flowchart illustrating a method for a device to add a background image, an icon, or background music by using emotional information corresponding to some handwritten content, according to an embodiment of the present invention.

In operation S4810, the device 100 may display the handwritten content. According to an embodiment of the present invention, the device 100 may receive a user input for selecting some handwritten content from among the handwritten content.

In operation S4820, the device 100 may receive a user input requesting to add additional information related to some handwritten content. For example, the device 100 may receive an input requesting to add a background image, icon, background music, etc. related to some handwritten content.

In operation S4830, the device 100 may extract emotional information corresponding to some handwritten content according to a request for additional information. For example, the device 100 may extract the user's mood state information (eg, joy, sadness), biometric information (eg, pulse information, electrocardiogram information, temperature information), facial expression information, and the like.

In operation S4840, the device 100 may add at least one of a background image, an icon, and background music by using emotion information corresponding to some handwritten content. For example, the device 100 may add an icon corresponding to the user's facial expression when generating some handwritten content.

49 is a diagram illustrating an example in which a device displays pulse information corresponding to some handwritten content.

Referring to FIG. 49 , the device 100 may display handwritten content 4900 including a Christmas card on the screen. The user may designate a region of interest 4910 among the handwritten content 4900 .

The device 100 may display a virtual control panel 4920 in response to an input designating the region of interest 4910 .

According to an embodiment of the present invention, the device 100 may automatically provide a virtual control panel 4920 when the region of interest 4910 is designated. According to another embodiment of the present invention, the device 100 may provide the control panel 4920 when a user input requesting the control panel 4920 is received.

According to an embodiment of the present invention, the device 100 may receive an input for selecting the emotion information output button 4930 from the control panel 4920 . The device 100 may extract pulse rate information corresponding to some handwritten content (eg, 'To. Yuna') included in the region of interest 4910 based on a user input for selecting the emotion information output button 4930 . have.

According to an embodiment of the present invention, the device 100 may display an image 4940 corresponding to the pulse rate information on the control panel 4920 . According to an embodiment of the present invention, the device 100 may add an image 4940 corresponding to the pulse rate information to the handwritten content 4900 according to a user request.

Meanwhile, although not shown in FIG. 49 , according to an embodiment of the present invention, the device 100 writes a heartbeat sound based on heart rate information corresponding to some handwritten content (eg, 'To. Yuna'). It may be inserted into the content 4900 .

50 is a flowchart illustrating a method for a device to reproduce a plurality of partial handwritten content according to an embodiment of the present invention.

In operation S5010, the device 100 may display the handwritten content on the screen. Since step S5010 corresponds to step S210 of FIG. 2 , a detailed description thereof will be omitted.

In operation S5020, the device 100 may receive a user input for selecting the first partial handwritten content and the second handwritten content from among the handwritten content displayed on the screen. For example, the device 100 may receive a user input for designating a plurality of ROIs. In this case, the device 100 may extract the first partial handwritten content and the second partial handwritten content corresponding to each of the plurality of ROIs.

In operation S5030, the device 100 may display a control panel. For example, the device 100 may display a control panel in response to a user input designating a plurality of regions of interest. In this case, the control panel may include a play button.

In operation S5040, the device 100 may receive an input for selecting a play button through the control panel.

In operation S5050, the device 100 may reproduce the first partial handwritten content and the second partial handwritten content based on an input of selecting the play button.

According to an embodiment of the present invention, the device 100 uses first time information of first strokes included in the first partial handwritten content and second time information of second strokes included in the second partial handwritten content. Accordingly, the first partial handwritten content and the second partial handwritten content may be reproduced.

For example, the device 100 may extract first strokes included in the first partial handwritten content and second strokes included in the second partial handwritten content. The device 100 may determine the relative time order of the strokes by comparing the time stamp information between the extracted strokes. The device 100 may generate the stack such that the oldest stroke among the extracted strokes is inserted at the top of the stack, and the most recently generated stroke is inserted as it goes down in the stack.

The device 100 may reproduce the first partial handwritten content and the second partial handwritten content included in the plurality of ROIs according to the drawing time order by using the coordinate values of the strokes included in the stack.

51A is a diagram for explaining an example in which a device reproduces a plurality of partial handwritten content.

Referring to FIG. 51A , the device 100 may display the sheet music handwriting content 5100 . The user may designate a first region of interest 5110 and a second region of interest 5120 from among the sheet music handwriting content 5100 .

The device 100 may provide a virtual control panel 5130 . The device 100 may receive an input for selecting the play button 5131 on the virtual control panel 5130 . In this case, the device 100 may extract stroke information of first notes included in the first region of interest 5110 and stroke information of second notes included in the second region of interest 5120 . The stroke information may include stroke identification information, time stamp information, and coordinate information.

The device 100 may erase the first notes and the second notes displayed in the first region of interest 5110 and the second region of interest 5120 based on an input of selecting the play button 5131 . In addition, the device 100 may regenerate the drawing process of the first notes and the second notes.

51B and 51C are diagrams for explaining an example in which a device exchanges positions of a plurality of partial handwritten contents.

Referring to FIG. 51B , the device 100 may receive an input for selecting the exchange button 5132 on the virtual control panel 5130 . In this case, the device 100 may extract stroke information of first notes included in the first region of interest 5110 and stroke information of second notes included in the second region of interest 5120 .

Referring to FIG. 51C , the device 100 may mutually change time stamp information and coordinate information between the first notes and the second notes based on an input of selecting the exchange button 5132 .

52 is a flowchart illustrating a method for a device to reproduce music by using information on strokes included in some handwritten content, according to an embodiment of the present invention.

In operation S5210, the device 100 may display handwritten content including a plurality of notes.

In operation S5220, the device 100 may receive an input for designating a region of interest from among the handwritten content.

In operation S5230, the device 100 may extract information on notes in the ROI. For example, the device 100 may extract coordinate information, time stamp information, and the like of notes in the ROI. In addition, the device 100 compares the coordinate values of the staff with the coordinate values of the notes or through image processing for the region of interest, the note length (eg, quarter note, 16th note, etc.), the height of the note (eg, Mi, sole, re), time signature, comma information, musical symbol information (eg, flat, up, place, double??, adagio, allegro) and the like may be extracted.

In operation S5240, the device 100 may reproduce music using information on notes. According to an embodiment of the present invention, the device 100 may provide a graphical user interface (GUI) for selecting a type of musical instrument. The types of musical instruments may be varied. For example, the musical instrument may include, but is not limited to, a piano, a guitar, a drum, a violin, a cello, a flute, and the like.

According to an embodiment of the present invention, when the user selects a type of instrument through the GUI, the device 100 may reproduce music with the instrument selected by the user using information on notes included in the region of interest. have.

53A, 53B, and 53C are diagrams for explaining an example in which a device reproduces music by using information on strokes included in some handwritten content.

Referring to FIG. 53A , the device 100 may display sheet music handwriting content 5300 . The user may designate a region of interest 5310 from among the sheet music handwriting content 5300 . In this case, the device 100 may provide a virtual control panel 5320 . The device 100 may receive an input for selecting the music button 5330 on the virtual control panel 5130 . In this case, the device 100 may extract information on notes 5311 included in the region of interest 5310 . For example, the device 100 may include time stamp information, coordinate information, note length (eg, quarter note, 16th note, etc.) of notes 5311 included in the region of interest 5310, note height ( For example, it is possible to extract information such as mi, sol, and re), time signature, comma information, and musical symbol information (eg, flattened, raised, fixed, double??, adagio, allegro).

Referring to FIG. 53B , the device 100 may display the type of musical instrument through the virtual control panel 5130 . In addition, the device 100 may receive a selection of one instrument through the virtual control panel 5130 . For example, the device 100 may receive an input for selecting the piano icon 5331 .

Referring to FIG. 53C , the device 100 may reproduce the piano performance music 5340 using information on notes 5311 included in the region of interest 5310 .

54 is a flowchart for explaining a method of extracting, by a device, video content or audio content corresponding to some handwritten content, according to an embodiment of the present invention.

In operation S5410, the device 100 may display the handwritten content on the screen.

According to an embodiment of the present invention, the handwritten content may include a plurality of strokes input in a predetermined order. Each of the plurality of strokes included in the handwritten content may have various shapes. For example, the thickness, color, brightness, saturation, transparency, etc. of each of the plurality of strokes may be different.

According to an embodiment of the present invention, the device 100 may display handwritten content on an execution window of a predetermined application. For example, the device 100 may display the handwritten content on the execution window of the video playback application.

According to an embodiment of the present invention, the device 100 may receive a selection of the specific handwritten content from the user and display the selected specific handwritten content. According to an embodiment of the present invention, the device 100 may receive a selection of specific video content from the user and display handwritten content corresponding to the specific video content.

In operation S5420, the device 100 may receive an input for selecting some handwritten content from among the handwritten content. For example, the device 100 may receive a user input designating a region of interest and select some handwritten content corresponding to the region of interest. Also, the device 100 may receive a user input of sequentially touching a plurality of strokes while pressing a predetermined button of the electronic pen 200 . In this case, the device 100 may select a plurality of strokes touched by the electronic pen 200 as some handwritten content.

In operation S5430, the device 100 may extract video content or audio content corresponding to some handwritten content. For example, the device 100 may select some sections (some frames) of a video or some sections of audio positioned on a timeline corresponding to a timeline of some handwritten content.

In step S5440, the device 100 may reproduce the extracted video content or audio content. Depending on the implementation, some steps may be omitted from steps S5410 to S5440. Hereinafter, an operation in which the device extracts video content corresponding to some handwritten content will be described in more detail with reference to FIGS. 55A, 55B, and 55C.

55A, 55B, and 55C are diagrams for explaining an example in which a device extracts video content corresponding to some handwritten content. A case in which the device 100 generates the handwritten content 5520 while playing the online lecture content 5510 will be described as an example in FIGS. 55A, 55B and 55C . The device 100 may store a mapping table (refer to FIG. 11 ) in which information of the online lecture content 5510 and information of the handwritten content 5520 are mapped.

Referring to FIG. 55A , based on a user input, the device 100 may display a start screen of the online lecture content 5510 and the handwritten content 5520 generated during reproduction of the online lecture content 5510 on the screen.

Referring to FIG. 55B , the device 100 may receive an input for selecting some handwritten content from among handwritten content. For example, the device 100 may receive an input designating the region of interest 5530 .

Referring to FIG. 55C , the device 100 may select a partial video section 5540 corresponding to the partial handwritten content based on information on strokes included in the partial handwritten content.

For example, the device 100 may extract stroke information (identification information, time stamp information, and coordinate information) of an 'atom' input first among strokes included in the region of interest 5530 . The device 100 uses a mapping table in which the extracted stroke information and information of the online lecture content 5510 and information of the handwritten content 5520 are mapped, and the first stroke drawn first in the region of interest 5530 . A first frame (or a first timeline) corresponding to may be extracted.

Also, the device 100 may extract stroke information (identification information, time stamp information, and coordinate information) of the latest input 'ion' among strokes included in the region of interest 5530 . The device 100 uses a mapping table in which the extracted stroke information and information of the online lecture content 5510 and the information of the handwritten content 5520 are mapped, the nth drawn last in the region of interest 5530 . An nth frame (or an nth timeline) corresponding to the stroke may be extracted.

The device 100 may select a partial video section 5540 from the first frame (or the first timeline) to the nth frame (or the nth timeline) from the online lecture content 5510 . In addition, the device 100 may display or play the selected partial video section 5540 .

According to an embodiment of the present invention, a user can easily find a video section that he or she wants to view again by designating a region of interest that includes some handwritten content among the handwritten content.

56 is a flowchart illustrating a method of providing, by a device, a list of a plurality of video contents corresponding to a plurality of partial handwritten contents, according to an embodiment of the present invention.

In operation S5610, the device 100 may receive an input for selecting a plurality of partial handwritten content. According to an embodiment of the present invention, a plurality of partial handwritten content may be selected from one handwritten content or may be selected from different handwritten content.

For example, a plurality of partial handwritten content may be all selected from the first handwritten content synchronized with the first moving image content. In addition, some handwritten content may be selected from the first handwritten content synchronized with the first moving image content, and another partial handwritten content may be selected from the second handwritten content synchronized with the second moving image content.

In operation S5620, the device 100 may extract a plurality of video contents corresponding to each of the plurality of partial handwritten contents. According to an embodiment of the present invention, the moving picture content corresponding to the partial handwritten content may be a partial section corresponding to the partial handwritten content among the entire section of the moving picture content.

In operation S5630, the device 100 may provide a list of a plurality of video content corresponding to a plurality of partial handwritten content. In this case, the user can collect and view only some sections of the video of interest. Reference will be made to FIG. 57 .

57 is a diagram illustrating a list of a plurality of video contents corresponding to a plurality of partial handwritten contents according to an embodiment of the present invention.

57 , the device 100 may provide a list of a plurality of video contents corresponding to a plurality of partial handwritten contents selected by the user. For example, when the user selects the first partial handwritten content from the first handwritten content synchronized with the first moving image content, the device 100 may display a first video section corresponding to the first partial handwritten content from among the first moving image content. (5710) can be extracted. In this case, the device 100 may receive a storage request for the first video section 5710 , and store information on the first video section 5710 in the history management page 5700 .

Also, when the user selects the second partial handwritten content from the first handwritten content synchronized with the first moving image content, the device 100 performs a second video section 5720 corresponding to the second partial handwritten content from among the first moving image content. ) can be extracted. In this case, the device 100 may receive a storage request for the second video section 5720 , and store information on the second video section 5720 in the history management page 5700 .

When the user selects the third partial handwritten content from the second handwritten content synchronized with the second moving image content, the device 100 selects a third video section 5730 corresponding to the third partial handwritten content from among the second moving image content. can be extracted. In this case, the device 100 may receive a storage request for the third video section 5730 and store information on the third video section 5730 in the history management page 5700 .

According to an embodiment of the present invention, the device 100 may display a history management page according to a user request. At this time, a plurality of devices including information on the first video section 5710 , information on the second video section 5720 , and information on the third video section 5730 through the device 100 and the history management page A list of video content may be provided.

58 is a flowchart illustrating a method for a device to provide a plurality of handwritten content received from a plurality of external devices, according to an embodiment of the present invention.

In operation S5805 , the device 100 may perform a communication connection with the first external device 5810 . For example, the device 100 may form a short-range communication link with the first external device 5810 or a mobile communication link (eg, 3G, 4G, 5G, etc.). The short-range communication may include, but is not limited to, Bluetooth, Bluetooth Low Energy (BLE), Wi-Fi Direct, ultra wideband (UWB), Zigbee, Near Field Communication unit (NFC), Ant+, and the like.

In operation S5810 , the device 100 may perform a communication connection with the second external device 5820 . For example, the device 100 may form a short-range communication link with the second external device 5820 or may form a mobile communication link (eg, 3G, 4G, 5G, etc.).

In operation S5815, the first external device 5810 may generate the first handwritten content. The first external device 5810 may acquire time stamp information and coordinate information corresponding to each of a plurality of strokes input for a predetermined time. For example, the first external device 5810 may acquire time stamp information and coordinate information corresponding to each of a plurality of strokes by tracking coordinate values for each stroke drawn on the first external device 5810 . The first external device 5810 may generate the first handwritten content by using time stamp information and coordinate information corresponding to each of the plurality of strokes.

In operation S5820 , the first external device 5810 may transmit the first handwritten content to the device 100 . For example, the first external device 5810 may transmit time stamp information and coordinate information of strokes included in the first handwritten content to the device 100 . According to an embodiment of the present invention, the first external device 5810 may transmit the first handwritten content to the device 100 using short-range communication or mobile communication.

In operation S5825, the second external device 5820 may generate the second handwritten content. The second external device 5820 may acquire time stamp information and coordinate information corresponding to each of a plurality of strokes input for a predetermined time. For example, the second external device 5820 may acquire time stamp information and coordinate information corresponding to each of a plurality of strokes by tracking coordinate values for each stroke drawn on the second external device 5820 . The second external device 5820 may generate the second handwritten content by using time stamp information and coordinate information corresponding to each of the plurality of strokes.

In operation S5830 , the second external device 5820 may transmit the second handwritten content to the device 100 . For example, the second external device 5820 may transmit time stamp information and coordinate information of strokes included in the second handwritten content to the device 100 . According to an embodiment of the present invention, the second external device 5820 may transmit the second handwritten content to the device 100 using short-range communication or mobile communication.

In operation S5835 , the device 100 may display the first handwritten content received from the first external device 5810 and the second handwritten content received from the second external device 5820 on the screen. Each of the first handwritten content and the second handwritten content may include a plurality of strokes input in a predetermined order.

According to an embodiment of the present invention, the device 100 may receive a user input requesting reproduction of some handwritten content among the first handwritten content. In this case, the device 100 may reproduce the partial handwritten content by using time information of strokes included in the partial handwritten content.

Meanwhile, according to an embodiment of the present invention, the device 100 may receive a sharing request for the first handwritten content and the second handwritten content.

In operation S5840 , the device 100 may transmit the second handwritten content to the first external device 5810 according to the sharing request. For example, the device 100 may transmit time stamp information and coordinate information of strokes included in the second handwritten content to the first external device 5810 .

In operation S5845 , the device 100 may transmit the first handwritten content to the second external device 5820 according to the sharing request. For example, the device 100 may transmit time stamp information and coordinate information of strokes included in the first handwritten content to the second external device 5820 .

In operation S5850, the first external device 5810 may display the first handwritten content and the second handwritten content on the screen. In operation S5855, the second external device 5820 may also display the first handwritten content and the second handwritten content.

59 is a diagram illustrating an example in which a device provides a plurality of pieces of handwritten content received from a plurality of external devices. In FIG. 59 , a case in which the device 100 is a teacher terminal and the teacher gives students a task to input the Pythagorean theorem will be described as an example.

As shown in FIG. 59 , when the first student handwrites the Pythagorean theorem on the first device 5910, the first device 5910 records the first student's writing process, 5901) can be created. In addition, the first device 5910 may transmit the first handwritten content 5901 to the teacher terminal 100 .

When the second student handwrites the Pythagorean theorem into the second device 5920 , the second device 5920 may record the second student's writing process to generate second handwritten content 5902 . In addition, the second device 5920 may transmit the second handwritten content 5902 to the teacher terminal 100 .

When the third student handwrites the Pythagorean theorem into the third device 5930 , the third device 5930 may record the third student's writing process to generate third handwritten content 5903 . In addition, the third device 5910 may transmit the third handwritten content 5903 to the teacher terminal 100 .

When the fourth student writes the Pythagorean theorem on the fourth device 5940 by hand, the fourth device 5940 may record the fourth student's writing process to generate fourth handwritten content 5904 . In addition, the first device 5940 may transmit the fourth handwritten content 5904 to the teacher terminal 100 .

The teacher terminal 100 may display the first handwritten content 5901 , the second handwritten content 5902 , the third handwritten content 5903 , and the fourth handwritten content 5904 . In this case, the teacher terminal 100 may provide a control panel including a play button for playing each handwritten content. By selecting the play button on the control panel, the teacher can play the specific handwritten content of the students or a part of the specific handwritten content.

60A and 60B are diagrams for explaining an example in which a device transmits a plurality of handwritten contents to each of a plurality of external devices. In FIGS. 60A and 60B , a case in which the device 100 is a teacher terminal and students submit a task for inputting the Pythagorean theorem through the teacher terminal will be described as an example.

The teacher terminal 100 transmits the first device 5910 and the first handwritten content 5901, the second handwritten content 5902, the third handwritten content 5903, and the fourth handwritten content 5904 of the first student. can share For example, the teacher terminal 100 may transmit the second handwritten content 5902 , the third handwritten content 5903 , and the fourth handwritten content 5904 to the first device 5910 of the first student.

The teacher terminal 100 transmits the first handwritten content 5901, the second handwritten content 5902, the third handwritten content 5903, and the fourth handwritten content 5904 with the second device 5920 of the second student. can share For example, the teacher terminal 100 may transmit the first handwritten content 5901 , the third handwritten content 5903 , and the fourth handwritten content 5904 to the second device 5920 .

The teacher terminal 100 transmits the third device 5930 and the first handwritten content 5901, the second handwritten content 5902, the third handwritten content 5903, and the fourth handwritten content 5904 of the third student. can share For example, the teacher terminal 100 may transmit the first handwritten content 5901 , the second handwritten content 5902 , and the fourth handwritten content 5904 to the third device 5930 .

The teacher terminal 100 transmits the first handwritten content 5901, the second handwritten content 5902, the third handwritten content 5903, and the fourth handwritten content 5904 with the fourth device 5940 of the fourth student. can share For example, the teacher terminal 100 may transmit the first handwritten content 5901 , the second handwritten content 5902 , and the third handwritten content 5903 to the fourth device 5940 .

According to an embodiment of the present invention, each student's device (5910, 5920, 5930, 5490) may share the handwritten content (5901, 5902, 5903, 5904) with the teacher terminal 100 in a download manner. , the handwritten contents 5901 , 5902 , 5903 , 5904 may be shared with the teacher terminal 100 in a streaming manner.

According to an embodiment of the present invention, each of the students' devices (5910, 5920, 5930, 5490) uses a screen mirroring technology, and the teacher terminal 100 on which handwritten contents (5901, 5902, 5903, 5904) are displayed. ) can be shared.

Referring to FIG. 60B , a first device 5910 , a second device 5920 , a third device 5930 , and a fourth device 5940 are a first handwritten content 5901 and a second handwritten content 5902 , respectively. , the third handwritten content 5903 and the fourth handwritten content 5904 may be displayed.

According to an embodiment of the present invention, the first student may check the task performance results of the second student, the third student, and the fourth student using the first device 5910 .

Also, when the first student selects the play button, the first device 5910 plays at least one of the second handwritten content 5902 , the third handwritten content 5903 , and the fourth handwritten content 5904 . can For example, when the first student selects the play all button, the first device 5910 plays all of the second handwritten content 5902, the third handwritten content 5903, and the fourth handwritten content 5904 at the same time. can When the first student selects the play button for the second handwritten content 5902 , the first device 5910 may reproduce only the second handwritten content 5902 . When the first student designates a region of interest to include a part of the third handwritten content and selects the play button, the first device 5910 may reproduce only some handwritten content included in the region of interest among the third handwritten content. .

61 is a diagram for explaining a method for a first device to receive handwritten content edited by a second device, according to an embodiment of the present invention.

In step S6110 , the first device 100 - 1 may perform a communication connection with the second device 100 - 2 . For example, the first device 100-1 may form a short-range communication link with the second device 100-2, or may form a mobile communication link (eg, 3G, 4G, 5G, etc.) . The short-range communication may include, but is not limited to, Bluetooth, Bluetooth Low Energy (BLE), Wi-Fi Direct, ultra wideband (UWB), Zigbee, Near Field Communication unit (NFC), Ant+, and the like.

In operation S6120, the first device 100-1 may generate handwritten content.

The first device 100 - 1 may acquire time stamp information and coordinate information corresponding to each of a plurality of strokes input for a predetermined time. For example, the first device 100-1 may obtain time stamp information and coordinate information corresponding to each of a plurality of strokes by tracking coordinate values for each stroke drawn on the first device 100-1. have. The first device 100 - 1 may generate handwritten content by using time stamp information and coordinate information corresponding to each of the plurality of strokes.

In operation S6130 , the first device 100 - 1 may transmit the handwritten content to the second device 100 - 2 . For example, the first device 100 - 1 may transmit time stamp information and coordinate information of strokes included in the handwritten content to the second device 100 - 2 .

According to an embodiment of the present invention, the first device 100-1 may transmit the handwritten content to the second device 100-2 using short-distance communication or mobile communication.

In operation S6140 , the second device 100 - 2 may display the handwritten content received from the first device 100 - 1 . The handwritten content may include a plurality of strokes input in a predetermined order.

In operation S6150 , the second device 100 - 2 may receive an editing request for some handwritten content.

According to an embodiment of the present invention, a user input for requesting editing of some handwritten content may be varied. For example, the second device 100 - 2 may receive a user input for selecting some handwritten content from among handwritten content. In this case, the second device 100-2 operates a virtual control panel including an editing function button (eg, an undo button, a redo button, a character recognition button, an add button, an exchange button). can be displayed The second device 100 - 2 may receive a user input for selecting an edit function button through the control panel.

Also, the second device 100 - 2 may receive a preset gesture corresponding to the edit command. For example, when the user double-taps the selected handwritten content with two fingers after selecting some handwritten content by a drag input, the second device 100 - 2 requests an undo for the partial handwritten content It can be determined that this has been received.

Meanwhile, the user may input an editing command for some handwritten content into the second device 100 - 2 using a touch input and a voice input. For example, the user may select some handwritten content through a touch input and input a command for requesting editing of some handwritten content into the second device 100 - 2 by voice.

In operation S6160, the second device 100 - 2 may edit some handwritten content. According to an embodiment of the present invention, the second device 100 - 2 may edit the partial handwritten content by using time information of strokes included in the partial handwritten content.

According to an embodiment of the present invention, the second device 100 - 2 may extract strokes included in some handwritten content. The second device 100 - 2 may determine an order between the extracted strokes by using time stamp information of each of the extracted strokes. For example, the second device 100 - 2 determines a first stroke input first among the extracted strokes as a start position, and the n-th stroke ( n is an integer greater than 1) may be determined as the end position.

According to an embodiment of the present invention, the second device 100 - 2 may edit the extracted strokes using time stamp information and coordinate information of the extracted strokes. For example, the second device 100 - 2 may undo, redo, or delete the extracted strokes, or may add other strokes.

In operation S6170 , the second device 100 - 2 may transmit the edited handwritten content to the first device 100 - 1 . According to an embodiment of the present invention, the second device 100-2 may transmit an original file of the handwritten content and an edit file including editing information to the first device 100-1, respectively. Also, the second device 100 - 2 may apply the editing information to the handwritten content to generate one file, and transmit the one generated file to the first device 100 - 1 .

According to an embodiment of the present invention, the second device 100 - 2 may transmit the edited handwritten content to the first device 100 - 1 using short-distance communication or mobile communication.

In operation S6180, the first device 100-1 may display the handwritten content edited by the second device 100-2.

According to an embodiment of the present invention, the first device 100 - 1 may receive a reproduction request for the edited handwritten content. For example, the first device 100 - 1 may display a virtual control panel including a play function button. The first device 100 - 1 may receive a user input for selecting a play function button through a virtual control panel.

Also, the first device 100 - 1 may receive a preset touch input corresponding to the play command or may receive an input for selecting a physical button corresponding to the play command. Meanwhile, the user may input a command for requesting reproduction of the handwritten content to the first device 100 - 1 by voice.

According to an embodiment of the present invention, the first device 100 - 1 may reproduce the handwritten content by reflecting editing information for some handwritten content. For example, the device 100 may reproduce the drawing process of the handwritten content in a state in which the editing information is reflected.

According to an embodiment of the present invention, the first device 100 - 1 may additionally reproduce the editing process after playing the drawing process of the original handwritten content. Also, the first device 100 - 1 may provide a selection window for selecting whether to reflect edit information for some handwritten content. Referring to FIGS. 62A to 62C , a more detailed description will be given.

62A is a diagram illustrating an example in which a second device reproduces handwritten content received from a first device. 62A to 62C, a case where the first device 100-1 is a student terminal and the second device 100-2 is a teacher terminal will be described as an example.

As shown in FIG. 62A , when the first student handwrites the formula for calculating the volume of a sphere and the Pythagorean theorem in the first device 100-1, the first device 100-1 performs the first student's handwriting By recording the process, the first handwritten content 6200 may be generated. In addition, the first device 100 - 1 may transmit the first handwritten content 6200 to the second device 100 - 2 .

The second device 100-2 may display the first handwritten content 6200 received from the first device 100-1 on the screen. According to an embodiment of the present invention, the second device 100 - 2 may provide a virtual control panel including a play function button.

According to an embodiment of the present invention, when the teacher designates the region of interest 6201 including 'v=3/4πr3' and selects the play button 6210 included in the control panel, the second device 100- 2), the drawing process of some handwritten content ('v=3/4πr3') may be reproduced by using time information of strokes included in the region of interest 6201 . For example, the first v drawn in the region of interest 6201 may be output.

62B is a diagram illustrating an example in which the second device edits some handwritten content.

Referring to FIG. 62B , the second device 100 - 2 receives an input for designating a region of interest 6220 including '3/4' and provides a control panel including a cancel button and an add button. can The second device 100 - 2 may receive an input for changing '3/4' to '4/3' through the control panel. In this case, the second device 100 - 2 deletes information on the strokes corresponding to '3/4' from the list of a plurality of strokes included in the handwritten content 6200, and the stroke corresponding to '4/3' By adding their information, the handwritten content 6200 may be edited. In addition, according to an embodiment of the present invention, the second device 100-2 does not delete information on strokes corresponding to '3/4', but adds a deletion tag to the handwritten content 6200. You can also edit it.

62C is a diagram illustrating an example in which the first device reproduces the entire handwritten content by reflecting editing information for some handwritten content.

Referring to FIG. 62C , the second device 100 - 2 may transmit the edited handwritten content 6200 ′ to the first device 100 - 1 .

According to an embodiment of the present invention, the first device 100-1 may display the edited handwritten content 6200' on the screen. In this case, the first device 100 - 1 may receive an input for selecting the play button 6240 on the virtual control panel.

According to an embodiment of the present invention, the first device 100-1 reflects the editing information (eg, '3/4' is changed to '4/3') for some handwritten content, and the edited handwriting is Content 2600' may be reproduced. For example, the first device 100 - 1 may reproduce the edited handwritten content 2600 ′ so that “4/3” is drawn instead of “3/4”.

According to an embodiment of the present invention, the first device 100-1 reproduces the drawing process of the original handwritten content 2600 and then deletes the editing process (eg, '3/4' and '4/3'). additional) can also be played. Also, the first device 100 - 1 may provide a selection window for selecting whether to reflect edit information for some handwritten content.

Meanwhile, in the above description, the case where the content displayed on the device 100 is handwritten content has been described as an example, but the present invention is not limited thereto. According to an embodiment of the present invention, the device 100 may display a transcript obtained by converting voice into text. An operation of providing a transcript in the device 100 will be described later in detail with reference to FIG. 69 .

62A to 62C, when receiving an input for selecting the play button 6240 from the first device 100-1, the first device 100-1 plays the edited handwritten content 2600' Although the case has been described as an example, the present invention is not limited thereto. For example, even when the first device 100-1 receives an input for selecting the play button 6240, the handwritten content 2600' edited by the second device 100-2 may be played.

Hereinafter, a case in which the handwritten content (or a part of the handwritten content) selected in the first device 100-1 is reproduced in the second device 100-2 will be described in detail with reference to FIG. 63 .

63 is a flowchart illustrating a method of reproducing a part of handwritten content selected in a first device in a second device, according to an embodiment of the present invention.

In operation S6310 , the first device 100 - 1 may perform a communication connection with the second device 100 - 2 . For example, the first device 100-1 may form a short-range communication link with the second device 100-2, or may form a mobile communication link (eg, 3G, 4G, 5G, etc.) . Short-range communication may include, but is not limited to, Bluetooth, Bluetooth Low Energy (BLE), Wi-Fi Direct, ultra wideband (UWB), Zigbee, Near Field Communication unit (NFC), Ant+, and the like. In step S6320, the first The device 100 - 1 may display handwritten content. According to an embodiment of the present invention, the handwritten content may include a plurality of strokes input in a predetermined order. Each of the plurality of strokes included in the handwritten content may have various shapes. For example, the thickness, color, brightness, saturation, transparency, etc. of each of the plurality of strokes may be different.

According to an embodiment of the present invention, the handwritten content displayed on the screen of the first device 100 - 1 may be the original handwritten content or the edited handwritten content, but is not limited thereto.

In operation S6330, the first device 100-1 may receive an input for selecting a part of the handwritten content (referred to as 'partial handwritten content'). For example, the first device 100 - 1 may receive a user input designating an ROI and select some handwritten content corresponding to the ROI. Also, the first device 100 - 1 may receive a user input of sequentially touching a plurality of strokes while pressing a predetermined button of the electronic pen 200 . In this case, the first device 100 - 1 may select a plurality of strokes touched by the electronic pen 200 as some handwritten content.

In operation S6340 , the first device 100 - 1 may transmit information about a part of the handwritten content (partial handwritten content) to the second device 100 - 2 . For example, the first device 100-1 may transmit information about a part of the handwritten content (partial handwritten content) to the second device 100-2 using a short-distance communication link or a mobile communication link. have.

According to an embodiment of the present invention, the first device 100 - 1 may transmit information on strokes included in some handwritten content (eg, stroke identification information, time stamp information, coordinate information, etc.). According to an embodiment of the present invention, the first device 100 - 1 may compress or encrypt information on strokes included in some handwritten content and transmit it to the second device 100 - 2 . In this case, the first device 100 - 1 may use various compression algorithms or encryption algorithms.

According to an embodiment of the present invention, the first device 100 - 1 may transmit information about a part of the handwritten content to the second device 100 - 2 while playing the part of the handwritten content. Also, the first device 100-1 may transmit information about a part of the handwritten content to the second device 100-2 without reproducing the part of the handwritten content.

In operation S6350 , the second device 100 - 2 may reproduce a part of the handwritten content in the same temporal manner as the handwritten content. For example, the second device 100 - 2 may reproduce a portion of the handwritten content using information about a portion (partial handwritten content) of the handwritten content received from the first device 100 - 2 ).

According to an embodiment of the present invention, the second device 100 - 2 may generate a stack in which at least one stroke included in some handwritten content is inserted in chronological order. For example, the second device 100 - 2 may determine the relative time order of the strokes by comparing the time stamp information between the extracted strokes. The second device 100 - 2 may insert the longest-generated stroke among the extracted strokes at the top of the stack, and insert the most recently-generated stroke as it goes down in the stack. The second device 100 - 2 may reproduce some handwritten content using the generated stack in the same temporal manner as it was written.

According to an embodiment of the present invention, when some handwritten content is reproduced in the first device 100-1, the second device 100-2 streams the partial handwritten content from the first device 100-1. data can be received. In addition, the second device 100 - 2 may reproduce some handwritten content using the received streaming data in the same temporal manner as the handwritten content.

64 is a diagram illustrating an example of reproducing a part of content selected by a device on an external display device.

Referring to FIG. 64 , the device 6410 may display handwritten content 6400 . In this case, the writing content 6400 may include data in which a process of writing a formula for calculating the volume of a sphere and a process of writing the Pythagorean theorem is recorded.

The device 6410 receives an input for designating a region of interest 6401 including 'a sphere shape and v=4/3πr3' among the handwritten content 6400 and an input for selecting a play button 6402 included in the control panel. can receive

In response to an input of selecting the play button 6402, the device 6410 transmits information about some handwritten content (eg, 'the shape of a sphere and v=4/3πr3') included in the region of interest 6401 to an external device. While transmitting to 6420 , it is possible to request (or command) the external device 6420 to play some handwritten content. For example, when the device 6410 is a tablet PC, the tablet PC may control to play some handwritten content selected by the user on a smart TV having a larger display than the tablet PC.

In this case, the external device 6420 may reproduce the drawing process 6403 of some handwritten content ('shape of a sphere and v=4/3πr3') using time information of strokes included in the region of interest 6401 . have. For example, the external device 6420 may draw a sphere first, then draw a straight line and r in the sphere, and then sequentially draw v=4/3πr3 according to the order of the handwriting input.

Hereinafter, with reference to FIG. 65 , an example in which a part of handwritten content selected on a device of a first user is reproduced on a device of a second user will be described.

65 is a diagram illustrating an example in which a part of handwritten content selected in the first device is reproduced in the same temporal manner as that in which it was written. In FIG. 65 , a case where the first device 100-1 is the teacher terminal 6500-1 and the second device 100-2 is the student terminal 6500-2 will be described as an example.

Referring to FIG. 65 , the teacher terminal 6500-1 may display handwritten content 6500. In this case, the writing content 6500 may include data in which a process of writing a formula for calculating the volume of a sphere and a process of writing the Pythagorean theorem is recorded.

The teacher terminal 6500-1 designates a region of interest 6501 including 'the shape of a sphere and v=4/3πr3' among the handwritten content 6500, and selects a play button 6502 included in the control panel. input can be received.

In response to an input for selecting the play button 6502, the teacher terminal 6500-1 is a student terminal 6500-2 connected to the teacher terminal 6500-1. Some handwritten content included in the region of interest 6501 ( For example, information regarding 'the shape of a sphere and v=4/3πr3') may be transmitted. In this case, the teacher terminal 6500-1 may request (or command) the student terminal 6500-2 to reproduce some handwritten content (eg, 'the shape of a sphere and v=4/3πr3').

In this case, the student terminal 6500 - 2 uses time information of strokes included in the region of interest 6501 in response to a request (or command) to reproduce some handwritten content, and uses the /3πr3') of the drawing process can be reproduced. For example, the student terminal 6500 - 2 may first output a process of drawing a sphere according to the order in which handwriting is input ( 6510 ). In addition, the student terminal 6500-2 may output a process of drawing a straight line and r in a sphere and drawing a v (6520), and the student terminal 6500-2 then sequentially calculates '=4/3πr3' can be drawn with

66 is a diagram illustrating an example of reproducing a part of handwritten content selected by the first device in a plurality of external devices. In FIG. 66 , a case in which the first device 100-1 is a teacher terminal 6610 and the second device 100-2 is a plurality of student terminals 6620, 6630, and 6640 will be described as an example.

Referring to FIG. 66 , the teacher terminal 6610 may display handwritten content 6600 . At this time, the teacher terminal 6610 designates a region of interest 6601 including 'the shape of a sphere and v=4/3πr3' among the handwritten content 6600, and selects the play button 6602 included in the control panel. input can be received.

In response to an input of selecting the play button 6602, the teacher terminal 6610 includes a plurality of student terminals 6620, 6630, 6640 connected to the teacher terminal 6610, some handwritten content included in the region of interest 6601 ( For example, information regarding 'the shape of a sphere and v=4/3πr3') may be transmitted. In this case, the teacher terminal 6610 may request (or command) the plurality of student terminals 6620 , 6630 , and 6640 to reproduce some handwritten content (eg, ‘the shape of a sphere and v=4/3πr3’).

In this case, each of the plurality of student terminals 6620 , 6630 , and 6640 uses time information of strokes included in the region of interest 6601 in response to a reproduction request (or command) of the partial handwritten content (' The shape of the sphere and the drawing process of v=4/3πr3') can be reproduced.

For example, the first student terminal 6620 may first draw a sphere according to the order in which handwriting is input, then draw a straight line and r in the sphere, and then sequentially draw v=4/3πr3. In addition, the second student terminal 6630 may first draw a sphere according to the order of the handwriting input, then draw a straight line and r in the sphere, and then sequentially draw v=4/3πr3. The third student terminal 6640 may also draw a sphere first according to the order of handwriting input, then draw a straight line and r in the sphere, and then sequentially draw v=4/3πr3.

67 is a flowchart for explaining a method of respectively reproducing a part of handwritten content selected by the first device by the first device and the second device;

In operation S6710 , the first device 100 - 1 may perform a communication connection with the second device 100 - 2 . For example, the first device 100-1 may form a short-range communication link with the second device 100-2, or may form a mobile communication link (eg, 3G, 4G, 5G, etc.) . The short-range communication may include, but is not limited to, Bluetooth, Bluetooth Low Energy (BLE), Wi-Fi Direct, ultra wideband (UWB), Zigbee, Near Field Communication unit (NFC), Ant+, and the like.

In operation S6720, the first device 100-1 may display the handwritten content. According to an embodiment of the present invention, the handwritten content may include a plurality of strokes input in a predetermined order. According to an embodiment of the present invention, the handwritten content displayed on the screen of the first device 100 - 1 may be the original handwritten content or the edited handwritten content, but is not limited thereto.

In operation S6730, the first device 100-1 may receive an input for selecting a part of the handwritten content (referred to as 'partial handwritten content'). For example, the first device 100 - 1 may receive a user input designating an ROI and select some handwritten content corresponding to the ROI. Also, the first device 100 - 1 may receive a user input of sequentially touching a plurality of strokes while pressing a predetermined button of the electronic pen 200 . In this case, the first device 100 - 1 may select a plurality of strokes touched by the electronic pen 200 as some handwritten content.

In operation S6740, the first device 100-1 may reproduce a part of the handwritten content in the same temporal manner as that in which it was written and display it on the screen.

According to an embodiment of the present invention, the second device 100 - 2 may generate a stack in which at least one stroke included in some handwritten content is inserted in chronological order. For example, the second device 100 - 2 may determine the relative time order of the strokes by comparing the time stamp information between the extracted strokes. The second device 100 - 2 may insert the longest-generated stroke among the extracted strokes at the top of the stack, and insert the most recently-generated stroke as it goes down in the stack. The second device 100 - 2 may reproduce some handwritten content using the generated stack in the same temporal manner as when it was written.

In operation S6750, the first device 100-1 may transmit information about a part of the handwritten content. For example, the first device 100-1 transmits information about a part of the handwritten content (partial handwritten content) to the second device 100-2 using a short-distance communication link, a mobile communication link, a wireless LAN, or the like. can be sent to

According to an embodiment of the present invention, the first device 100 - 1 may transmit information on strokes included in some handwritten content (eg, stroke identification information, time stamp information, coordinate information, etc.). According to an embodiment of the present invention, the first device 100 - 1 may compress or encrypt information on strokes included in some handwritten content and transmit it to the second device 100 - 2 . In this case, the first device 100 - 1 may use various compression algorithms or encryption algorithms.

According to an embodiment of the present invention, the first device 100 - 1 may transmit streaming data regarding the reproduced partial handwritten content to the second device 100 - 2 while playing the partial handwritten content.

According to an embodiment of the present invention, the first device 100-1 encodes some handwritten content through a mirroring technology using Miracast, and transmits the encoded data to the second device 100- through a network channel. 2) can be transmitted. Here, the network channel may be various, such as WFD (Wi-Fi Direct), Bluetooth, Zigbee, NFC, BLE. Unlike existing Wi-Fi, Miracast provides a basis for using content and services between devices through direct communication between devices without a separate AP or router.

In operation S6760, the second device 100 - 2 may reproduce a part of the handwritten content in the same temporal manner as the handwritten content. For example, the second device 100 - 2 may reproduce a portion of the handwritten content using information about a portion (partial handwritten content) of the handwritten content received from the first device 100 - 2 ).

According to an embodiment of the present invention, the second device 100 - 2 may reproduce and display a portion of the handwritten content directly on the screen without storing information about a portion of the handwritten content in a streaming manner.

Also, according to another embodiment of the present invention, the second device 100 - 2 stores information about a part of the handwritten content in a download method, and then uses the stored information about the part of the handwritten content to download the handwritten content. You can even play some.

According to an embodiment of the present invention, the screen of the first device 100-1 that reproduces some handwritten content and the screen of the second device 100-2 may be synchronized. An embodiment in which the screen of the first device 100-1 and the screen of the second device 100-2 are synchronized will be described in more detail with reference to FIG. 68 .

68 is a diagram illustrating an example in which the first device and the second device respectively reproduce a part of the handwritten content selected by the first device in the same temporal manner as when the handwritten content was written.

Referring to 6810 of FIG. 68 , the first device 680 - 1 may display handwritten content 6800 . In this case, the writing content 6800 may include data in which a process of writing a formula for calculating the volume of a sphere and a process of writing the Pythagorean theorem is recorded.

The first device 6800-1 receives an input for designating a region of interest 6801 including 'a sphere shape and v=4/3πr3' among the handwritten content 6800 and a play button 6802 included in the control panel. It can receive input to select.

Referring to 6820 of FIG. 68 , the first device 6800-1 reproduces the drawing process of some handwritten content ('the shape of a sphere and v=4/3πr3') in response to an input of selecting the play button 6802 . can For example, the first device 680 - 1 may first output a process of drawing a sphere according to the order in which handwriting is input by using time information of strokes included in the region of interest 6801 .

In this case, the process of drawing a sphere may also be output from the second device 6800 - 2 . For example, the first device 6800-1 transmits information about some handwritten content (eg, 'the shape of a sphere and v=4/3πr3') included in the region of interest 6801 to the second device 6800-2 ), it is possible to request (or command) the second device 6800 - 2 to play some handwritten content. In this case, the second device 6800 - 2 may first output the process of drawing a sphere like the first device 680 - 1 .

Also, the first device 6800 - 1 may periodically transmit data on a screen on which some handwritten content is displayed to the second device 6800 - 2 by using a mirroring technique. In this case, the second device 6800 - 2 may display the same screen as the first device 680 - 1 .

Referring to 6830 of FIG. 68 , the first device 6800-1 may display a screen on which v=4/3πr3 is drawn after drawing a sphere. In this case, a screen drawing v=4/3πr3 may also be output to the second device 6800 - 2 .

69 is a flowchart illustrating a method for a device to provide multimedia content of a partial section corresponding to a selected transcript, according to an embodiment of the present invention.

In step S6910, the device 100 may display a list of transcripts corresponding to multimedia content (eg, video content or audio content). In the present specification, a transcript may mean a record in which dictated content is translated into words. According to an embodiment of the present invention, one transcript may consist of one sentence.

According to an embodiment of the present invention, the device 100 may generate a transcript corresponding to the multimedia content by converting a voice included in the multimedia content into text based on a speech to text (STT) technology. In this case, the transcript may be mapped with reproduction time information of the multimedia content.

According to an embodiment of the present invention, the device 100 may generate a transcript by converting some voices included in the multimedia content into text. For example, when the voice is not output for a certain period of time during video lecture content playback and the voice is output again at a specific time point, the device 100 converts the voice output at a specific time point into text to generate a transcript. can According to an embodiment of the present invention, the device 100 may generate a transcript by converting the inserted voice into text when the screen is switched in the video content.

Meanwhile, according to an embodiment of the present invention, when the syllables of the voice are greater than or equal to a predetermined length, the device 100 may convert the voice into text to generate a transcript. For example, the device 100 is 'well... Transcripts may not be generated for voices with short syllables such as ' or 'no'.

In operation S6920, the device 100 may receive an input for selecting the first transcript from the list of transcripts.

An input for selecting the first transcript from the transcript list may be various. According to an embodiment of the present invention, the device 100 may receive an input for designating a region of interest in the transcript list. The shape of the region of interest may vary. For example, the ROI may have a circular shape, an elliptical shape, a rectangular shape, a free curve shape, and the like, but is not limited thereto. Also, the color and pattern of the ROI may be varied.

Also, the device 100 may receive a preset gesture for selecting the first transcript. For example, when the user touches the first transcript for more than a preset time (eg, touch and hold for 3 seconds or more) or touches the first transcript more than a predetermined number of times (eg, double tap), the device 100 receives the first transcript You can choose.

In step S6930, the device 100 may extract a partial section corresponding to the first transcript from the multimedia content.

For example, the device 100 may select the first frame reproduced at the start of generating the first transcript. Also, the device 100 may select the n-th frame reproduced at the time when the generation of the first transcript is finished. The device 100 may select the nth frame corresponding to the generation time of the second transcript generated after the first transcript. The device 100 may extract the section from the first frame to the nth frame as a partial section corresponding to the first transcript.

In step S6940, the device 100 may provide the multimedia content of the extracted partial section.

For example, the device 100 may move to a start frame of a partial section corresponding to the first transcript and reproduce the partial section corresponding to the first transcript. Also, the device 100 may move the timeline of the multimedia content to the start time of a partial section corresponding to the first transcript.

70 is a diagram illustrating an example in which a device provides a partial video section corresponding to a selected transcript according to an embodiment of the present invention.

Referring to FIG. 70 , the device 100 may generate a plurality of transcripts by converting a voice included in the video content 7010 into text. In addition, the device 100 may display a list 7020 of a plurality of transcripts.

According to an embodiment of the present invention, the device 100 may display the entire sentence in which the voice is converted into text on the list 7020, or display the text for a preset length from the front of the sentence on the list 7020. may be

According to an embodiment of the present invention, the device 100 may display information on the playback start time (eg, 00.00.10) of a partial video section corresponding to each transcript in the list 7020 .

According to an embodiment of the present invention, the device 100 may receive an input for selecting a second transcript 7021 (eg, let's take a continuous example) from a list 7020 of a plurality of transcripts. can The device 100 may extract a video section corresponding to the second transcript 7021 . For example, when the second section (00.13.20 - 00.20.30) of the video content 7010 is reproduced when the second transcript 7021 is generated, the device 100 performs the second section of the video content 7010 can be extracted. In addition, the device 100 may move the timeline of the video content 7010 to the start frame of the second section.

According to an embodiment of the present invention, the user can easily search for a frame of interest in the multimedia content by selecting a specific transscript.

71 is a diagram illustrating an example in which a device searches for a partial video section based on a search keyword.

Referring to FIG. 71 , the device 100 may generate a plurality of transcripts by converting a voice included in the video content 7010 into text. In addition, the device 100 may store a list 7020 of a plurality of transcripts. In this case, the transcript may be mapped with reproduction time information of the multimedia content.

According to an embodiment of the present invention, the device 100 may receive a search keyword input. For example, when the user wants to search for a partial section related to 'example of discontinuity', the user may input 'example of discontinuity' as a search keyword in the search window 7100 of the device 100 .

According to an embodiment of the present invention, the device 100 may extract the third transcript including the 'example of discontinuity' from the list 7020 of the plurality of transcripts. In addition, the device 100 may select the first frame 7110 reproduced at the start time of generation of the third transcript. Also, the device 100 may select the second frame 7120 reproduced at the end of generation of the third transcript.

The device 100 may extract a section from the first frame 7110 to the second frame 7120 as a partial video section 7130 corresponding to the third transcript, and reproduce the partial video section 7130 .

72A and 72B are diagrams illustrating an example in which a device provides a partial audio section corresponding to a selected word according to an embodiment of the present invention.

Referring to FIG. 72A , the device 100 converts a voice included in the audio content 7200 into text, and a transcript (eg, Hi, Let me introduce myself. My name is Kim Chul Su. I live in seoul. ) can be created. The device 100 may display the transcript during playback of the audio content 7200 .

According to an embodiment of the present invention, the device 100 may receive an input for selecting a specific word 7210 (eg, 'myself') included in the transcript.

Referring to FIG. 72B , the device 100 may extract a first timeline (eg, 00:10) on which a specific word 7210 (eg, 'myself') is output. The device 100 may move the timeline of the audio content 7200 to the first timeline (eg, 00:10).

73 and 74 are block diagrams for explaining the configuration of a device according to an embodiment of the present invention.

73 , the device 100 according to an embodiment of the present invention may include a display unit 111 , a user input unit 120 , and a control unit 130 . However, not all illustrated components are essential components. The device 100 may be implemented by more elements than the illustrated elements, and the device 100 may be implemented by fewer elements than that.

For example, as shown in FIG. 74 , the device 100 according to an embodiment of the present invention includes a communication unit 140 and a sensing unit in addition to the output unit 110 , the user input unit 120 , and the control unit 130 . 150 , an A/V input unit 160 , and a memory 170 may be further included.

Hereinafter, the above components will be described in turn.

The output unit 110 is for outputting an audio signal, a video signal, or a vibration signal, and may include a display unit 111 , a sound output unit 112 , a vibration motor 113 , and the like.

The display unit 111 displays and outputs information processed by the device 100 . For example, the display 111 may display the handwritten content on the screen. Also, when some handwritten content is selected, the display unit 111 may display a control panel including at least one of a play function button and an edit function button. In this case, the display unit 111 may display the control panel within a predetermined distance from some handwritten content.

The display unit 111 may display a transcript obtained by converting voice into text. For example, the display unit 111 may display a list of transcripts corresponding to multimedia content (eg, video content or audio content).

Meanwhile, when the display unit 111 and the touchpad form a layer structure to form a touch screen, the display unit 111 may be used as an input device in addition to an output device. The display unit 111 includes a liquid crystal display, a thin film transistor-liquid crystal display, an organic light-emitting diode, a flexible display, a three-dimensional display ( 3D display) and electrophoretic display (electrophoretic display) may include at least one. In addition, depending on the implementation form of the device 100 , the device 100 may include two or more display units 111 .

The sound output unit 112 outputs audio data received from the communication unit 140 or stored in the memory 170 . Also, the sound output unit 112 outputs a sound signal related to a function (eg, a call signal reception sound, a message reception sound, and a notification sound) performed by the device 100 . The sound output unit 112 may include a speaker, a buzzer, and the like.

The vibration motor 113 may output a vibration signal. For example, the vibration motor 113 may output a vibration signal corresponding to the output of audio data or video data (eg, a call signal reception sound, a message reception sound, etc.). Also, the vibration motor 113 may output a vibration signal when a touch is input to the touch screen.

The user input unit 120 means a means for a user to input data for controlling the device 100 . For example, the user input unit 120 includes a key pad, a dome switch, and a touch pad (contact capacitive method, pressure resistance film method, infrared sensing method, surface ultrasonic conduction method, integral type). There may be a tension measurement method, a piezo effect method, etc.), a jog wheel, a jog switch, and the like, but is not limited thereto.

The user input unit 120 may receive a user input for selecting some handwritten content from among handwritten content displayed on the screen. For example, the user input unit 120 may receive an input specifying an ROI on the handwritten content displayed on the screen.

According to an embodiment of the present invention, the user input unit 120 may receive an input for selecting a play function button or an edit function button through a virtual control panel.

The playback function button may include at least one of a playback button, a pause button, a forward button, a reverse button, and a playback speed control button. The edit function button may include at least one of an undo button, a redo button, a character recognition button, an add button, and an exchange button. The control panel may include a search bar for searching for a partial section of some handwritten content.

The controller 130 generally controls the overall operation of the device 100 . For example, the controller 130 executes programs stored in the memory 170 , and thus the output unit 110 , the user input unit 120 , the communication unit 140 , the sensing unit 150 , and the A/V input unit 160 . ), etc., can be controlled in general.

The controller 130 may generate the handwritten content by acquiring time stamp information and coordinate information corresponding to each of the plurality of strokes. Also, the controller 130 may reproduce the partial handwritten content by using time information of strokes included in the partial handwritten content.

The controller 130 may generate the handwritten content while the multimedia content is being reproduced, synchronize the handwritten content with the multimedia content and store it in the memory. The multimedia content may include at least one of video content and audio content.

The controller 130 may select some handwritten content corresponding to the ROI. For example, the controller 130 may select some handwritten content corresponding to the ROI based on a stroke unit or a coordinate unit.

The controller 130 may extract strokes included in some handwritten content and determine an order between the strokes by using time stamp information corresponding to the strokes. For example, the controller 130 may determine a first inputted stroke among strokes as a start position, and determine a last inputted stroke among strokes as an end position.

The controller 130 may extract video content or audio content corresponding to some handwritten content, and provide the extracted video content or audio content.

When a plurality of partial handwritten contents are selected, the controller 130 may provide a list of a plurality of video contents corresponding to the plurality of partial handwritten contents.

The controller 130 may edit the partial handwritten content by using time information of strokes included in the partial handwritten content.

The communication unit 140 may include one or more components that enable communication between the device 100 and an external device or between the device 100 and a server. For example, the communication unit 140 may include a short-distance communication unit 141 , a mobile communication unit 142 , and a broadcast receiving unit 143 .

Short-range wireless communication unit 141 is a Bluetooth communication unit, BLE (Bluetooth Low Energy) communication unit, short-range wireless communication unit (Near Field Communication unit), WLAN (Wi-Fi) communication unit, Zigbee (Zigbee) communication unit, infrared ( It may include an IrDA, infrared Data Association) communication unit, a Wi-Fi Direct (WFD) communication unit, an ultra wideband (UWB) communication unit, an Ant+ communication unit, and the like, but is not limited thereto.

The mobile communication unit 142 transmits/receives wireless signals to and from at least one of a base station, an external terminal, and a server on a mobile communication network. Here, the wireless signal may include various types of data according to transmission and reception of a voice call signal, a video call signal, or a text/multimedia message.

The broadcast receiving unit 143 receives a broadcast signal and/or broadcast-related information from the outside through a broadcast channel. The broadcast channel may include a satellite channel and a terrestrial channel. According to an implementation example, the device 100 may not include the broadcast receiver 143 .

The communication unit 140 may transmit the handwritten content to an external server or an external device. The communication unit 140 may receive the handwritten content edited in the external device from the external device.

The communication unit 140 may receive pulse information, an electrocardiogram degree, temperature information, and the like from an external wearable device. For example, the device 100 may receive pulse information and temperature information from the smart watch.

The sensing unit 150 may detect a state of the device 100 or a state around the device 100 , and transmit the sensed information to the controller 130 .

The sensing unit 150 includes a magnetic sensor 151 , an acceleration sensor 152 , a temperature/humidity sensor 153 , an infrared sensor 154 , a gyroscope sensor 155 , and a position sensor. (eg, GPS) 156 , a barometric pressure sensor 157 , a proximity sensor 158 , and at least one of an illuminance sensor 159 , but is not limited thereto. Since a function of each sensor can be intuitively inferred from the name of a person skilled in the art, a detailed description thereof will be omitted.

The A/V (Audio/Video) input unit 160 is for inputting an audio signal or a video signal, and may include a camera 161 and a microphone 162 . The camera 161 may obtain an image frame such as a still image or a moving image through an image sensor in a video call mode or a shooting mode. The image captured through the image sensor may be processed through the controller 130 or a separate image processing unit (not shown).

The image frame processed by the camera 161 may be stored in the memory 170 or transmitted to the outside through the communication unit 150 . Two or more cameras 161 may be provided according to the configuration of the device 100 .

The microphone 162 receives an external sound signal and processes it as electrical voice data. For example, the microphone 162 may receive an acoustic signal from an external device or a speaker. The microphone 162 may use various noise removal algorithms to remove noise generated in the process of receiving an external sound signal.

The memory 170 may store a program for processing and control of the controller 130 , and may store input/output data (eg, an application, handwritten content, multimedia content, transcript, etc.).

The memory 170 may include a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (eg, SD or XD memory), and a RAM. (RAM, Random Access Memory) SRAM (Static Random Access Memory), ROM (Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), PROM (Programmable Read-Only Memory), magnetic memory, magnetic disk , may include at least one type of storage medium among optical disks. In addition, the device 100 may operate a web storage or a cloud server that performs a storage function of the memory 170 on the Internet.

Programs stored in the memory 170 can be classified into a plurality of modules according to their functions, for example, the UI module 171 , the touch screen module 172 , the notification module 173 , and STT (Speak to Text). ) may be classified into a module 174 and the like.

The UI module 171 may provide a specialized UI, GUI, etc. that are interlocked with the device 100 for each application. The touch screen module 172 may detect a touch gesture on the user's touch screen and transmit information about the touch gesture to the controller 120 . The touch screen module 172 according to an embodiment of the present invention may recognize and analyze a touch code. The touch screen module 172 may be configured as separate hardware including a controller.

Various sensors may be provided inside or near the touch screen to detect a touch or a proximity touch of the touch screen. As an example of a sensor for detecting a touch of a touch screen, there is a tactile sensor. A tactile sensor refers to a sensor that senses a touch of a specific object to the extent or higher than that of a human being. The tactile sensor may sense various information such as the roughness of the contact surface, the hardness of the contact object, and the temperature of the contact point.

In addition, as an example of a sensor for detecting a touch of a touch screen, there is a proximity sensor.

The proximity sensor refers to a sensor that detects the presence or absence of an object approaching a predetermined detection surface or an object existing in the vicinity without mechanical contact using the force of an electromagnetic field or infrared rays. Examples of the proximity sensor include a transmission type photoelectric sensor, a direct reflection type photoelectric sensor, a mirror reflection type photoelectric sensor, a high frequency oscillation type proximity sensor, a capacitive type proximity sensor, a magnetic type proximity sensor, and an infrared proximity sensor. The user's touch gesture may include tap, touch & hold, double tap, drag, pan, flick, drag and drop, swipe, and the like.

The notification module 173 may generate a signal for notifying the occurrence of an event in the device 100 . Examples of events generated in the device 100 include call signal reception, message reception, key signal input, schedule notification, and the like. The notification module 173 may output a notification signal in the form of a video signal through the display unit 111 , may output a notification signal in the form of an audio signal through the sound output unit 112 , and the vibration motor 113 . It is also possible to output a notification signal in the form of a vibration signal through

The Speech to Text (STT) module 174 may generate a transcript corresponding to the multimedia content by converting a voice included in the multimedia content into text. In this case, the transcript may be mapped with reproduction time information of the multimedia content.

The method according to an embodiment of the present invention may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc. alone or in combination. The program instructions recorded on the medium may be specially designed and configured for the present invention, or may be known and available to those skilled in the art of computer software. Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic such as floppy disks. - includes magneto-optical media, and hardware devices specially configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine language codes such as those generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like.

According to an embodiment of the present invention, the device 100 may reproduce a drawing process for some handwritten content selected by the user by using the time information of the torques.

Although the embodiments of the present invention have been described in detail above, the scope of the present invention is not limited thereto. belong to the scope of the right.

Claims (19)

a display unit configured to display handwritten content including a group of strokes inputted in a temporal order and created based on a user's analog handwriting input;
a memory storing time information of strokes included in the group of strokes;
a user input unit receiving a first user input for selecting a first area from among the handwritten content displayed on the display unit; and
and a control unit configured to reproduce a first drawing process corresponding to the first area of the handwritten content according to time information of first strokes included in the first area of the handwritten content based on the first user input, ,
The user input unit receives a second user input for editing the first area of the handwritten content,
The control unit is configured to edit the first area of the handwritten content by using the time information of the first strokes included in the first area of the handwritten content according to the second user input, and When the whole is reproduced, the first area of the handwritten content is reproduced through a second drawing process corresponding to the handwriting input of the edited first area instead of the first drawing process. delete delete According to claim 1, wherein the control unit,
The device of claim 1, wherein the handwritten content of the selected first area is reproduced in the same temporal order as the handwritten content. The method of claim 1,
the time information of the strokes includes time stamp information,
The control unit is
Obtaining time stamp information corresponding to the first strokes included in the selected first area from the memory, and using time stamp information corresponding to the first strokes, an order between the first strokes ) to determine the device. According to claim 5, wherein the control unit,
A first stroke input first among the first strokes is determined as a start position, and a last stroke input last among the first strokes is determined as an end position. device. According to claim 1, wherein the control unit,
When a plurality of regions of the handwritten content displayed on the display unit are selected, the plurality of regions of the handwritten content are reproduced according to time information of second strokes included in the plurality of selected regions. According to claim 1, wherein the control unit,
when the first area of the handwritten content is selected, controlling the display unit to display a control panel including a playback function button within a predetermined distance from the first area;
The user input unit, the device characterized in that it receives an input for selecting the play function button through the control panel. delete A method for playing content on a device, comprising:
displaying handwriting content that is created based on a user's analog handwriting input and includes a group of strokes inputted in a temporal order;
storing time information of strokes included in the group of strokes in a memory;
receiving a first user input for selecting a first area of the handwritten content;
reproducing a first drawing process corresponding to the first area of the handwritten content according to time information of first strokes included in the first area of the handwritten content based on the first user input;
When a second user input for requesting editing of the first area of the handwritten content is received, the second user input of the handwritten content is generated using the time information of the first strokes included in the first area of the handwritten content. 1 editing an area; and
reproducing the first area of the handwritten content by a second drawing process corresponding to the handwriting input of the edited first area instead of the first drawing process when the entire handwriting content is reproduced;
Content playback method comprising a. delete delete The method of claim 10 , wherein reproducing the first drawing process corresponding to the selected first area comprises:
and reproducing the handwritten content of the selected first area in the same temporal manner as the handwritten content. The method of claim 10 , wherein reproducing the first drawing process corresponding to the selected first area comprises:
obtaining time stamp information corresponding to the first strokes included in the selected first area from the memory; and
and determining an order between the first strokes by using time stamp information corresponding to the first strokes. 15. The method of claim 14, wherein determining an order between the first strokes comprises:
determining a first stroke input first among the first strokes as a start position, and determining a last stroke input last among the first strokes as an end position Content playback method, characterized in that. The method of claim 10, wherein the content reproduction method comprises:
and reproducing the plurality of regions of the handwritten content according to time information of second strokes included in the plurality of selected regions when a plurality of regions of the displayed handwritten content are selected. Way. 11. The method of claim 10, wherein receiving the first user input comprises:
displaying a control panel including a play function button within a predetermined distance from the first area when the first area of the handwritten content is selected; and
and receiving an input for selecting the play function button through the control panel. delete 18. A computer-readable recording medium having recorded thereon a program for implementing the method of any one of claims 10 and 13 to 17.

Images