KR20230149797A - An electronic device and a method for operating an electronic device - Google Patents

Abstract

According to an embodiment of the present disclosure, in an electronic device, content including handwriting information - the handwriting information corresponds to pen data obtained by converting a handwriting trace written on a medium by an electronic pen - and image information. memory to store; output device; and a control unit, wherein the control unit reproduces the handwriting information in a first window of the output device so that the writing trace is reproduced in chronological order, and displays the image information in a second window of the output device - the first window. 2. The position of the window may be adjusted. An electronic device may be provided that is set to be reproduced on the window and automatically shifts an area in which the handwriting information is reproduced along a handwriting trace written in the medium.

Description

{An electronic device and a method for operating an electronic device}

The present invention relates to electronic devices and methods of driving electronic devices, and more specifically, to an invention for creating content including handwriting information written by an electronic pen, video information and audio information related to the user.

Recently, technologies and products that can recognize a user's handwriting using an electronic pen and digitize and store the recognized content have been commercialized. However, the development of a method to recognize and provide digitized handwriting contents to users using these electronic pens in a meaningful way is still insufficient, and methods to enable users to utilize digitized handwriting contents more effectively are not yet available. development is required.

In particular, attempts are being made to produce lecture materials consisting of videos using electronic pens, but this is still mostly done on computers that provide touch screens. Content created by handwriting on a computer conveys the fatigue of creating content on a digital device to users who are still accustomed to using paper, and it is difficult to provide this to users who want to preserve the original content created through hard copies.

The problem to be solved according to the present invention is to provide an electronic device and a method of driving the electronic device that efficiently and easily create content that a user wants to create.

According to one aspect of the present invention, there is a control method of a data recording application that records and stores content so that the writing trace created by an electronic pen is reproduced in chronological order, wherein the image of the medium on which the writing trace is written is stored in the electronic device. displaying on a display; arranging or outputting handwriting information converted from a handwriting trace written in the medium, video or audio information photographed or recorded by the electronic device, and event information of a menu selected from the menu medium according to the time sequence; A control method of a data recording application may be provided, including the step of playing at least one of a video file or audio file stored in the electronic device and a video file or audio file on the web together with the content.

According to the present invention, the creator's convenience can be increased when creating content for lectures or explanations, and the functionality of content production can be improved.

According to the present invention, after content is stored, new content can be created in addition to existing stored content, thereby improving content update.

According to the present invention, by creating content in a document containing graphic information such as text or pictures, the efficiency of content creation can be improved compared to when creating content in a blank state.

According to the present invention, the convenience and pleasure of content creation can be improved by controlling electronic devices, changing the visual of handwriting, or including exciting events when creating content.

According to the present invention, handwriting information can be edited while content production is stopped, thereby improving user convenience.

According to the present invention, information about content can be automatically recognized and extracted, and then tag information can be generated or stored for each content, thereby improving the search function.

1 is a schematic diagram illustrating a system including an electronic pen and electronic devices linked to the electronic pen related to embodiments of the present invention.
Figure 2 is a diagram showing a medium related to embodiments of the present invention.
Figure 3 is a diagram showing an exemplary appearance of an electronic pen according to embodiments of the present invention.
Figure 4 is a block diagram for explaining the components of an electronic pen according to embodiments of the present invention.
Figure 5 is a block diagram for explaining components of electronic devices according to embodiments of the present invention.
Figure 6 is a diagram showing a screen displayed on an electronic device for a data recording application according to embodiments of the present invention.
Figure 7 is a flowchart of a data recording application running in an electronic device according to an embodiment of the present invention.
Figure 8 is a screen that appears when executing a data recording application according to embodiments of the present invention.
Figure 9 is a screen of a data recording application that appears when blank medium is selected according to embodiments of the present invention.
10 and 11 are screens for connecting communication between an electronic pen and an electronic device according to embodiments of the present invention.
12 and 13 are screens showing the step of starting recording in a data recording application according to embodiments of the present invention.
14 to 17 are diagrams showing the process of creating content on Blank Medium according to embodiments of the present invention.
18 and 19 are screens that reproduce additional handwriting information according to another embodiment of the present invention.
Figure 20 is a screen for creating content in a data recording application according to another embodiment of the present invention.
Figure 21 is a diagram showing a menu medium according to embodiments of the present invention.
Figures 22 to 26 are screens explaining the progress of the data recording application after selecting the graphic medium.
Figure 27 is a diagram showing content reproduced on a graphic medium image according to embodiments of the present invention.
Figure 28 is a diagram showing a handwriting trace written on a graphic medium according to embodiments of the present invention.
Figure 29 is a diagram reproducing content written on a graphic medium according to embodiments of the present invention.
Figure 30 is a diagram showing content production according to another embodiment of the present invention.

The embodiments described in this specification are intended to clearly explain the idea of the present invention to those skilled in the art to which the present invention pertains, and the present invention is not limited to the embodiments described in this specification, and the present invention The scope of should be construed to include modifications or variations that do not depart from the spirit of the present invention.

The terms used in this specification are general terms that are currently widely used as much as possible in consideration of their function in the present invention, but this may vary depending on the intention, custom, or the emergence of new technology of a person skilled in the art in the technical field to which the present invention pertains. You can. However, if a specific term is defined and used with an arbitrary meaning, the meaning of the term will be described separately. Therefore, the terms used in this specification should be interpreted based on the actual meaning of the term and the overall content of this specification, not just the name of the term.

The drawings attached to this specification are intended to easily explain the present invention, and the shapes shown in the drawings may be exaggerated as necessary to aid understanding of the present invention, so the present invention is not limited by the drawings.

In this specification, if it is determined that a detailed description of a known configuration or function related to the present invention may obscure the gist of the present invention, the detailed description thereof will be omitted as necessary.

Electronic devices linked to the electronic pen described in this specification include mobile phones, smart phones, laptop computers, digital broadcasting terminals, PDAs (Personal Digital Assistants), PMPs (Portable Multimedia Players), navigation devices, etc. It may not only include mobile terminals, but may also include fixed terminals such as digital TVs, desktop computers, etc.

According to one aspect of the present invention, a control method of a data recording application that records and stores content so that the writing trace created by an electronic pen is reproduced in chronological order is to display an image of the medium in which the writing trace is written on the display of the electronic device. displaying steps; arranging or outputting handwriting information converted from a handwriting trace written in the medium, video or audio information photographed or recorded by the electronic device, and event information of a menu selected from the menu medium according to the time sequence; It may include playing at least one of a video file or audio file stored in the electronic device and a video file or audio file on the web together with the content.

The medium may include blank medium or graphic medium.

The graphic medium image may be displayed on the display by converting a document file stored in the electronic device into a graphic medium file.

The blank medium or the graphic medium may include a pattern code that can be detected by the electronic pen.

The graphic medium can be produced by printing the graphic media file on the blank medium.

The graphic medium image may include a mark that can be calibrated with the graphic medium.

The menu medium may include a system menu, handwriting menu, or event menu.

The menu medium includes a plurality of menus, each menu having a pattern code that can be detected by the electronic pen, and the electronic device selects one of the plurality of menus included in the menu medium. When detected, the event of the selected menu can be reproduced in the output module.

The menu medium includes a stop menu, and when the electronic device detects the occurrence of a stop event, editing of the handwriting information may proceed while reproduction of the video information or audio information is stopped.

When the electronic device detects that one of the plurality of menus included in the event menu is selected, a visual effect or sound effect may be displayed on the electronic device.

When the electronic device detects that one of the plurality of menus included in the handwriting menu is selected, the color or thickness of the stroke included in the handwriting information may be changed.

The handwriting information may be reproduced in a first window of the electronic device and the image information may be displayed in a second window of the electronic device.

The additionally played video file may be displayed in a third window of the electronic device.

The position of the second window or the third window can be adjusted in the electronic device.

The image information may be captured by either a first camera or a second camera built into the front and rear of the electronic device, respectively.

When the entire area or a partial area of the Medium image is selectively displayed on the electronic device, and only the partial area is displayed on the electronic device, the electronic device may reproduce the handwriting trace written in the Medium in the partial area. If it is determined that this is not possible, the medium image is shifted to an area other than the partial area so that the handwriting information can be reproduced.

<System configuration>

1 is a schematic diagram illustrating a system including an electronic pen and electronic devices linked to the electronic pen related to embodiments of the present invention.

The system includes at least one medium 1000, at least one menu medium 1500, at least one electronic pen 2000, at least one electronic device 3000, at least one server 4000, and a data recording application ( 5000).

The medium 1000 provides a surface for the user to write desired information using an electronic pen according to embodiments of the present invention.

The medium 1000 may be made of materials such as paper, plastic, and metal, and the material of the medium 1000 is not limited by the present invention.

If the electronic pen according to embodiments of the present invention is an electronic pen (hereinafter referred to as an optical electronic pen) implemented using optical components including a camera module, the medium 1000 is used by the electronic pen. It may have a code pattern that can be recognized. When the medium 1000 is provided with a code pattern, the code pattern can be provided to the medium 1000 through a method such as printing.

Menu Medium (1500) displays the system settings of the electronic device or data recording application (5000) and the color of the strokes included in the written content when the content written by the user on Medium (1000) is reproduced or saved on the display of the electronic device. and thickness, sound effects, or visual effects.

The menu medium 1500 may be made of a material such as paper like the medium described above, but is not limited to this and may be provided with a code pattern that can be recognized by an electronic pen. The code pattern may be provided to the menu medium 1500 through methods such as printing.

The electronic pen 2000 can detect information about the user's handwriting and generate pen data. The pen data can be used to generate stroke data for the user's handwriting, etc. The electronic device 3000 can receive pen data from the electronic pen 2000 and interpret it. Stroke data may be generated according to the interpretation of pen data.

The electronic device 3000 can appropriately process the interpreted stroke data and reproduce and display the user's handwriting in real time according to the stroke data through a display provided in the electronic device 3000 or replay and display it at a later time.

In addition, the electronic device 3000 may store pen data and/or stroke data generated according to the interpretation of the pen data in a memory provided in the electronic device 3000, and may store the pen data and/or stroke data generated according to the interpretation of the pen data in a memory provided in the electronic device 3000. Alternatively, appropriate processing may be performed on the stroke data.

The electronic device 3000 is connected to the server 4000 through a network and can transmit or receive necessary information through the network.

The server 4000 is connected to the electronic device 3000 through a network and can receive or transmit necessary information.

The server 4000 is connected to a plurality of electronic devices 3000 and can form a database of pen data (or stroke data) received from the electronic devices 3000.

The server 4000 may appropriately process pen data (or stroke data) stored in a database according to requests from a plurality of electronic devices 3000 or requests from users.

The server 4000 may share and perform some or all of the functions of the electronic device 3000 described above. For example, the electronic device 3000 may receive pen data from the electronic pen 2000 and directly transmit the pen data to the server 4000 without performing a special processing procedure for the pen data. In this case, , the server 4000 may perform operations such as receiving pen data and generating stroke data, storing stroke data, and additional processing of stroke data, as described above.

The data recording application 5000 is an authoring tool that creates lecture materials using an electronic pen or materials to explain content to viewers.

The data recording application 5000 can reproduce, display, or store the user's image information and audio information captured by an electronic device along with handwriting information such as text or pictures (handwriting traces) written by the user on Medium. Additionally, the data recording application 5000 can output other video information or audio information stored in the electronic device and reproduce, play, or store it together with the handwriting information, as needed.

In Figure 1, one medium (1000-1) is shown as corresponding to one electronic pen (2000-1), but the correspondence relationship between the medium (1000) and the electronic pen (2000) is 1:N or N: It is okay to correspond according to the correspondence relationship of 1.

In addition, in Figure 1, one electronic pen (2000-1) is shown as corresponding to one electronic device (3000-1), but the correspondence relationship between the electronic pen (2000) and the electronic device (3000) is It is okay to correspond according to the correspondence relationship of 1:N or N:1.

In addition, although FIG. 1 shows a plurality of electronic devices 30000 connected to one server 4000, systems according to embodiments of the present invention may include two or more servers.

Meanwhile, in FIG. 1, the electronic pen 2000 is shown communicating with the electronic device 3000, and the electronic device 3000 is shown communicating with the electronic pen 2000 and the server 4000. However, if necessary, the electronic pen 2000 may directly communicate with the server 4000.

<Medium>

Next, the medium 1000 related to embodiments of the present invention will be described.

Figure 2 is a diagram showing a medium related to embodiments of the present invention.

Referring to FIG. 2, medium 1000 can be divided into blank medium 1200 and graphic medium 1400. Blank medium (1200) may refer to a medium consisting mostly of blank space for writing, such as a notebook. As shown at the top of FIG. 2, the blank medium 1200 may be a medium composed of multiple sheets, such as a notebook, or it may be a medium composed of a single sheet. In the blank medium 1200 composed of a plurality of chapters, each chapter may be referred to as a sub-medium. When the electronic pen 1000 is implemented as an optical electronic pen, code patterns according to embodiments of the present invention may be provided on at least some or all of the pages constituting the note.

Meanwhile, only at least a portion of the entire surface of the page constituting the note may be implemented as medium 1000 according to embodiments of the present invention. For example, if the electronic pen 1000 is implemented as an optical electronic pen, the code pattern may not be provided on the entire surface of the page constituting the note, but only in some areas. You can.

The graphic medium 1400 may be a medium that includes specific graphics consisting of text or pictures, as shown at the bottom of FIG. 2. Graphic medium 1400 can be created by printing a document file desired by the user on blank medium 1200. The graphic medium 1400 may also be a singular medium and may include a plurality of sub-mediums. If the user prints multiple document files, the graphic medium 1400 may be composed of multiple sub-mediums. A coat pattern may also be provided on the surface of the graphic medium 1400. Graphic Medium (1400) will be described in detail later.

Meanwhile, although not shown in the drawings, the medium 1000 according to embodiments of the present invention may be implemented in the form of a blackboard or the like. That is, the medium 1000 according to embodiments of the present invention can be implemented in the form of a blackboard (electronic blackboard) provided with a code pattern.

<Electronic pen>

Next, an electronic pen according to embodiments of the present invention will be described.

FIG. 3 is a diagram illustrating an exemplary appearance of an electronic pen according to embodiments of the present invention, and FIG. 4 is a block diagram for explaining components of an electronic pen according to embodiments of the present invention.

Referring to FIG. 3, the electronic pen 2000 largely includes a main body portion 2020 that fixes the position of the core portion 2010 (see FIG. 42) that enables writing on the medium 1000, and the core portion 2010 when not in use. ) may include a cap portion 2030 that protects the cap portion 2030, etc.

Here, the non-use state may mean a state in which the electronic pen 2000 according to an embodiment of the present invention is being stored. In this case, the core portion 2010 is not exposed to the outside, so writing on the medium is impossible. It can mean.

The core portion 2010 is a component that enables writing on the medium 1000 and may include a ball and ink, and the ink may be oil-based, water-based, or neutral.

Additionally, the core portion 2010 may be a black core used in a pencil or a lead used in a general writing instrument.

The main body portion 2020 is a component that is inserted so that a portion of the core portion 2010 is exposed to fix the position of the core portion 2010, and may refer to a type of grip portion that a user can grip with his or her hand.

The cap portion 2030 may be inserted into one side of the main body 2020 when not in use and may have a predetermined internal space to protect the core portion 2010, and may be inserted into the other side of the main body 2020. If possible, a portion of the core portion 2010 may be exposed to the outside to enable writing.

However, when writing is enabled by the core part 2010, the cap part 2030 does not necessarily need to be inserted into the other side of the main body 2020, and is separated from the main body 2020 to leave a predetermined space. It is okay to place it in .

Referring to FIG. 4, the electronic pen 2000 according to an embodiment of the present invention includes an input unit 2100, an output unit 2200, a pattern detection unit 2300, a communication unit 2400, a sensor unit 2500, It may include a memory 2600, a power unit 2700, and a control unit 2800. However, since the components shown in FIG. 4 are not essential, an electronic pen may be implemented that has more components than the illustrated components or fewer components than the illustrated components.

Hereinafter, each component of the electronic pen 2000 will be looked at in turn.

The input unit 2100 may receive user input from the user. User input can take various forms, including button input, key input, touch input, and voice input. For example, the electronic pen 2000 may receive a button input from the user and control the power on/off of the electronic pen 2000.

Examples of the input unit 2100 include traditional buttons, jog wheels, etc., as well as a touch sensor that detects the user's touch, a microphone that receives voice signals, a camera that recognizes gestures through image recognition, and a device that detects user access. A comprehensive range of input means that includes a proximity sensor consisting of an illumination sensor or an infrared sensor, a motion sensor that recognizes user movements through an acceleration sensor or a gyro sensor, and various other types of input means that detect or receive user input. It is a concept.

The output unit 2200 can output various information and provide it to the user. This output unit 2200 includes a display that outputs visible light that can be recognized through the user's vision, a speaker that outputs sound that can be recognized through the user's hearing, and a device that generates vibration that can be recognized through the user's sense of touch. It should be interpreted as a comprehensive concept that includes both haptic devices and various other types of output means. For example, the electronic pen 2000 is equipped with a buzzer ( A sound with predetermined characteristics can be output through a buzzer, etc. For another example, in order to provide the user with information about the operating state of the electronic pen 2000, predetermined visible light may be output through a display such as an LED lamp.

The pattern detection unit 2300 is configured to detect a code pattern in which position information for determining the position of the electronic pen 2000 with respect to the medium 1000 is encoded, and includes a camera module 2320 and a lighting module ( 2340).

The camera module 2320 is used to obtain an image of the code pattern provided in the medium 1000. The camera module 2320 moves the electronic pen 2000 on the medium 1000 according to the user's handwriting, at a predetermined time interval (for example, 1/60 second to 1/240 second, etc.). An image of a partial area of the code pattern provided in the medium 1000 is acquired. Accordingly, the electronic pen 1000 analyzes the information about the unit cell included in the acquired image according to the code pattern interpretation method described later, and determines the position of the electronic pen 2000 with respect to the medium 1000. can be interpreted in coordinate form.

The lighting module 2340 irradiates light of a specific predetermined frequency toward the medium 1000 so that an image of the code pattern can be obtained through the camera module 2320. The code pattern provided in the medium 1000 according to an embodiment of the present invention may be printed using ink that absorbs infrared rays. In this case, the lighting module 2340 may be well absorbed by the ink. Light of different wavelengths may be irradiated to the medium (1000). At this time, the camera module 2320 may be configured to capture images in the infrared band, and for this purpose, the camera module 2320 may be equipped with an infrared filter that selectively passes only the infrared band.

The communication unit 2400 can communicate with external devices. For example, the electronic pen 2000 may communicate with the electronic device 3000 or the server 4000 through the communication unit 2400. Accordingly, the electronic pen 2000 can transmit and receive various information with external devices. For example, the electronic pen 2000 can transmit pen data generated by the electronic pen 2000 to the electronic device 3000 or the server 4000 through the communication unit 2400, and the electronic pen 2000 ) Commands and/or data for controlling the operation of the electronic pen 2000 or providing necessary information to the electronic pen 2000 may be received from the electronic device 3000 or the server 4000 through the communication unit 2400. .

The communication unit 2400 may include a wired module 2420 and a wireless module 2440.

The wired module 2420 is a module for transmitting and receiving data to and from an external device by wire, and may include components such as a USB port, PS/2 port, parallel port, and serial port that can transmit and receive data to and from an external device. , Furthermore, it may include an Ethernet port for accessing the Internet, etc. through a LAN (Local Area Network).

The wireless module 2440 is a mobile communication module that transmits and receives data by connecting to a mobile communication network via a mobile communication base station, a WLAN (Wireless Local Area Network) type communication method such as Wi-Fi, Bluetooth, It may include a short-distance communication module that uses a WPAN (Wireless Personal Area Network) communication method such as Zigbee.

The sensor unit 2500 can sense various information. Here, various information can be used to generate control signals for controlling the operation of the electronic pen 2000 and can be used to generate pen data generated by the electronic pen 2000. Various information sensed through the sensor unit 2500 may be provided to the electronic device 3000 or the server 4000.

The sensor unit 2500 may include a pressure sensor 2520, a pen core sensor 2540, a slip sensor 2560, etc.

The pressure sensor 2520 can sense the pressure generated when the user writes. For example, when a user writes using the electronic pen 2000, the electronic pen 2000 comes into contact with the writing surface, and the pressure generated at this time is sensed by the pressure sensor 2520. You can. In particular, the pressure sensor 2520 is provided at the rear end of the pen core (core 2010) mounted on the electronic pen 2000 (an end other than the pen tip among both ends of the pen core) to sense pressure. The electronic pen 2000 can use the pressure value sensed through the pressure sensor 2520 to generate pen data, which will be described in detail later. At this time, the pressure value included in the pen data can be processed into pen pressure information applied by the user to the medium 1000 when writing and used to reproduce the content written by the user. Alternatively, based on the signal output through the pressure sensor 2520, the electronic pen 2000 in the off state may be used as a trigger signal to turn on the electronic pen 2000.

The pen nib sensor 2540 can detect a pen nib identifier to indicate the type of pen nib (core, 2010) mounted on the electronic pen 2000. For example, a pen nib identifier to indicate the color of the pen nib (core 2010) may be provided on the outside of the nib (core 2010) mounted on the electronic pen 2000, and in this case, the pen nib sensor 2540 The pen nib identifier can be detected through optical/magnetic methods and other various available methods. The value sensed by the pen nib sensor 2540 or information about the pen nib (simbu, 2010) interpreted by the sensed value may be provided to the electronic device 3000, and the electronic device 3000 may provide the received information It can be used to reproduce content written by the user based on.

The sleep sensor 2560 can detect the state of the electronic pen 2000, which is used to determine whether the electronic pen 2000 in the on state will automatically switch to the off state. For example, the slip sensor 2560 may include an acceleration sensor, and whether the electronic pen 2000 is moving can be detected through the acceleration sensor. If it is determined that the electronic pen 2000 is not moving for a predetermined period of time based on the value sensed through the slip sensor 2560, the electronic pen 2000 automatically moves the electronic pen 2000. The state can be switched to the off state.

According to some embodiments of the present invention, when the electronic pen 2000 is in the on state, power is normally supplied to the components provided in the electronic pen 2000 and the components operate normally as needed. However, when the electronic pen 2000 is in an off state, power to all or some components provided in the electronic pen 2000 is cut off and the electronic pen 2000 is not equipped with the electronic pen 2000. All or some of the components cannot operate normally. In particular, according to some embodiments of the present invention, even when the electronic pen 2000 is in an off state, the pressure sensor 2520 and/or an analysis module for interpreting the signal output from the pressure sensor normally operates. Power can be supplied, and accordingly, the electronic pen 2000 in the off state can be automatically switched to the on state based on the sensing value output from the pressure sensor 2520.

The memory 2600 can store various types of information. The memory 2600 can store data temporarily or semi-permanently. The memory 2600 may store operating programs such as firmware for driving the electronic pen 2000, and, if necessary, pen data generated by the electronic pen 2000 and the sensor unit 2500. Information, commands, and data necessary for operating the electronic pen 2000, such as sensing values output by , may be stored.

Examples of memory 2600 include hard disk drive (HDD), solid state drive (SSD), flash memory, read-only memory (ROM), random access memory (RAM), etc. This can be.

The power supply unit 2700 can provide power necessary for the operation of each component of the electronic pen 2000. The power unit 2700 may be implemented as a rechargeable battery, and if the communication unit 2400 of the electronic pen 2000 includes a charging port (for example, a USB port), the electronic pen ( When 2000 is connected to the electronic device 3000, power can be supplied from the electronic device 3000, and the battery can be charged accordingly.

The control unit 2800 controls the overall operation of the electronic pen 2000. To this end, the control unit 2800 can perform computation and processing of various information and control the operation of components of the electronic pen 2000.

In addition, the control unit 2800 may analyze the image for the code pattern obtained by the pattern detection unit 2300 and decode the information encoded in the unit cell included in the image of the obtained code pattern. Furthermore, the location coordinates of the medium 1000 of the electronic pen 2000 and identification information (for example, book information or page information, etc.) of the medium 1000 can be obtained from the decoded information.

The control unit 2800 may generate pen data based on information obtained by decoding information encoded in a unit cell. As described above, the generated pen data is stored in the memory 2600 or the It can be transmitted to the electronic device 3000 and/or the server 4000 through the communication unit 2400. A detailed explanation of the creation of pen data will be provided later.

Meanwhile, the control unit 2800 analyzes the image acquired by the first control unit and the pattern detection unit 2300 for controlling the general operation of the electronic pen 2000, decodes the necessary information for the unit cell, and It may be divided into a second control unit for generating pen data. The control unit 2800 may be logically divided into the first control unit and the second control unit, but the first control unit and the second control unit may be physically divided into the first control unit and the second control unit. It can also be implemented in a form that is separated from each other.

The control unit 2800 may be implemented as a computer or similar device based on hardware, software, or a combination thereof. A hardware-type control unit may be provided in the form of an electronic circuit that processes electrical signals to perform a control function, and a software-type control unit may be provided in the form of a program that drives the hardware-type control unit.

Meanwhile, unless otherwise specified in the following description, the operation of the electronic pen 2000 may be interpreted as being performed under the control of the control unit 2800.

<Electronic devices>

Next, an electronic device 3000 according to embodiments of the present invention will be described.

Figure 5 is a block diagram for explaining components of electronic devices according to embodiments of the present invention.

Referring to FIG. 5 , the electronic device 3000 may include an input module 3100, an output module 3200, a communication module 3300, a memory module 3400, and a control module 3500. However, since the components shown in FIG. 5 are not essential, an electronic device may be implemented that has more components than the illustrated components or fewer components than the illustrated components.

Hereinafter, each component of the electronic device 3000 will be looked at in turn.

The input module 3100 can receive user input from the user. User input can take various forms, including key input, touch input, and voice input. Examples of the input module 3100 that can receive such user input include a traditional keypad, keyboard, and mouse, as well as a touch sensor that detects the user's touch, a microphone that receives voice signals, and recognition of gestures through image recognition. a camera, a proximity sensor consisting of an illumination sensor or an infrared sensor that detects the user's approach, a motion sensor that recognizes the user's movements through an acceleration sensor or a gyro sensor, and various other forms of detecting or receiving user input. It is a comprehensive concept that includes all input means. Here, the touch sensor may be implemented as a piezoelectric or capacitive touch sensor that detects touch through a touch panel or touch film attached to the display panel, or an optical touch sensor that detects touch by an optical method.

The output module 3200 can output various information and provide it to the user. This output module 3200 includes a display that outputs visible light that can be recognized through the user's vision, a speaker that outputs sound that can be recognized through the user's hearing, and a device that generates vibration that can be recognized through the user's sense of touch. It is a comprehensive concept that includes both haptic devices and various other types of output means.

The display may output an image to reproduce content written by a user using the electronic pen 2000 and/or the electronic device 3000 according to an embodiment of the present invention.

In addition, the display is a liquid crystal display (LCD),

Light emitting diode (LED) display, Organic Light Emitting Diode (OLED) display, Flat Panel Display (FPD), transparent display, Curved Display, flexible display A concept that refers to a wide range of image display devices, including flexible displays, 3D displays, holographic displays, projectors, and other various types of devices that can perform image output functions. am. This display may be in the form of a touch display integrated with the touch sensor of the input module 3100.

The communication module 3300 can communicate with external devices. For example, the electronic device 3000 may communicate with the electronic pen 2000 or the server 4000 through the communication module 3300. Accordingly, the electronic device 3000 can transmit and receive various information with external devices. For example, the electronic device 3000 can receive pen data generated by the electronic pen 2000 through the communication module 3300 and transmit necessary information to the electronic pen 2000. In addition, the electronic device 3000 can transmit pen data received from the electronic pen 2000 or stroke data obtained by analysis of the pen data to the server 4000 through the communication module 3300, and provide the necessary information. can be received from the server 4000.

The communication module 3300 is a wired communication module including a LAN port, USB port, PS/2 port, parallel port, serial port, etc. that connects to the Internet, etc. through a LAN (Local Area Network), and mobile communication via a mobile communication base station. A mobile communication module that connects to a network to transmit and receive data, a WLAN (Wireless Local Area Network) type communication method such as Wi-Fi, or a WPAN (Wireless Personal Area Network) type such as Bluetooth or Zigbee. It may be composed of a short-range communication module using a communication method, a satellite communication module using a GNSS (Global Navigation Satellite System) such as a GPS (Global Positioning System), or a combination thereof.

The memory module 3400 can store various types of information. The memory module 3400 can store data temporarily or semi-permanently. The memory module 3400 may store an operating program (OS; Operating system) for driving the electronic device 3000, and, if necessary, various data received from the electronic pen 2000 and the electronic pen 2000. Applications for processing various data received from may be stored.

Examples of the memory module 3400 include hard disk drive (HDD), solid state drive (SSD), flash memory, read-only memory (ROM), and random access memory (RAM). There may be etc.

The control module 3500 controls the overall operation of the electronic device 3000. To this end, the control module 3500 can perform computation and processing of various information and control the operation of components of the electronic device 3000.

The control module 3500 may be implemented as a computer or similar device based on hardware, software, or a combination thereof. A hardware-type control module may be provided in the form of an electronic circuit that processes electrical signals to perform a control function, and a software-type control module may be provided in the form of a program that drives the hardware-type control module.

Meanwhile, unless otherwise specified in the following description, the operation of the electronic device 3000 may be interpreted as being performed under the control of the control module 3500.

<Description of operation of electronic devices>

Figure 6 is a diagram showing a screen displayed on an electronic device for a data recording application according to embodiments of the present invention.

Referring to FIG. 6, the data recording application 5000 according to embodiments of the present invention is an authoring tool for creating lecture materials using the electronic pen 2000 or materials for explaining content to viewers.

Data recording application 5000 may be software. Therefore, it can be installed in the form of an app on the electronic device 3000 and operated within the electronic device 3000. The data recording application can be stored in a recording medium in the form of software and installed and operated on an electronic device. The data recording application is stored on a specific server and can be downloaded by anyone who wants to use it.

The data recording application 5000 may include program codes to perform functions specific to the electronic device 3000. The data recording application 5000 may be stored in a memory module, and when the data recording application is executed, necessary functions may be provided to the electronic device 3000 by the control module 3500 or the operating system of the electronic device 3000. Therefore, the functions provided by the data recording application 5000 communicate with the input module 3100, output module 3200, communication module 3300, memory module 3400, and control module 3500 of the electronic device 3000. It can be implemented on the electronic device 3000.

The data recording application 5000 and the electronic device 3000 are separate components corresponding to software and hardware, but since the data recording application 5000 runs within the electronic device 3000, strict distinction between them is not necessary. Contents described according to embodiments of the present invention may apply to both the data recording application 5000 and the electronic device 3000. In other words, even if the description of the present invention is based on the data recording application 5000, the data recording application 5000 runs on the electronic device 3000, so it can be viewed as a description of the electronic device 3000. Therefore, the following description will focus on the data recording application 5000, but it is assumed that it is implemented in the electronic device 3000.

The data recording application 5000 reproduces the user's handwriting traces such as writing or drawings on Medium 1000 and the user's video information or audio information captured or recorded by the electronic device 3000 in the output module of the electronic device. It can be done and stored in the memory module. In this specification, when the handwriting trace, such as text or a picture written by a user on Medium, is reproduced on the output module of an electronic device, it is expressed as handwriting information.

Additionally, the data recording application 5000 can output other video information or audio information stored in the electronic device 3000 and reproduce, play, or store it together with the handwriting information, as needed.

The data recording application 5000 may provide an option to select either the blank medium 1200 or the graphic medium 1400, and display an image of the selected medium on the display of the electronic device 3000.

The data recording application 5000 converts the document file stored in the electronic device 3000 into a graphic medium (1400) file so that it can be used in the data recording application 5000, and allows the user to output it as a graphic medium (1400). provides. At this time, the graphic medium file can be produced as a graphic medium (1400) by printing it on a blank medium (1200) on which a pattern code is printed.

The electronic pen 2000 can acquire an image of at least a partial area of the code pattern provided in the medium 1000 and obtain location information by interpreting the information encoded in the unit cell included in the acquired image. there is.

While the user is writing, the electronic pen (2000) continuously acquires and processes/stores images of code patterns and location information interpreted therefrom, thereby enabling the reproduction of the corresponding handwriting information on the data recording application (5000). there is. The handwriting trace written on the medium by the electronic pen 2000 is digitized and processed into pen data, and the pen data can be reproduced as handwriting information on the electronic device 3000 running the data recording application 5000.

Hereinafter, how the data recording application 5000 is driven within the electronic device 3000 will be described in detail.

Figure 7 is a flowchart of a data recording application running in an electronic device according to an embodiment of the present invention.

Referring to FIG. 7, the steps of executing the data recording application 5000, selecting a medium, outputting the medium image to the display, connecting communication between the electronic pen 2000 and the electronic device 3000, It includes steps of starting to record content, creating content, and ending recording and saving the content.

The step of selecting medium (1000) can be divided into selecting blank medium (1200) and selecting graphic medium (1400). When blank medium 1200 is selected, a blank medium image may be output on the display of the electronic device 3000. When selecting Graphic Medium (1400), the following steps include selecting a document file, creating a Graphic Medium (1400) file, sharing or printing a Graphic Medium (1400) file, and displaying a Graphic Medium (1400) image. After further steps such as printing on the screen and calibrating the graphic medium 1400, the user can proceed to pairing between the electronic pen 2000 and the electronic device 3000.

Here, content is used to mean the sum of all information that can be output to the data recording application 5000. Handwriting information written on the medium 1000 by the electronic pen 2000 and reproduced on the display of the electronic device 3000, video information and audio information photographed or recorded by the input module 3100 of the electronic device 3000 , event sound effects, event visual effects generated by the menu selected in the menu medium 1500, video information and audio information that appear when a video file or audio file stored in the electronic device 3000 is executed and displayed on the electronic device, This may refer to video information, audio information, etc. displayed on the output module 3200 of the electronic device 3000 when video files and audio files linked by a web address are executed. In a narrow sense, it may refer to all information and records appearing on the display after recording begins in the data recording application 5000. The term recording does not only mean recording/storing video information, but can also mean recording/storing audio information. Therefore, the word recording can also mean recording.

Figure 8 is a screen that appears when executing a data recording application according to embodiments of the present invention.

Referring to FIG. 8 along with FIG. 7, the step of selecting a medium after executing the data recording application 5000 will be described. The data recording application 5000 may guide the user to select the medium 1000. The user selects a blank medium (1200) on the display, which is the output module (3200) of the electronic device (3000) (“Use a blank notebook in the drawing”) or selects a document file stored in the electronic device (3000) (“Use my notebook in the drawing”). You can create a graphic medium (1400) file (own material).

Blank medium and graphic medium, which are types of medium, have been described previously, so detailed explanations will be omitted here.

Below, the operation of the data recording application when the blank medium 1200 is selected will first be described, and then the steps for selecting the graphic medium 1400 will be described later.

Figure 9 is a screen of a data recording application that appears when blank medium is selected according to embodiments of the present invention.

Referring to FIG. 9 , when blank medium 1200 is selected, blank medium image 1210 is output on the display of the electronic device 3000. The image displayed on the Blank Medium actual product and data recording application may differ. In the data recording application 5000, the blank medium image 1210 is set to black, but the color of the image may be any color, including white. Since the blank medium 1200 does not contain any content elements such as letters or pictures, when the blank medium 1200 is imaged on the data recording application 5000, it may be an image composed of blank space.

10 and 11 are screens for connecting communication between an electronic pen and an electronic device according to embodiments of the present invention.

Referring to FIGS. 10 and 11 , the data recording application 5000 may create a screen 1220 requesting connection with the electronic pen 2000 on the display of the electronic device 3000. A channel for communication between the electronic pen 2000 and the electronic device 3000 is connected. A communication channel is secured between the communication unit 2400 of the electronic pen 2000 and the communication module 3300 of the electronic device 3000 to exchange data using a wired or wireless communication method, and this is referred to as pairing. can do. In the case of pairing, a Bluetooth wireless communication method can be adopted, but it is not limited to this, and any wired or wireless method is suitable.

Accordingly, when the user proceeds to pair the electronic pen 2000 and the electronic device 3000 and the pairing is completed, a pairing completion indicator 1230 may appear.

If the electronic pen 2000 and the electronic device 3000 are paired, the data recording application 5000 can reproduce or display handwriting information and the user's image information through the electronic pen 2000 even if recording does not start. You can.

12 and 13 are screens showing the step of starting recording in a data recording application according to embodiments of the present invention.

Referring to FIGS. 12 and 13 , the data recording application 5000 may execute the record button 1240 to start reproducing and recording content. When recording begins, the red indicator on the record button 1240 changes to blue, indicating that recording is starting. Starting and ending recording can also be performed through the menu medium 1500, which will be described later.

Below, the process of reproducing/playing/recording handwriting information written with the electronic pen (2000) and the user's video/audio information will be described.

14 to 17 are diagrams showing the process of creating content on Blank Medium according to embodiments of the present invention.

Referring to Figures 14 to 17, the woman in the drawing is creating content using Blank Medium.

Handwriting contents such as pictures and writing written on the medium 1000 (can be referred to as a writing trace) may be stored as pen data in the memory unit built into the electronic pen 2000. Pen data may refer to information used to convert the user's handwriting trace into stroke data on the electronic device 3000. The handwriting trace written on Medium can be digitized and converted into first fan data. Stroke data may be information for the electronic device 3000 to receive pen data from the electronic pen 2000 and to indicate a writing trace written in medium on the electronic device 3000. The electronic device 3000 receives pen data from the electronic pen 2000, interprets it, and then generates stroke data. Handwriting information can be defined as the minimum unit of stroke data or the sum of stroke data of more than the minimum unit, and corresponds to the type of writing trace that appears on the medium being implemented on the display of the electronic device 3000.

That is, the writing trace on the medium is stored as pen data in the electronic pen, and the pen data can be received and interpreted by the electronic device and then converted into stroke data. When stroke data is reproduced and displayed on an electronic device, it may be referred to as handwriting information 1260.

The handwriting trace, such as a picture or text written on the medium 1000, can be reproduced in real time as handwriting information 1260 on the medium image 1210 of the data recording application. Figures 14 to 17 show that the bear's drawing and the handwriting traces of the letters are reproduced as handwriting information according to the order of the drawings.

In addition, video information 1270 and audio information (not shown) of the user taking notes may be photographed or recorded through a camera or microphone built into the electronic device 3000 and displayed on the display. The user's video information 1270 and audio information may be stored in the memory module 3400 in the electronic device 3000.

The electronic device 3000 may include a first camera and a second camera on the front and rear sides, respectively, and image information 1270 may be captured by the first camera or the second camera. When you want to take a picture of the user entering handwriting information (1260), you can use the first camera where the display is located, and when you do not want to take a picture of the user, you can use the second camera to capture the background or objects located on the back. You can shoot. The shooting function can also be turned off through the menu provided by the data recording application 5000 or the menu medium 1500.

The area where the handwriting information 1260 is reproduced on the display of the electronic device 3000 may be referred to as the first window 1310, and the area where the user's appearance is displayed may be referred to as the second window 1320. The first window may be a larger area including the area of the second window. The position of the second window can be adjusted by the user's touch and drag within the first window. In the drawing, an example is shown where the second window is located at the bottom right of the display, but the second window can be located anywhere within the first window.

Referring to Figures 15 and 17, it can be seen that in the picture shaped like a bear on the display, the position of the bear's head has changed from the lower part of the display to the upper part of the display.

This is because the blank medium image 1210 provided by the data recording application 5000 does not display the entire surface of the blank medium 1200 on which the writing trace is currently being written, but only a partial area is enlarged and displayed.

Therefore, when the writing trace written on the blank medium 1200 deviates from a certain area and is written in another area, the data recording application 5000 creates the blank medium image 1210 in the first window where the writing information is reproduced based on the pen data. ) can be shifted to correspond to another area of the blank medium 1200.

To explain in more detail, when the medium is divided into a first area and a second area distinct from the first area, the pen data generated in each area may have different location information regarding the writing trajectory of each area. there is.

The electronic device 3000 reads the area where the writing trace is written based on pen data including the location information of the medium, and determines the area where the handwriting information is reproduced on the electronic device 3000 according to the writing trace written on the medium. It can shift automatically.

Additionally, the user can change or enlarge/zoom out the area of the blank medium (1200) image by touching and dragging the display.

If the user creates a writing trace while displaying the entire surface of the blank medium 1200 rather than using it with the display screen enlarged, a blank medium image displaying the entire surface can be output on the display. there is. In this case, even if the blank medium image is not shifted on the display according to each area of the blank medium 1200, all writing traces written on the entire surface of the blank medium 1200 can be reproduced.

When the creation of content including handwriting information, video information, and audio information is completed, the user terminates the recording again, so that the information reproduced, played, or recorded so far can be stored in the memory module 3400 of the electronic device 3000. there is.

18 and 19 are screens that reproduce additional handwriting information according to another embodiment of the present invention.

Referring to Figures 18 and 19, the user can input new handwriting information while all already saved handwriting information is maintained in the blank medium image. The content reproduced and stored in the data recording application 5000 is a video (including audio) file recorded over time. Therefore, new handwriting information cannot be input into content that has already been stored as a video. The technology of creating new text on video through video editing is a separate topic.

Therefore, if the content is saved in a state where it is not yet completed, it is not easy to create the content even if there is content that you want to create in addition to the handwriting information reproduced and recorded before storage.

In preparation for this case, the data recording application 5000 can create a separate file that retains only the written content, even if one content is saved and created as a multimedia file including video and audio information.

When defining the handwriting information included in the already stored content as the first handwriting information and the video or audio information included in the already stored content as the first video or audio information, the user can select the file in which only the first handwriting information 1260 is stored. The second handwriting information 1280 can be reproduced again by printing it again on the electronic device. In this case, the second handwriting information may reuse the medium previously used to create the first handwriting information, or a new medium may be used. At this time, the fan data corresponding to the second handwriting information may be defined as the second pen data.

Therefore, the user can create different content each time through the data recording application 5000, but can also create updated content to which handwriting information is added based on existing content. When updating content, since the content is produced based on time, the first video or audio information included in the previously created content is not included in the updated content.

The data recording application 5000 can recognize information included in content. The data recording application 5000 may recognize handwriting information, video information, and audio information included in the content, and automatically generate one or more tag information included in the content when the content is stored in the electronic device 3000.

For example, the data recording application 5000 may include a text recognition function and an audio recognition function. When content is stored, the data recording application 5000 can recognize handwriting information and audio information included in the stored content, extract the most frequently used words, etc., and attach tag information to the content.

Only one tag can be created, but multiple tags can also be created.

Tags generated in content, along with the title of the content, can later be used to classify, sort, or find content by type.

Figure 20 is a screen for creating content in a data recording application according to another embodiment of the present invention.

Referring to FIG. 20, when reproducing the handwritten information of the data recording application 5000, the user can create content using a tool that can add aesthetics to the handwritten information. In this case, when stroke data, which is the basis of handwriting information, is expressed as handwriting information, the color and thickness of individual strokes are changed, so that handwriting information with added aesthetics can be reproduced.

The color and thickness of the handwriting information, and more precisely, the color and thickness of the strokes constituting the handwriting information, can be reproduced by reproducing the event information of the menu selected in the menu medium 1500 on the electronic device 3000.

Figure 21 is a diagram showing a menu medium according to embodiments of the present invention.

* Referring to FIG. 21, the menu medium 1500 may largely include a system menu 1510, a handwriting menu 1520, and an event menu 1530. When performing a necessary function while running the data recording application 5000, when the electronic pen 2000 selects a specific menu in the menu medium 1500, the effect of the selected menu is transmitted to the output module 3200 of the electronic device 3000. It can be delivered and performed on the display.

Like the medium described above, the menu medium 1500 may be made of a specific material with a code pattern printed on it, such as paper, plastic, or metal, but may also be coded within the data recording application 5000 and implemented in the electronic device 3000. When the menu medium 1500 is coded in the data recording application 5000 and implemented in the electronic device 3000, the user can select a specific menu on the display of the electronic device 3000 when creating content. When the menu medium 1500 is composed of a recording medium other than the electronic device 3000, such as paper, the user selects a menu included in the menu medium 1500 using the electronic pen 2000, so that the menu effect is performed on the display. You can.

The menu medium 1500 includes pattern codes that can be detected by the electronic pen 2000 for each menu. At this time, the method for the electronic pen 2000 to detect the menu configured in the menu medium 1500 is for the electronic pen 2000 to recognize a code pattern on the medium to generate pen data (handwriting information implemented on the display of the electronic device). Since it is the same as the method, the explanation for this will be omitted.

The communication module of the electronic device 3000 can detect that a menu included in the menu medium 1500 has been selected by the electronic pen 2000. An event may be defined as occurring when each menu included in the menu medium 1500 is selected by the electronic pen 2000. The event may be output by the output module 3200, a display or speaker.

Hereinafter, detailed menus included in the menu medium 1500 will be described in detail. In the embodiment of the present invention, each menu is named in order to explain the menus, but each menu may be named with a name different from the name described here.

The system menu may provide functions for starting and ending recording, screen configuration, and information editing when running the data recording application 5000. The system menu is Record menu, Finish menu, Screen menu, Camera menu, Zoom-in menu, Zoom-out menu, Fit menu, and Full menu. It may include a (Full) menu, Undo menu, Redo menu, or Erase all menu.

The Record menu may indicate the start of saving content including handwriting information, video information, and audio information.

The Finish menu can instruct to end production of content and save it when content production is completed.

The Screen menu is used when there is a display (first window) that displays handwriting information and a display (second window) that displays the user's video information, in a display state where the first window and the second window are displayed together. You can instruct a change to a display state in which only the second window is displayed, or a change to the opposite display state.

The Camera menu can instruct to turn on/off the display (second window) that displays the user's image information.

The Zoom-in menu can instruct the medium image, which is the area displaying the handwriting information, to be enlarged or increased in magnification on the display to make the handwriting information appear larger.

The zoom-out menu can instruct the medium image, which is the area displaying the handwriting information, to be reduced or magnified on the display to make the handwriting information appear smaller.

The Fit menu can instruct the medium image to be enlarged in the width direction.

The Full menu can direct all medium images to be displayed on the display. If a medium consists of multiple sub-mediums, the user can instruct to display all areas of the sub-medium being created.

The Undo menu can be used to instruct the user to delete the reproduction of the handwritten information that was just previously performed.

The Redo menu can instruct you to reproduce handwriting information that was immediately deleted.

The Erase all menu can be instructed to erase all handwriting information reproduced on the display.

The system menu may include a Pause (not shown) menu. The stop menu stops recording video or audio information about the user during content reproduction and activates only the display (first window) where the handwriting information is reproduced, so that the reproduced handwriting information can be modified, deleted, or added until the stop is in progress. An editing function can be provided to add handwritten information.

The production of content can occur over time. Modification and editing of content can be done using Menu Medium's undo, redo, erase all functions, but this is also a process that occurs over time.

Therefore, when you need to spend time editing handwritten information while creating content, when you need to run other multimedia files in a data recording application without interruption in time, or when you need to proceed with other tasks on an electronic device, you can use the stop menu. You can use it to edit content smoothly or proceed with content creation without interruption.

The handwriting menu may provide a visual effect to the handwriting information reproduced on the display of the electronic device 3000 when the user writes handwriting information in Medium. The handwriting menu may include a Pen menu, Highlighter menu, Stroke Size menu, or color (Red, Yellow, Green, Light Blue, Black, White). Menus included in the handwriting menu can change the color or thickness of strokes included in the handwriting information.

The Pen menu represents the initially set value to appear on the display of the electronic device (3000) when the electronic pen (2000) inputs handwriting information into Medium, and can usually instruct writing or pictures to be reproduced with thin lines. there is.

The Highlighter menu allows you to highlight specific areas in Medium or instruct text or pictures to be reproduced with thick lines. The highlighter menu allows the user to instruct the line thickness to vary when text or pictures are reproduced by selecting the Stroke Size menu. The stroke size you can select from the highlighter menu may be thicker than the stroke size when you select the pen menu.

The color (Red, Yellow, Green, Light Blue, Black, White) menu can be used to instruct the user to change the color of the text or picture when the handwritten information is reproduced on the display. The color menu can be applied to both the pen menu and highlighter menu.

An event whose menu is selected from the event menu may cause a visual effect or sound effect to be displayed on the output module of the electronic device 3000.

The event menu may include the Applause menu, Star menu, Correct menu, Wrong menu, Fail menu, and Drum menu to generate sound effects. there is. For example, when the electronic pen 2000 selects the applause menu when handwriting information is reproduced on the display, the applause sound already stored in the memory module 3400 of the electronic device 3000 is output from the speaker of the electronic device 3000. It can be.

The event menu may include a mask menu to generate image effects. The basic settings of the data recording application 5000 are to reproduce handwriting information written in Medium and to display the user's video and audio information. In order to prevent the user's face from appearing in the content, the user selects the mask menu. It can be masked so that its face is obscured. Of course, the data recording application 5000 may turn off the image information through the menu medium 1500 in order to block the image information about the user from being displayed on the display, but this fundamentally blocks filming of the user, so the user It is not appropriate when you want to cover only the face.

Below, a method of creating content in the graphic medium 1400 by allowing the user to select a document file stored in the electronic device 3000 will be described. The contents previously explained regarding the blank medium 1200 can also be applied to the graphic medium 1400.

Figures 22 to 26 are screens explaining the progress of the data recording application after selecting the graphic medium.

Referring to FIGS. 7 and 22 to 26 , the user may select a graphic medium 1400 from the data recording application 5000.

In order to create a graphic medium (1400), the user creates a file by taking a photo directly through the data recording application (5000), or writes a picture or text already stored in the memory module (3400) of the electronic device (3000). You can select a document file.

When the data recording application 5000 loads document files stored in the electronic device 3000 and selects a file 1410 to be used as a graphic medium 1400, the file is stored in the data recording application as shown in FIG. 23. Convert it to a graphic medium image (1420) that can be used in (5000). The graphic medium image 1420 may be created as a graphic medium file and output as a graphic medium 1400 through an output module 3200 connected to the electronic device 3000, for example, a printer. Figure 25 shows the actual output of a graphic medium file.

If the file stored in the electronic device 3000 consists of multiple sheets, the graphic medium image 1420 may also consist of multiple sheets. When a graphic medium file including a graphic medium image 1420 composed of multiple chapters is output to the graphic medium 1400, the graphic medium 1400 may also be composed of a plurality of sub-mediums.

Additionally, Graphic Medium files can be sent by email and other transmission media through the sharing function and output as Graphic Medium through another output module connected to the electronic device that received them.

When a graphic medium file is output to the graphic medium 1400, it may be printed and output on a blank medium 1200 on which a code pattern is formed. If a Graphic Medium (1400) file needs to be printed on a medium for which no code pattern is provided, the data recording application must also indicate the code pattern in the Graphic Medium file, which means printing the code pattern and the graphics on the Graphic Medium (1400) file together. Therefore, the output speed decreases. However, according to an embodiment of the present invention, by printing a graphic medium file on a blank medium 1200 on which a code pattern is already printed at the time of printing, the code pattern and the graphic on the graphic medium 1400 file are printed on a medium without a code pattern. The speed of output can be improved compared to printing together. When printing a print containing a pattern code with a color printer, the object being printed, excluding the pattern code, must be printed in a color other than black. This is because the electronic pen can detect the pattern code by radiating infrared rays to the medium and recognizing the non-reflective area with an infrared sensor. When a pattern code is printed on medium together with a graphic medium file, if the graphic medium file contains content expressed in black, it may be difficult to recognize the code pattern because it is printed with black print quality containing carbon. there is. In order to overcome this, the graphic medium (1400) file is converted to a specific color series (for example, blue color) as shown in Figure 25, and the converted graphic medium file is printed on the blank medium (1200) on which the code pattern is already printed. do. However, the graphic medium is expressed in converted colors only when printing, and in the data recording application 5000, the graphic medium image is expressed in the original color.

Referring to FIG. 26, the graphic medium image 1420 may be output on the display of the electronic device 3000 and then undergo a calibration process. The user taps the electronic pen 2000 on the cross mark 1430, which is a mark displayed at the top and bottom of the printed graphic medium 1400, and prints the graphic medium printed on the display screen of the graphic medium 1400 and the electronic device 3000. (1400) Calibration between images can be performed. If the graphic medium 1400 is composed of a plurality of sub-mediums, a calibration process may be performed for each sub-medium. In the embodiment of the present invention, the process of calibrating by tapping the electronic pen 2000 on the cross mark displayed on the graphic medium 1400 has been described, but the calibration process is not limited to this and can be performed in various ways. .

Figures 27 and 28 are diagrams showing content creation in a graphic medium according to embodiments of the present invention. Figure 27 shows content reproduced on an electronic device, and Figure 28 shows a writing trace written on a graphic medium.

Referring to FIGS. 27 and 28 , when a graphic medium image is output on the display of the electronic device 3000, the user can create content by writing a picture or writing on the printed graphic medium 1400. Content written on the graphic medium 1400 can be reproduced on the display of the electronic device 3000.

Since the ink inherent in the electronic pen (2000) is black, the writing trace displayed on the graphic medium (1400) appears only in black, but the electronic pen (2000) is set in the menu medium (1500) while writing content. When you select a desired color or desired stroke size from the handwriting menu, an event in the selected menu occurs and the color and size of the stroke change and can be reproduced on the display.

When the core of the electronic pen 2000 has various inks and the electronic pen 2000 writes in different colors on the graphic medium 1400 or blank medium 1200, data is converted into data in the electronic pen 2000. The pen data may include color information, and when received in the electronic device 3000, the pen data is interpreted and the handwriting information can be reproduced on the display according to the color information included in the pen data. Therefore, in this case, even if there is no menu medium 1500, the color of the handwriting information can be reproduced on the display.

Figure 29 is a diagram reproducing content written on a graphic medium according to embodiments of the present invention.

Referring to FIG. 29, the user outputs a map representing a region in Thailand as a graphic medium (1400) image on the display and provides an explanation of the region.

The blue brackets, green circles, and red bent straight lines shown on the drawing represent the handwriting traces drawn by the user with the electronic pen (2000) on the Graphic Medium (1400) image, reproduced on the Graphic Medium (1400) image printed on the display. am.

Figure 30 is a diagram showing content production according to another embodiment of the present invention.

Referring to FIG. 30, the data recording application 5000 may further display other video information or audio information embedded in the electronic device 3000.

When using the data recording application 5000, the user stores video files, audio files, or files in which video and audio are recorded together (which may be referred to as multimedia files) stored in the electronic device 3000 together in the electronic device 3000. You can create content by running it in .

Alternatively, content can be created by entering a web address in the data recording application 5000 and executing a multimedia file retrieved from the web.

The woman displayed at the bottom in FIG. 30 is a user screen for creating content, and this can be defined as first image information 1270 or first audio information. The screen displayed in the upper left corner can be defined as second video information 1290 or second audio information that displays video (including video and audio information) embedded in the electronic device 3000.

The area where the handwriting information 1260 is reproduced on the display of the electronic device 3000 may be referred to as the first window 1310, and the area where the user's appearance is displayed may be referred to as the second window 1320. Additionally, the area where the second image information 1290 is displayed may be referred to as the third window 1330. The first window 1310 may be a wider area including the area of the second window 1320 or the third window 1330. The position of the second window or third window can be adjusted by the user's touch and drag within the first window.

The present invention described above is capable of various substitutions, modifications, and changes without departing from the technical spirit of the present invention to those of ordinary skill in the technical field to which the present invention pertains. It is not limited by the drawings. Additionally, the embodiments described in this document are not limited to application, and all or part of each embodiment may be selectively combined so that various modifications can be made. Furthermore, the steps constituting each embodiment may be used individually or in combination with the steps constituting other embodiments.

1000: Medium
1200: Blank Medium
1210: Blank medium image
1240: Record button
1260: Handwritten information
1270: Video information
1280: Second handwriting information
1290: 2nd video information
1310: 1st window
1320: 2nd window
1330: Third window
1400: Graphic Medium
1410: file
1420: Graphic Medium Image
1430: Cross sign
1500: Menu Medium
1510: System menu
1520: Handwriting menu
1530: Event menu
2000: Electronic pen
3000: Electronic devices
4000: Server
5000: Data recording application

Claims (8)

In electronic devices,
a memory that stores content including handwriting information - the handwriting information corresponds to pen data obtained by converting a handwriting trace written on the medium with an electronic pen - and image information;
output device; and
Includes a control unit;
The control unit,
Reproducing the handwriting information in a first window of the output device so that the writing trace is reproduced in chronological order,
set to reproduce the image information on a second window of the output device, the second window having an adjustable position;
An electronic device characterized in that it automatically shifts the area where the handwriting information is reproduced according to the handwriting trace written on the medium.
According to paragraph 1,
An electronic device, characterized in that the image information is captured by a camera included in the electronic device.
According to claim 1,
The medium includes at least one of a blank medium or a graphic medium, and the electronic device displays at least one of an image corresponding to the blank medium or an image corresponding to the graphic medium on the output device. .
According to clause 3,
The blank medium or the graphic medium is an electronic device including a pattern code that can be detected by the electronic pen.
According to clause 3,
The graphic medium is an electronic device produced by printing a graphic media file on the blank medium.
According to claim 1,
The medium includes menu medium,
The menu medium is an electronic device that includes a system menu, a handwritten menu, or an event menu.
According to clause 6,
The menu medium includes a plurality of menus, each menu has a pattern code that can be detected by the electronic pen, and the electronic device selects one of the plurality of menus included in the menu medium. An electronic device characterized in that when a selection is detected, the event of the selected menu is reproduced on the output device.
According to clause 6,
The menu medium includes a stop menu, and when the electronic device detects the occurrence of a stop event, editing of the handwriting information is performed while reproduction of the image information is stopped.