CN220064793U

CN220064793U - Writing device and writing system

Info

Publication number: CN220064793U
Application number: CN202320965900.4U
Authority: CN
Inventors: 孙建成
Original assignee: TIANJIN JIHAO TECHNOLOGY CO LTD
Current assignee: TIANJIN JIHAO TECHNOLOGY CO LTD
Priority date: 2023-04-25
Filing date: 2023-04-25
Publication date: 2023-11-21
Anticipated expiration: 2033-04-25

Abstract

The utility model relates to the technical field of computers, and provides a writing device and a writing system. Wherein, writing device includes: the camera is connected with the input end of the processing unit, and the input end of the storage unit is connected with the output end of the processing unit; the camera is used for collecting multiple frame object surface images of the surface of the object to be written when the writing device writes, and sending the multiple frame object surface images to the processing unit; the processing unit is used for obtaining a first writing track when the writing device writes, and writing the first writing track into the storage unit. The writing device can recognize writing tracks through the object surface images acquired by the camera of the writing device, does not need to be matched with special external equipment such as an electronic whiteboard and a signal receiver, is very convenient to use, and reduces the use cost of users.

Description

Writing device and writing system

Technical Field

The utility model relates to the technical field of computers, in particular to a writing device and a writing system.

Background

With the popularity of computers and projectors, multimedia presentations are becoming more and more frequent, teacher teaching, product presentation and conference discussion all adopt multimedia, in the presentation process, a presenter often needs to mark emphasis and simply write to draw attention and exchange interaction of audiences, and at present, a laser pen is mostly used to realize the function, however, a laser pen in the prior art usually needs to be matched with a special external device (such as an electronic whiteboard and a signal receiver) to identify writing tracks of a user, and great inconvenience is brought to the user.

Disclosure of Invention

An object of an embodiment of the present utility model is to provide a writing device and a writing system, so as to improve the above technical problems.

In order to achieve the above purpose, the present utility model provides the following technical solutions:

in a first aspect, an embodiment of the present utility model provides a writing device, including: the camera is arranged on the body, the output end of the camera is connected with the input end of the processing unit, and the input end of the storage unit is connected with the output end of the processing unit; the camera is used for collecting multi-frame object surface images of the surface of the object to be written when the writing device writes, and sending the multi-frame object surface images to the processing unit; the processing unit is used for obtaining a first writing track when the writing device writes, and writing the first writing track into the storage unit.

The writing device can recognize writing tracks through the object surface images acquired by the camera of the writing device, does not need to be matched with special external equipment such as an electronic whiteboard and a signal receiver, is very convenient to use, and reduces the use cost of users. In addition, the writing device also comprises a storage unit, and if the storage unit is nonvolatile or at least comprises a nonvolatile part, the writing device can be independently used and record writing tracks without depending on any external equipment, so that the use scene of the writing device is greatly widened, and the subsequent review of writing contents is also facilitated.

It should be noted that the use scenario of the writing device described above is not limited to multimedia presentations, but may also be used for example for classroom notes, conference recordings, etc.

In one implementation of the first aspect, the writing device further includes: the gyroscope and the accelerometer are arranged on the body, and the output end of the gyroscope and the output end of the accelerometer are connected with the input end of the processing unit; the gyroscope and the accelerometer are used for collecting motion data when the writing device writes and sending the motion data to the processing unit.

In the above implementation, the motion data collected by the gyroscope and the accelerometer reflect the motion trail of the writing device, so that the processing unit can improve the recognition effect of the first writing trail according to the motion data.

In one implementation of the first aspect, the body includes a pen tip, and the gyroscope and the accelerometer are both disposed on the pen tip.

In the above implementation manner, since the movement of the pen point is most consistent with the actual writing track, the gyroscope and the accelerometer are arranged on the pen point, and the obtained movement data can represent the actual writing track, so that the recognition effect of the first writing track can be improved by using the movement data.

In one implementation of the first aspect, the body includes a nib, and the writing device further includes: the contact sensing unit is arranged on the body, the output end of the contact sensing unit is connected with the input end of the processing unit, and the input end of the camera is connected with the output end of the processing unit; the contact sensing unit is used for triggering the camera to acquire the object surface image through the processing unit when sensing that the pen point contacts the surface of the object to be written.

The writing device in the implementation mode is in contact type, namely the device is required to be contacted with the surface of an object to complete writing, the contact type writing has a writing sense, the writing habit of most users is more met, and the user can write more accurately.

In an implementation manner of the first aspect, the apparatus further includes: the laser emission unit is arranged on the body, and the input end of the laser emission unit is connected with the output end of the processing unit; the contact sensing unit is also used for triggering the laser emitting unit to generate laser irradiating the surface of the object to be written through the processing unit when the pen point is sensed to contact the surface of the object to be written.

In the implementation manner, the laser irradiates the surface of the object to be written to form the laser spot, so that the first writing track can be identified through the laser spot in the image of the surface of the object, and the identification accuracy of the writing track is improved due to the fact that the position of the laser spot in the image is very obvious. In addition, the writing track is identified through the laser light spots, so that writing marks, dirt and the like are not required to be left on the surface of the written object by the writing device, the written object can be kept intact, writing on various objects can be supported, the requirement of traditional writing on paper is avoided, and the environment protection trend is met.

In one implementation of the first aspect, the writing device further includes: the mode change-over switch is arranged on the body, and the output end of the mode change-over switch is connected with the input end of the processing unit; the mode switching switch is used for triggering the processing unit to switch the working mode of the writing device, and the working mode comprises a writing mode and an indicating mode; the laser emission unit is used for generating laser irradiating the surface of the written object under the triggering of the contact sensing unit when the writing device is in the writing mode; when the writing device is in the indicating mode, the laser emitting unit is used for continuously generating laser, and the camera does not collect the image of the object surface.

In the implementation manner, the mode switch is arranged, so that the writing device can be used for writing and also can be used for indicating (for example, laser emitted by the laser unit irradiates a distance), the use scene of the writing device is enriched, and the writing device is convenient to realize some interactive functions.

In an implementation manner of the first aspect, the body further includes: a pen body and a connecting part; the diameter of the pen point is smaller than that of the pen body, the connecting portions are respectively connected with the pen point and the pen body, the diameter of the connecting portions gradually increases from one end connected with the pen point to one end connected with the pen body, inclined side faces of the connecting portions are formed, and the camera and the laser emitting units are arranged on the side faces of the connecting portions.

In the above implementation manner, the camera and the laser emission unit are arranged on the side surface of the connecting portion, so that the structure of the writing device is more compact and attractive (no component protrudes out of the surface of the body), and the writing device is convenient for a user to hold.

In one implementation of the first aspect, the writing device further includes: the lighting unit is arranged on the body, and the input end of the lighting unit is connected with the output end of the processing unit; the contact sensing unit is also used for triggering the illumination unit to generate illumination light for illuminating the surface of the object to be written through the processing unit when the pen point is sensed to contact the surface of the object to be written.

In the implementation manner, the illumination unit is arranged, so that the effect of light supplementing can be achieved, the brightness of the object surface image acquired by the camera is increased, and the track recognition effect is improved.

In an implementation manner of the first aspect, the camera is disposed at a side surface of the pen head.

In the implementation manner, the camera is arranged on the side face of the pen point, so that the writing device can be prevented from appearing in the object surface image as much as possible, and the recognition effect of the writing track is improved. In addition, since the writing device at this time does not use the laser light emitting unit, there is no fear that the camera blocks the propagation path of the laser light.

In an implementation manner of the first aspect, the body further includes: a pen body and a connecting part; the diameter of the pen point is smaller than that of the pen body, the connecting portions are respectively connected with the pen point and the pen body, the diameter of the connecting portions gradually increases from one end connected with the pen point to one end connected with the pen body, inclined side faces of the connecting portions are formed, and the illuminating units are arranged on the side faces of the connecting portions.

In the above implementation manner, the lighting unit is disposed on the side surface of the connecting portion, so that the writing device is more compact and attractive in structure (no component protrudes out of the surface of the body), and is convenient for a user to hold.

In one implementation of the first aspect, the writing device further includes: the laser emission unit is arranged on the body, the input end of the laser emission unit is connected with the output end of the processing unit, and the output end of the working switch is connected with the input end of the processing unit; the working switch is used for triggering the laser emitting unit to generate laser irradiating the surface of the object to be written through the processing unit when being started.

The writing device in the implementation manner is non-contact, namely, the writing can be completed without the device contacting the surface of the object (but in some implementations, the non-contact writing device also supports the surface of the object to write), so that the writing device is convenient for a user to write at intervals, and the user requirements of specific scenes such as multimedia presentation and the like can be met. In addition, the position of the laser spot in the image is very obvious, so that the recognition accuracy of the writing track is improved. In addition, the writing device is non-contact, so that handwriting, dirt and the like are not required to be left on the surface of the written object by the writing device, the written object can be kept intact, writing on various objects can be supported, the requirement of traditional writing on paper is avoided, and the environment protection trend is met.

In an implementation manner of the first aspect, the camera is disposed at one end of the body.

In the above implementation manner, since the writing device is non-contact, that is, the writing device is usually spaced from the surface of the object to be written, the camera may be disposed at one end of the body, so that the writing device itself will not appear in the image of the surface of the object, which is beneficial to improving the recognition effect of the writing track.

In one implementation manner of the first aspect, the Camera is a Camera Cube Chip (CCC for short).

The chip-scale camera module adopted in the implementation mode is a miniature camera, so that the chip-scale camera module is convenient to integrate on a writing device, and the implementation cost of the device is reduced.

In one implementation of the first aspect, the writing device further includes: the communication unit is arranged on the body, and the input end of the communication unit is connected with the output end of the processing unit; the processing unit is further configured to read the first writing track from the storage unit, and send the first writing track to a terminal device through the communication unit.

In the implementation manner, by setting the communication unit, the first writing track recognized by the writing device can be sent to the terminal device, so that richer functions of displaying, projecting and the like of the writing track are realized.

In a second aspect, embodiments of the present application provide a writing system comprising: the writing device provided in the first aspect or any implementation manner of the first aspect and the terminal device, where the writing device further includes a communication unit disposed on a body of the writing device (only for a case where the writing device provided in the first aspect or any implementation manner of the first aspect does not include the communication unit), an input end of the communication unit is connected to an output end of a processing unit of the writing device, and an output end of the communication unit is connected to the terminal device in a communication manner; the processing unit is further configured to read a first writing track from a storage unit of the writing device, and send the first writing track to a terminal device through the communication unit; the terminal device is used for displaying the first writing track and/or controlling other devices to display the first writing track.

The writing system has the beneficial effects of the writing device in the first aspect, and the first writing track can be displayed through the terminal equipment and/or other equipment controlled by the terminal equipment, so that a writer and other users can observe writing results conveniently, and communication interaction among the users is realized.

In one implementation manner of the second aspect, the system further includes: the projector is in communication connection with the terminal equipment; the projector is used for displaying the first writing track under the control of the terminal equipment.

Drawings

In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings that are needed in the embodiments of the present application will be briefly described below, it should be understood that the following drawings only illustrate some embodiments of the present application and should not be considered as limiting the scope, and other related drawings can be obtained according to these drawings without inventive effort for a person skilled in the art.

FIG. 1 is a first block diagram of a writing instrument according to an embodiment of the present application;

FIG. 2 is a schematic view of a frame of an object surface image captured by a writing instrument;

FIG. 3 is a schematic view of a multi-frame object surface image captured by a writing device;

FIG. 4 is a schematic diagram of a writing instrument performing writing trace recognition;

FIG. 5 is a second block diagram of a writing instrument according to an embodiment of the present application;

FIG. 6 is a third block diagram of a writing instrument according to an embodiment of the present application;

Fig. 7 is a block diagram of a writing system according to an embodiment of the present application.

Detailed Description

The following describes the technical solution in the embodiment of the present application in detail with reference to the drawings in the embodiment of the present application. It should be noted that: like reference numerals and letters denote like items in the following figures, and thus once an item is defined in one figure, no further definition or explanation thereof is necessary in the following figures.

The terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises the element.

The terms "first," "second," and the like, are used merely to distinguish one entity or action from another entity or action, and are not to be construed as indicating or implying any actual such relationship or order between such entities or actions.

The writing device provided by the embodiment of the application comprises a contact type writing device and a non-contact type writing device, wherein the contact type writing device is required to contact the surface of a written object to write, the use mode of the writing device is similar to that of a traditional pen, and the non-contact type writing device supports the writing device to write at intervals. The contact writing device may further be divided into two types including a laser emitting unit and a non-contact writing device, and the non-contact writing device basically includes the laser emitting unit (but there are exceptions), and fig. 1, fig. 5, and fig. 6 respectively correspond to three main types of implementations of the writing device, which will be described in turn, and these three types of implementations are simply referred to as a first type of implementation (contact type and including the laser emitting unit), a second type of implementation (contact type and including no laser emitting unit) and a third type of implementation (non-contact type) of the writing device.

Referring to fig. 1, in a first type of implementation, the writing instrument 10 includes at least the following components: the device comprises a body 100, a camera 108, a processing unit 118, a storage unit 120, a contact sensing unit 104 and a laser emitting unit 106. Wherein the combination of the two components of the touch sensing unit 104 and the laser emitting unit 106 is specific to the first type of implementation of the writing device 10, the other two types of implementations generally do not include both components at the same time.

The body 100 may refer to the housing of the writing instrument 10, with the remaining components of the writing instrument 10 each being mounted to the body 100. As shown in fig. 1, in an alternative, the writing instrument 10 is similar in appearance to a conventional pen in that the body 100 of the writing instrument 10 may include a nib 100a, a barrel 100c, and a connecting portion 100b connecting the nib 100a and the barrel 100c, the nib 100a having a smaller diameter than the barrel 100c, the connecting portion 100b having a gradually increasing diameter from one end of the connecting nib 100a to one end of the connecting barrel 100c, forming an inclined side of the connecting portion 100 b. Note that the diameter is generally a circle or a quasi-circle, but the diameter in the above-described structure can be understood more broadly, for example, if the pen body 100c is a hexahedron, the diameter of the pen body 100c can be understood as the diameter of a circumscribed circle of the hexahedron cross section (a hexagon).

It should be appreciated that the body 100 of the writing instrument 10 may also have a different structure than that of fig. 1: for example, the body 100 may include only the nib 100a and the body 100c, which are directly connected, without the connection portion 100b; for another example, the entire body 100 may be cylindrical without a distinct parting line between the nib 100a and the barrel 100c, etc.

The camera 108 is not limited to a specific position, but may be capable of capturing a clear image, for example, if the camera 108 is disposed at the front end of the pen head 100 a. For example, referring to fig. 1, the camera 108 may be disposed at a side of the connection portion 100b, so that the writing device 10 is more compact and beautiful (the camera 108 does not protrude beyond the column formed by the body 100), and is convenient for a user to hold.

The output end of the camera 108 is connected to the input end of the processing unit 118, and the input end of the camera 108 is connected to the output end of the processing unit 118. In the embodiment of the application, the connection between the input end and the output end of different components refers to circuit connection, and the circuit connection can refer to direct connection between the components through a circuit or indirect connection between the components through the circuit and other components on the circuit.

The camera 108 is used to capture multiple frames of object surface images of the surface of the object being written while the writing device 10 is writing. For the contact writing instrument 10, the object to be written may be an object having a certain hardness, such as a table, a wall, a blackboard, etc., or an object having a certain hardness under support, such as paper, cloth, etc. (itself relatively soft but having hardness after being supported on an object such as a desk). The written object surface may refer to a written surface of the written object. The contact type writing apparatus 10 performs writing in a similar manner to a conventional pen, in which the pen tip 100a is moved in contact with the surface of the object to be written, and the movement of the pen tip 100a in contact with the surface of the object to be written is performed, that is, the writing process, and the movement trace of the pen tip 100a is the writing trace. If the body 100 of the writing instrument 10 does not have a distinct nib 100a (for example, in the case of a unitary body 100 with a cylinder), it is sufficient to write with one end of the writing instrument 10 in contact with the surface of the object to be written, which will not be described in particular hereinafter.

The image of the surface of the object being written, referred to as an object surface image, may be acquired by camera 108 at a frequency (e.g., 200 frames per second, 500 frames per second) during writing by writing device 10 and sent via its own output to the input of processing unit 118 for recognition of the writing trajectory within processing unit 118. Optionally, increasing the frequency with which the camera 108 captures images of the object surface is beneficial to improving the accuracy of recognition of the writing trajectory.

If the writing device 10 is not writing, the camera 108 may stop capturing images of the object surface to save device power and avoid identifying incorrect writing tracks. At this time, the camera 108 may be in a power-off state or may be in a sleep state waiting to be awakened after the writing device 10 starts writing. Of course, when the writing device 10 is not writing, it is also possible that the camera 108 only captures images of the object surface but does not use these images.

The camera 108 should be capable of sensing the laser light generated by the laser emitting unit 106, i.e. the laser spot formed by the laser irradiation is required in the image of the object surface. If the laser light generated by the laser light emitting unit 106 is visible light, the camera 108 may be a visible light camera, and if the laser light generated by the laser light emitting unit 106 is not visible light (for example, infrared light), the camera 108 cannot be a visible light camera, but a camera capable of sensing infrared light is used.

The laser emitting unit 106 is disposed on the surface of the body 100, and its specific position is not limited. For example, referring to fig. 1, the laser emitting unit 106 may be disposed at a side of the connection part 100b, so that the structure of the writing device 10 is more compact and beautiful (the laser emitting unit 106 does not protrude beyond the column formed by the body 100), and is convenient for a user to hold.

The input end of the laser emitting unit 106 is connected to the output end of the processing unit 118, so that the processing unit 118 may send a control signal to the input end of the laser emitting unit through its own output end to control the light emitting state of the laser emitting unit 106.

The laser emitting unit 106 is configured to generate laser light for irradiating the surface of the object to be written when the writing device 10 performs writing, where the laser light forms a laser spot on the surface of the object to be written, the positions of the laser emitting unit 106 and the camera 108 need to be reasonably set, and the camera 108 is reasonably selected (as mentioned above, the camera 108 capable of sensing the laser signal is selected), so that the image of the surface of the object collected by the camera 108 includes the laser spot. For example, the laser emitting unit 106 may be aligned to a position near the pen tip 100a (for example, a position within a range of 2cm near the pen tip 100 a) to emit laser light, so that a laser spot may appear near the pen tip 100a, and the shooting range of the camera 108 is also aligned to the pen tip 100a, so that it is ensured that the laser spot is always included in the image of the object surface.

If the writing device 10 is not writing, the laser emitting unit 106 may stop generating laser light to save device power consumption and avoid identifying an incorrect writing trajectory.

The laser light emitted by the laser emitting unit 106 is not limited in wavelength, and may be, for example, blue light, green light, red light, infrared light, etc., and if invisible light is adopted, the formed laser spot is not easily interfered by the color of the surface of the written object, which is beneficial to improving the recognition effect of the writing track. Alternatively, the laser emitting unit 106 may be implemented by a laser diode, a gas laser, or the like, and the camera 108 may be further provided with a filter corresponding to the laser wavelength in order to better identify the laser spot.

The touch sensing unit 104 is disposed on the body 100, and the specific location is not limited, and may be located on the surface of the body 100 or may be located inside the body 100, and fig. 1 is a case of being disposed inside the body 100. An output of the touch sensing unit 104 is connected to an input of the processing unit 118.

The contact sensing unit 104 is used for triggering the camera 108 to collect an image of the surface of the object through the processing unit 118 and triggering the laser emitting unit 106 to generate laser light irradiating the surface of the object when sensing that the pen head 100a contacts the surface of the object to be written. The contact sensing unit 104 senses that the pen tip 100a contacts the surface of the object to be written, that is, the writing device 10 starts writing. For example, when the contact sensing unit 104 senses that the pen tip 100a contacts the surface of the object to be written, a trigger signal may be sent to the input end of the processing unit 118 through the output end of the contact sensing unit, and after the processing unit 118 receives the trigger signal, a corresponding control signal is sent to the input end of the camera 108 and the input end of the laser emitting unit 106 through the output end of the contact sensing unit, so as to control the camera 108 to start to collect an image of the surface of the object, and control the laser emitting unit 106 to start to emit laser.

Referring to fig. 1, for example, the contact sensing unit 104 may be implemented as a pressure sensing unit (e.g., a pressure switch) connected to the pen tip 100a, and when the pen tip 100a contacts the surface of the object to be written, the pressure generated by the surface of the object to be written on to the pen tip 100a is sensed by the pressure sensing unit, and the pressure sensing unit sends a trigger signal to the processing unit 118 to trigger the camera 108 and the laser emitting unit 106 to start operating through the processing unit 118. It should be appreciated that in other implementations of the contact sensing unit 104, other signals (or information) may be perceived that are representative of the nib 100a contacting the surface of the object being written: for example, the contact sensing unit 104 may be a distance sensing unit for sensing the distance between the pen tip 100a and the surface of the object to be written, and a distance of 0 or less is considered to be contact; for another example, the contact sensing unit 104 may be a touch sensing unit (capacitive, resistive, etc.) for sensing the behavior of the surface of the object to be written contacting (touching) the pen tip 100a, etc.

The contact sensing unit 104 may also be used to trigger the camera 108 to stop capturing images of the surface of the object to be written by the processing unit 118 and to trigger the laser emitting unit 106 to stop generating laser light to irradiate the surface of the object to be written when it is sensed that the pen tip 100a stops contacting the surface of the object to be written. This process is similar to the process by which the contact sensing unit 104 senses that the nib 100a contacts the surface of the object to be written, and will not be described in detail.

The processing unit 118 is provided on the body 100, and a specific position is not limited and may be provided inside the body 100, for example. An output of the processing unit 118 is connected to an input of the storage unit 120, and an input of the processing unit 118 is connected to an output of the storage unit 120.

The processing unit 118 is configured to determine a first writing track when the writing device 10 writes according to the multi-frame object surface image acquired by the camera 108, and send the first writing track to the input end of the storage unit 120 through the output end thereof, so as to implement writing the first writing track into the storage unit 120.

Briefly, the main function of the processing unit 118 is recognition of writing tracks based on images of the object surface, the recognition principle of which is briefly described below in connection with fig. 2 to 4:

fig. 2 is a schematic view of a frame of an image of the surface of an object captured by the writing instrument 10. Referring to fig. 2, the black rectangular frame of the outermost layer indicates the surface of the object to be written, the gray circular area indicates the shooting range of the camera 108, the black dot indicates the position where the pen tip 100a contacts the surface of the object to be written, the white dot indicates the position of the laser spot, and the grid marked with numerals indicates the characteristics of the surface of the object to be written.

Wherein the position of the black dot with respect to the gray circular area is always fixed, since the relative positions of the nib 100a and the camera 108 are fixed, and the position of the white dot with respect to the black dot is always fixed, since the relative positions of the laser emitting unit 106 and the nib 100a are fixed. In addition, the characteristic of the surface of the object to be written is represented by the grid for the convenience of describing the recognition principle of the writing track, and the surface of the object to be written is likely to have no obvious characteristic such as the grid in practice, but only features such as wood texture, pits, use marks and the like on a tabletop, but the difference of the characteristic forms does not affect the recognition principle of the writing track.

Fig. 3 is a schematic view of a multi-frame object surface image captured by the writing instrument 10. Referring to fig. 3, there are shown a total of five successive frames of object surface images, which are similar to those shown in fig. 2. As can be seen from fig. 3, from the first frame to the fifth frame, the position of the white dot is changed with respect to the grid of the surface of the object to be written, if the number in the grid is regarded as the coordinate, in the first frame, the abscissa on the left side of the white dot is about 7.9 units, the ordinate is about 7 units, the abscissa on the left side of the white dot is about 9.2 units, the ordinate is about 7 units, that is, the white dot is moved forward by 1.3 units toward the abscissa, the longitudinal direction is not moved, and since the relative positions of the laser emitting unit 106 and the pen point 100a are fixed, the movement locus of the white dot may be or at least may represent the movement locus of the black dot to a certain extent, that is, it may be regarded that the black dot is also moved forward by 1.3 units toward the abscissa, the longitudinal direction is not moved, and since the black pen point is the contact point of 100a and the surface of the object to be written, the movement locus of the black dot is the writing locus, it is recognized that the writing locus is 1.3 units long (short left to right of the pen point).

Fig. 4 is a schematic diagram of the writing instrument 10 for writing trace recognition. Referring to fig. 4, the five frame object surface images in fig. 3 may be matched, the matching may be understood as overlapping the same parts in each frame of image, the image matching algorithm may refer to the prior art, and the image on the left side of fig. 4 is obtained after matching. Wherein, by aligning the characteristics of the written object surface in each frame of object surface image, matching of the object surface image is not difficult to complete (for example, the object surface images of each frame may be overlapped after aligning the grids). Since the positions of the white dots (which are easily obtained by detecting the laser spots) in the five-frame object surface image of fig. 3 are continuously changed with respect to the surface of the object to be written, after image matching, the white dots are connected into a white stub, the white stub is filtered out from the matched object surface image, and the direction in which the stub is formed is combined, so that a motion track of the white dots, or a recognized writing track (short horizontal, left to right) is obtained, as shown in the right side of fig. 4 (a black frame is added to the white stub for convenience of observation), the writing track is composed of five points, each point corresponds to the position of the laser spot in the one-frame object surface image.

Note that black dots are not used in the track recognition process described above, and are drawn merely to facilitate understanding of the position of the nib 100a, which also indicates that in some alternatives of the writing device 10, the range of the camera 108 allows for the nib 100a to be excluded.

The principle of recognition of the writing track in fig. 4 can be further generalized as: and determining a first writing track according to the position change of the laser spot in the multi-frame object surface image relative to the written object surface.

The number of frames of the multi-frame object surface image is not limited, and may be 50 frames, 100 frames, or the like. Taking 100 frames as an example, since writing is a continuous process, the object surface image generated from the beginning of writing to the end of writing (meaning that all the content to be written is written) may be far more than 100 frames, and the first writing track may be understood as a writing track determined based on the object surface image of every 100 frames (determining method refers to the principle of fig. 4), or may be understood as a result of stitching a plurality of small writing tracks, each small writing track may be determined based on the object surface image of 100 frames, and the number of object surface images used for determining the first writing track is an integer multiple of 100 frames.

Furthermore, the first writing track may be the result of directly splicing the writing tracks of the small segments, or may be the original writing track obtained by directly splicing the writing tracks of the small segments, which is input into an artificial intelligence algorithm, and the writing track which is output after being processed by the artificial intelligence algorithm (the situation that the writing tracks of the small segments are directly input into the artificial intelligence algorithm may be similar to the analysis). The artificial intelligence algorithm can be an algorithm based on a neural network model or other machine learning models, and the processing performed by the artificial intelligence algorithm comprises one or more of track optimization, writing content recognition and the like.

For example, if the frame rate at which the camera 108 captures images of the object surface is low, the original written trajectory may be relatively discrete, and the artificial intelligence algorithm may process it into a more continuous written trajectory, which is a trajectory optimization approach; for example, if the quality of the image of the object surface acquired by the camera 108 is poor, which results in a discontinuous condition in the original writing track, the artificial intelligence algorithm can process the original writing track into a more continuous writing track, which is also a track optimization mode; for another example, the artificial intelligence algorithm may identify what the user writes specifically based on the original writing trajectory (assuming the user is writing), then take the morphology of the user written word as the output writing trajectory, which is a type of writing content identification, and so on.

Optionally, the orientation of the laser emitting unit 106 is reasonably set such that the laser spot is always located near the point where the pen tip 100a contacts the surface of the object to be written (e.g., no more than 2 cm), not only to ensure that the image of the object surface always contains the laser spot, but also to make the first writing track identified based on the laser spot closer to the actual writing track of the pen tip 100 a.

The processing unit 118 may be implemented by a micro control unit (Micro Controller Unit, abbreviated as MCU), a digital signal processor (Digital Signal Processor, abbreviated as DSP), an application specific integrated circuit (Application Specific Integrated Circuits, abbreviated as ASIC), a field programmable gate array (Field Programmable Gate Array, abbreviated as FPGA), or the like.

The storage unit 120 is configured to store the first writing track, although the storage unit 120 may also store other data, for example, one or more items of data such as computer program instructions executed by the processing unit 118, an image of an object surface, an original writing track that has not been processed by the artificial intelligence algorithm, and the like, where the data in the storage unit 120 may be read or written by the processing unit 118.

The Memory unit 120 may be implemented by one or more memories such as a random access Memory (Random Access Memory, RAM), a Read Only Memory (ROM), a programmable Read Only Memory (Programmable Read-Only Memory, PROM), an erasable programmable Read Only Memory (Erasable Programmable Read-Only Memory, EPROM), and an electrically erasable programmable Read Only Memory (Electric Erasable Programmable Read-Only Memory, EEPROM).

Referring to fig. 1, the writing instrument 10 may further include a bus 122, the bus 122 for better enabling electrical connection between the various components of the writing instrument 10, for example, the bus 122 may be implemented as a printed circuit board (Printed Circuit Board, abbreviated as PCB) on which some of the components of the writing instrument 10, such as the processing unit 118, the memory unit 120, etc., may be mounted.

Briefly summarizing the writing device 10 in fig. 1, the writing device 10 is in a contact type, and is capable of sensing the writing behavior of a user on the surface of the object to be written by the contact sensing unit 104, and identifying the writing track by the position change of the laser spot in the image of the surface of the object collected by the camera 108 relative to the surface of the object to be written.

The contact type writing has the writing sense, is more in line with the writing habit of most users, and is convenient for the users to write more accurately. In addition, the writing device 10 does not need to be matched with special external equipment such as an electronic whiteboard and a signal receiver, and can obtain writing tracks, so that the use cost of a user is reduced. In addition, the writing device 10 recognizes the first writing track by the position change of the laser spot, and since the position of the laser spot in the image is very obvious, it is advantageous to improve the recognition accuracy of the writing track. In addition, the writing track is identified through the laser light spot, so that writing, dirt and the like are not required to be left on the surface of the written object by the writing device 10, the written object can be kept intact, writing on various objects can be supported, the requirement of paper for traditional writing is avoided, and the trend of environmental protection is met.

It should be noted that, in the process of the processing unit 118 identifying the first writing track, the algorithm for detecting the position of the laser spot or the algorithm for matching the image may be an existing algorithm, so the innovation to be protected in the present application is mainly structural innovation, that is, the components of the camera 108, the laser emitting unit 106, etc. are integrated on the writing device 10, so that the first writing track can be obtained locally on the writing device 10, thereby avoiding the dependence on special external devices such as an electronic whiteboard, a signal receiver, etc.

Further, the writing device 10 further includes a storage unit 120, and in some implementations, the storage unit 120 can permanently store (i.e. the data is not lost after the power is turned off) the first writing track, so that the writing device 10 can be used independently and record the writing track, without relying on any external device, thereby greatly widening the usage scenario of the writing device 10 and facilitating subsequent review of the writing content.

The writing instrument 10 may be used in a variety of scenarios: for example, for multimedia presentations, i.e. the transmission of a first writing track stored locally to the writing means 10 to an external device for display, in order to share the written content to others (see later for details); for example, for classroom notes, conference recordings, writing the content to be recorded (first writing track) directly on the desktop, which is recorded in the storage unit 120, and which is subsequently exported to an external device for review when convenient, etc. The following continues with the description of other components that may be included in the writing instrument 10 of fig. 1:

On the basis of any of the above embodiments, optionally, the writing device 10 further includes a gyroscope and an accelerometer 102, where the gyroscope and the accelerometer 102 are disposed on the body 100, and may be disposed inside the body 100 or on a surface of the body 100, or may be disposed together or in different positions of the body 100, where fig. 1 illustrates the two disposed together. The output of the gyroscope and the output of the accelerometer are both connected to inputs of the processing unit 118. During writing by the writing instrument 10, the gyroscope and accelerometer 102 may sense the motion of the writing instrument 10 and collect corresponding data at a certain frequency (e.g., 30 frames per second, 50 frames per second), referred to as motion data, which is sent by the gyroscope and accelerometer 102 through its own output to the input of the processing unit 118 for use by the processing unit 118.

The gyroscope may collect an angular velocity of the movement of the writing device 10, and after converted by the processing unit 118, a direction of the movement of the writing device 10 (within a certain time) may be obtained, and the accelerometer may collect an acceleration of the movement of the writing device 10, and after converted by the processing unit 118, a distance of the movement of the writing device 10 (within a certain time) may be obtained. Optionally, increasing the frequency at which the gyroscope and accelerometer 102 collect data is beneficial to improving the accuracy of recognition of the written trace.

If the writing device 10 is not writing, the gyroscope and accelerometer 102 may stop collecting motion data to save device power consumption and avoid identifying incorrect writing trajectories. At this point, the gyroscope and accelerometer 102 may be in a powered off state or may be in a dormant state waiting to wake up after the writing device 10 begins writing. Of course, the gyroscope and accelerometer 102 may collect data without using it when the writing device 10 is not writing.

It was mentioned before that, when the contact sensing unit 104 senses that the pen tip 100a contacts the surface of the object to be written, the processing unit 118 triggers the camera 108 to collect an image of the surface of the object, when the processing unit 118 senses that the pen tip 100a stops contacting the surface of the object to be written, and when the writing device 10 further includes the gyroscope and the accelerometer 102, the contact sensing unit 104 may further trigger the gyroscope and the accelerometer 102 to collect motion data when the processing unit 118 senses that the pen tip 100a contacts the surface of the object to be written, and trigger the gyroscope and the accelerometer 102 to collect motion data when the processing unit 118 senses that the pen tip 100a stops contacting the surface of the object to be written. The specific triggering process for the gyroscope and accelerometer 102 is similar to that for the camera 108 and will not be repeated.

Further, in the case where the gyroscope and the accelerometer 102 are provided, the first writing trace may be obtained in a different manner from the foregoing, for example:

the writing trace obtained by the processing unit 118 from the multiple frame object surface image acquired by the camera 108 is referred to as a second writing trace, and the method of obtaining the writing trace is as described above.

In addition, after obtaining the movement direction and distance of the writing device 10 in a period of time according to the movement data collected by the gyroscope and the accelerometer 102, the processing unit 118 may obtain the movement track of the writing device 10 in the period of time, and since the movement data is collected only during the writing process of the writing device 10, the movement track at this time is the writing track of the writing device 10 and is called a third writing track.

Further, the processing unit 118 may obtain the first writing track by fusing the second writing track and the third writing track. The specific fusion method is not limited, and may be, for example, weighted fusion, conditional fusion (for example, only the second writing track is used as the first writing track in a period of time meeting a specific condition, only the third writing track is used as the first writing track in a period of time meeting another specific condition), fusion by an artificial intelligence algorithm, and the like.

For example, the second writing trace and the third writing trace may be input into an artificial intelligence algorithm, processed by the artificial intelligence algorithm, and then output the first writing trace. The artificial intelligence algorithm can be an algorithm based on a neural network model or other machine learning models, and the processing performed by the artificial intelligence algorithm at least comprises track fusion and one or more of track optimization, written content recognition and the like. The track optimization and the writing content identification can be performed on the writing track before fusion or the writing track after fusion, and specific operations can refer to the previous contents. If the second writing track and the third writing track are fused by an artificial intelligence algorithm, the artificial intelligence algorithm may not be used in the process of obtaining the second writing track.

The second writing track obtained based on the data acquired by the camera 108 has higher accuracy, but is easily affected by problems such as illumination variation, picture quality and the like; the third writing trace based on the data collected by the gyroscope and the accelerometer 102 is relatively low in accuracy, but is simple in acquisition mode and high in reliability. According to the alternative scheme, the first writing track is obtained through track fusion, so that the advantage complementation of the two writing tracks is realized, and the identification effect of the first writing track is improved. Particularly, under the condition of adopting an artificial intelligence algorithm, the method can autonomously determine how to fuse the second writing track and the third writing track by the algorithm, and the fusion effect is good.

In particular, for surfaces with insignificant features, such as solid surfaces, the determination of the change in position of the laser spot relative to the surface of the object being written by matching between the images of the object surfaces may not be effective because the images of the object surfaces lacking features are difficult to match accurately, i.e. the second writing track obtained at this time has a lower accuracy and even is not available at all, but the usability of the writing track is greatly improved after combining the third writing track, and it is seen that the applicability of the writing device 10, i.e. the support for writing on more object surfaces, may also be improved with the gyroscope and the accelerometer 102.

Further, the gyroscope and the accelerometer 102 may be disposed at any position of the body 100, but both may be disposed on the pen tip 100a (as shown in fig. 1), and since the movement of the pen tip 100a is most consistent with the actual writing track of the user, the movement data obtained by the gyroscope and the accelerometer 102 at this time can also represent the actual writing track of the user, so that the use of such movement data is more beneficial to improving the recognition effect of the first writing track, for example, a more accurate third writing track may be obtained, and thus after the third writing track and the second writing track are fused, a more accurate first writing track may be obtained.

Conversely, if, for example, the gyroscope and accelerometer 102 are disposed on the pen body 100c, the small movement of the pen tip 100a reflected on the pen body 100c may be a large movement, which results in a decrease in the accuracy of the obtained third writing track and thus affects the quality of the first writing track generated by fusion.

Optionally, the writing device 10 further includes a mode switch 112 disposed on the surface of the body 100, for example, referring to fig. 1, the mode switch 112 may be disposed on the surface of the pen body 100c of the body 100, so as to facilitate the user operation. An output of the mode switch 112 is connected to an input of the processing unit 118.

The mode switch 112 is used for triggering the processing unit 118 to switch the operation mode of the writing device 10, and the operation mode of the writing device 10 at least comprises a writing mode and an indication mode, so that the mode switch 112 at least comprises two states, corresponding to the two operation modes. For example, when the mode switch 112 is switched to a certain operation mode, a trigger signal corresponding to the mode may be sent to the input terminal of the processing unit 118 through the output terminal thereof, so that the processing unit 118 controls itself and other components to operate in a manner specified by the mode.

Wherein when the writing device 10 is in the writing mode, the laser emitting unit 106 generates laser light irradiating the surface of the object to be written under the triggering of the touch sensing unit 104 (if the writing device 10 further includes a gyroscope and an accelerometer 102, the gyroscope and the accelerometer 102 are also triggered to collect motion data), the camera 108 collects an image of the surface of the object under the triggering of the touch sensing unit 104, and the processing unit 118 may determine the first writing track according to the multiple frame object surface map (and possibly the motion data), which has been described above.

When the writing device 10 is in the pointing mode, the laser light emitting unit 106 continues to generate laser light and the camera 108 does not collect an image of the object surface (nor does the gyroscope and accelerometer 102 collect motion data if the writing device 10 further includes the gyroscope and accelerometer 102). In the pointing mode, the laser may be used to illuminate a remote object (e.g., projector curtain, blackboard) to support interaction with other users, etc.

The type of mode switch 112 is not limited and may be, for example, a tact switch, a toggle switch, a virtual switch on a touch screen, and the like.

Optionally, the writing device 10 further includes one or more function keys 114 disposed on the surface of the body 100, for example, referring to fig. 1, the function keys 114 may be disposed on the surface of the pen body 100c of the body 100, so as to facilitate the user's operation. The output of the function key 114 is connected to the input of the processing unit 118 for triggering the processing unit 118 to implement different functions. For example, one or more of a page turning function, a control function, a delete of written text, etc., some or all of which may be functions that are only validated when the writing instrument 10 communicates with the terminal device (via a communication unit described later), i.e., functions that the function keys 114 are directed to may be functions on the terminal device other than functions local to the writing instrument 10. For example, the page flip function may only be enabled when the writing instrument 10 is connected to a computer for PPT presentation.

The function keys 114 may be physical keys or virtual keys on a touch screen.

Optionally, the writing device 10 further includes a display screen 110 disposed on a surface of the body 100, for example, referring to fig. 1, the display screen 110 may be disposed on a surface of the pen body 100c of the body 100, so that a user may observe contents of the display screen 110. The input end of the display screen 110 is connected to the output end of the processing unit 118, and the processing unit 118 may send the content to be displayed to the input end of the display screen 110 through its own output end, so as to control the content displayed on the display screen 110, for example, the displayed content may include one or more information such as a first writing track, an operation record of the function keys 114 (such as the number of deleted words, etc.), remaining capacity information, a connection state of the terminal device (whether the terminal device is connected or not), and the like. When provided with a display 110, the writing instrument 10 may support local review of the writing.

The display screen 110 may be a normal screen or a touch screen, and if the display screen 110 adopts a touch screen, functions such as the function keys 114 and the mode switch 112 may be integrated into the display screen 110, and presented in a virtual key and virtual switch manner.

Optionally, the writing device 10 further includes a communication unit disposed on the body 100, where the specific location is not limited, for example, the communication unit may be disposed inside the pen body 100c of the body 100. The input end of the communication unit is connected with the output end of the processing unit 118, the output end of the communication unit is connected with the input end of the processing unit 118, meanwhile, the output end of the communication unit (when data transmission is needed) can be also connected with the terminal equipment in a communication mode, the processing unit 118 can read the first writing track from the storage unit 120, the read first writing track is sent to the input end of the processing unit 118 from the output end of the storage unit 120, the processing unit 118 sends the first writing track to the input end of the communication unit through the output end of the processing unit 118, and the first writing track is further sent to the terminal equipment by the communication unit.

The terminal device refers to devices other than the writing device 10 itself, for example, may be a PC, a notebook, a tablet, a mobile phone, a server, etc., and the terminal device may further process the first writing track, for example, save the first writing track, display, project the first writing track, etc. It will be appreciated that the communication unit may also be used to send other data than writing tracks to the terminal device and to receive data sent by the terminal device to the writing instrument 10 and to send it to the processing unit 118, in any case with which communication between the writing instrument 10 and the terminal device may be achieved, either in real time or non-real time.

The communication unit may employ either a wireless communication module 116, such as a WiFi module, a bluetooth module, or a mobile communication module, or a wired communication module 126, such as a USB module, or the like, and of course, the communication unit may include both the wireless communication module 116 and the wired communication module 126, so as to support external communication in various environments, as shown in fig. 1.

Optionally, the writing device 10 further includes a battery 124 disposed on the body 100, and the specific location is not limited, for example, the battery 124 may be disposed inside the pen body 100c of the body 100. The battery 124 may be electrically connected to the processing unit 118, the camera 108, the laser emitting unit 106, etc. for powering the components requiring power. In the alternative, the writing instrument 10 may be connected to an external power source through a charging cord instead of the built-in battery 124.

It should be understood that fig. 1 is merely an example, and that even a first type of implementation of the writing instrument is contemplated, it is possible to include more components than shown in fig. 1, or to have a different configuration than that of fig. 1. For example, the writing instrument 10 may also include a projection assembly for projecting the content written by the user onto the surface of the object, for ease of review by the user in a timely manner, and so forth.

Referring to fig. 5, in a second type of implementation, writing instrument 20 includes at least the following components: the camera device comprises a body 200, a camera 208, a processing unit 218, a storage unit 220 and a contact sensing unit 204.

The structure of the body 200 is also divided into a nib 200a, a connecting portion 200b and a pen body 200c, and the description thereof with reference to fig. 1 will not be repeated.

The camera 208 is disposed on the surface of the body 200, and the specific position is not limited, and an output end of the camera 208 is connected to an input end of the processing unit 218, and an input end of the camera 208 is connected to an output end of the processing unit 218. The camera 208 is used for acquiring multiple frame object surface images of the surface of the object to be written when the writing device 20 writes, and sending the acquired object surface images to the input end of the processing unit 218 through its own output end, so as to identify writing tracks inside the processing unit 218. Since the second type of implementation of the writing instrument 20 is still contact, the object being written may choose an object having a certain hardness, or an object having a certain hardness under support. If writing device 20 is not writing, camera 208 may stop capturing images of the object surface to save device power and avoid identifying incorrect writing tracks.

In a second type of implementation of the writing device 20, since no laser emitting unit is included, the camera 208 may be a normal visible camera, and there is no laser spot in the image of the object surface collected by the camera 208, and no trace recognition is performed by the laser spot later, for example, the image of the surface may be as shown in fig. 2 (but without white dots).

Further, since the laser emitting unit is not included, the camera 208 may be disposed on the side of the pen head 200a, there is no concern that the camera 208 blocks the propagation path of the laser, and the image of the object surface collected by the camera 208 may be captured as little as possible by the writing device 20, thereby improving the recognition effect of the writing track. Alternatively, because the nib 200a is smaller, the camera 208 may be a miniature camera, such as a chip-scale camera module, which is not only convenient to integrate with the nib 200a, but also beneficial to reduce implementation costs. Note that for the aforementioned camera 108 and the later-mentioned camera 308, a miniature camera may also be selected.

Regarding the touch sensing unit 204, reference may be made to the relevant description of fig. 1 and will not be repeated. It should be noted that, since the writing device 20 does not include a laser emitting unit, the contact sensing unit 204 only needs to trigger the camera 208 to collect the image of the object surface by the processing unit 218 when sensing that the pen point 200a contacts the surface of the object to be written, and does not need to trigger the laser emitting unit.

In a second type of implementation of writing apparatus 20, processing unit 218 may determine the first writing trace in at least one of the following ways:

(1) The first writing trace is determined according to the position change of the pen point 200a relative to the surface of the object to be written in the multi-frame object surface image.

This way, which is similar to the way the first writing track is determined based on the laser spot, it will be understood in connection with fig. 3 that the writing track is determined based on the movement of the black dots instead of the white dots, the principle of which is not repeated.

It should be noted, however, that the manner (1) requires that the position of the nib 200a be detected from the object surface image, and that the position of the nib 200a is less obvious in the image with respect to the position of the laser spot, and is susceptible to external light and the like, and this problem can be ameliorated by providing the illumination unit 206 in the writing device 20, as will be described later.

(2) A first writing track is determined according to the position change of the characteristic of the surface of the object to be written in the multi-frame object surface image.

Since the movement is relative, the writing means 20 is moving relative to the surface of the object to be written, and the surface of the object to be written is also moving relative to the writing means 20, so that if the movement trace of the feature of the surface of the object to be written can be obtained, the writing trace of the writing means 20 can be obtained as well. During writing, the camera 208 changes the shooting range due to the movement of the writing device 20, so that the same feature on the surface of the object to be written is different in the positions of the multiple frames of object surface images, and the change of the positions reflects the movement track of the feature, which is the principle of recognizing the writing track in the mode (2).

For example, a pit may be provided on the table, considering only the lateral movement of the writing instrument 20, in a first frame the pit is 3 units from the left boundary of the object surface image, and in a fifth frame the pit is 5.5 units from the left boundary of the object surface image, indicating that the pit has moved 2.5 units to the right in the five frames, i.e., the movement trace of the pit is a short traverse of 2.5 (left to right) length, so that it may be inferred that the writing trace of the writing instrument 20 is a short traverse of 2.5 (right to left, opposite direction to the movement trace of the pit).

It should be noted, however, that the method (2) relies heavily on the detection of the characteristics of the surface of the object to be written, and if the surface of the object to be written has no obvious characteristics or is disturbed by external factors such as illumination, the quality of the first writing track obtained by the method (2) will be reduced, and this problem can be improved by providing the lighting unit 206 in the writing device 20, or by adding a gyroscope and an accelerometer, or by deliberately selecting some of the object surfaces with more obvious characteristics for writing.

In addition, the writing trajectory identification in the manner (2) is not dependent on the nib 200a in the object surface image, which also means that in some alternatives of the writing device 20, the shooting range of the camera 208 is allowed to exclude the nib 200a.

Regarding the memory unit 220, reference may be made to the relevant description of fig. 1 and will not be repeated.

Briefly summarizing the writing device 20 in fig. 5, the writing device 20 is a contact type, and can sense the writing behavior of a user on the surface of the object to be written by the contact sensing unit 204, and identify the writing track by the position change of the pen tip 200a relative to the surface of the object to be written or the position change of the feature of the surface of the object to be written in the image of the surface of the object collected by the camera 208.

The contact type writing has the writing sense, is more in line with the writing habit of most users, and is convenient for the users to write more accurately. In addition, the writing device 20 does not need to be matched with special external equipment such as an electronic whiteboard and a signal receiver, and can obtain writing tracks, so that the use cost of a user is reduced. In addition, since the writing device 20 does not use a laser emitting unit, it is advantageous to save the cost and power consumption of the writing device 20. In addition, the writing track is identified by the position of the pen point 200a in the image or the characteristic position of the surface of the object to be written, so that the writing device 20 is not required to leave writing marks, dirt and the like on the surface of the object to be written, the object to be written is kept intact, writing on various objects can be supported, the requirement of traditional writing on paper is avoided, and the trend of environmental protection is met.

Further, the writing device 20 further includes a storage unit 220, and in some implementations, the storage unit 220 may store the first writing track permanently, so that the writing device 20 may be used independently and record the writing track, without relying on any external device, so that the usage scenario of the writing device 20 is greatly widened, and further, the subsequent review of the writing content is also facilitated.

The bus 222, the function keys 214, the display 210, the communication unit (including the wireless communication unit 216 and the wired communication unit 226), and the battery 224 shown in fig. 5 may be implemented with reference to the same-name components in fig. 1, and will not be repeated.

The writing device 20 in fig. 5 does not include a gyroscope and an accelerometer, so that the cost of the device and the power consumption of the device can be saved, but in order to improve the recognition accuracy of the writing track, the gyroscope and the accelerometer can be added, and the installation position and the use of the gyroscope and the accelerometer can be referred to in the related description in fig. 1 and are not repeated.

Optionally, the writing device 20 further includes an illumination unit 206 disposed on the surface of the body 200, and the specific location thereof is not limited. For example, referring to fig. 5, the lighting unit 206 may be disposed at a side of the connection portion 200b, so that the structure of the writing device 20 is more compact and beautiful (the lighting unit 206 does not protrude beyond the column formed by the body 200), and is convenient for the user to hold.

An input of the lighting unit 206 is connected to an output of the processing unit 218, so that the processing unit 218 may send a control signal to the input of the lighting unit 206 through its own output to control the lighting state of the lighting unit 206.

The lighting unit 206 is configured to generate lighting light for illuminating the surface of the object to be written when the writing device 20 writes, and the positions of the lighting unit 206 and the camera 208 are reasonably set, so that the lighting range of the lighting unit 206 covers the shooting range of the camera 208, thereby increasing the brightness of the image of the surface of the object collected by the camera 208, making the position of the pen point 200a in the image, the characteristic position of the surface of the object to be written, and the like more obvious, thereby being beneficial to improving the recognition effect of the writing track, or making the writing device 20 write normally in some darker environments. For example, if the shooting range of the camera is centered on the pen tip 200a, the illumination unit 206 may be oriented to align with the pen tip 200a or a position near the pen tip 200 a.

If writing is not performed by the writing device 20, the illumination unit 206 may stop generating illumination light to save device power consumption.

Since the writing device 20 in fig. 5 is a touch type, whether the illumination unit 206 emits illumination light can be triggered by the touch sensing unit 204, and the triggering manner is similar to that of the touch sensing unit 104 in fig. 1 for the camera 108 and the laser emitting unit 106, and will not be repeated.

The color of the illumination light emitted by the illumination unit 206 is not limited, and may be, for example, white light, yellow light, or the like. Alternatively, the lighting unit 206 may be implemented using a light emitting diode or the like.

Further, the intensity of the illumination light generated by the illumination unit 206 may be constant or variable. For example, in one alternative, the processing unit 218 may control the intensity of the illumination light generated by the illumination unit 206 according to the brightness of the object surface image, for example, if the brightness of the object surface image is low, control the illumination unit 206 to increase the intensity of the illumination light, and if the brightness of the object surface image is high, control the illumination unit 206 to decrease the intensity of the illumination light, or even directly stop emitting light. This adaptively adjusts the luminous intensity of the illumination unit 206, so that the object surface image can be always maintained at a brightness optimum for recognizing the writing trace, thereby improving the recognition effect of the writing trace and also contributing to a reduction in power consumption of the writing device 20.

Optionally, the writing device 20 further includes a mode switch 212 disposed on the surface of the body 200, for example, referring to fig. 5, the mode switch 212 may be disposed on the surface of the pen body 200c of the body 200, so as to facilitate the user's operation. An output of the mode switch 212 is connected to an input of the processing unit 218.

The mode switch 212 is used to trigger the processing unit 218 to switch the operation mode of the writing device 20, and the operation mode of the writing device 20 includes at least a writing mode and an illumination mode. For example, when the mode switch 212 is switched to a certain operation mode, a trigger signal corresponding to the mode may be sent to the processing unit 218, so that the processing unit 218 controls itself and other components to operate in a manner specified by the mode.

Wherein when the writing device 20 is in the writing mode, the illumination unit 206 generates illumination light for illuminating the surface of the object under writing under the triggering of the touch sensing unit 204, and the processing unit 218 determines the first writing trace according to the position change of the pen point in the multi-frame object surface image relative to the surface of the object under writing, or according to the position change of the feature of the surface of the object under writing in the multi-frame object surface image, which has been described above.

When the writing device 20 is in the illumination mode, the illumination unit 206 continuously generates illumination light, and the camera 208 does not capture an image of the object surface. The illumination mode may be used in a dark environment where the writing instrument 20 is functionally similar to a flashlight.

The type of mode switch 212 is not limited and may be, for example, a tact switch, a toggle switch, a virtual switch on a touch screen, and the like.

It should be appreciated that in describing the components contained in the writing apparatus 20 of fig. 5, the repeated parts with fig. 1 may be omitted in detail for simplicity, and reference may be made to the relevant contents of fig. 1 for these omitted details.

It should also be appreciated that fig. 5 is merely an example, and that even a second type of implementation of the writing instrument is contemplated, it is possible to include more components than shown in fig. 5, or to have a different configuration than that of fig. 5.

In addition, although both the first and second types of implementations of the writing device 20 support writing without writing (referred to as ink marks), it is not excluded that in some implementations the nib 200a may be handwriting-retaining, in which case the writing device 20 may perform the writing function of a conventional pen, and may electronically store the writing track in real time. Further, for writing device 20 that may leave a writing, an optical character recognition (Optical Character Recognition, abbreviated OCR) algorithm may also be run on processing unit 218 to effect recognition of the writing trajectory without employing the recognition method described above.

Referring to fig. 6, in a third class of implementations, writing device 30 includes at least the following components: the device comprises a body 300, a camera 308, a laser emitting unit 306, a working switch 312, a processing unit 318 and a storage unit 320.

The structure of the body 300 is also divided into a nib 300a, a connecting portion 300b and a pen body 300c, and the description thereof with reference to fig. 1 will not be repeated.

The camera 308 is disposed on the surface of the body 300, and its specific position is not limited. The output end of the camera 308 is connected with the input end of the processing unit 318, the input end of the camera 308 is connected with the output end of the processing unit 318, the camera 308 is used for collecting multi-frame object surface images of the surface of the object to be written when the writing device 30 writes, and the collected object surface images are sent to the input end of the processing unit 318 through the output end of the camera 308 so as to identify writing tracks in the processing unit 318.

For the non-contact writing device 30, the object to be written may be an object that can form a laser spot on its surface after being irradiated by laser light, and its coverage is wider than that of the contact writing device 30, for example, a window curtain in a falling state is generally not suitable for the contact writing device 30 to write on because it is difficult to apply a force to the window curtain, but for the non-contact writing device 30, there is no obstacle to write on it. The written object surface may refer to a written surface of the written object. The non-contact writing device 30 is different from the conventional pen in writing, the writing device 30 does not need to contact the surface of the object to be written, and only needs to irradiate laser light to the surface of the object to be written, then the writing device 30 is moved according to the blank content to be written, and the movement track of the laser light spot on the surface of the object to be written is the writing track.

If writing device 30 is not writing, camera 308 may stop capturing images of the object surface to conserve device power consumption and avoid identifying incorrect writing trajectories.

The camera 308 may be a conventional visible light camera because it is generally necessary for the non-contact writing device 30 to display a laser spot visible to the user on the surface of the object being written, so that the user is aware that the writing device 30 is currently operating, so that the use of a visible light camera is sufficient to capture an image of the surface of the object containing the laser spot.

Alternatively, referring to fig. 6, a camera 308 may be provided at the front end of the nib 300 a. Since the writing device 30 is non-contact, i.e. the writing device 30 is usually located at a distance from the surface of the object to be written, the writing device 30 itself will not be photographed in the image of the surface of the object captured by the camera 308 after this arrangement, which is advantageous for improving the recognition effect of the writing track.

Further, for the non-contact writing device 30, the body 300 is likely to not include a significant nib 300a (because the writing does not require contact with the surface of the object to be written, and the end of the body 300 does not have to be shaped sharply), and for these non-nib 300a implementations, the camera 308 is disposed at the end of the body 300 to achieve the same effect as in fig. 6.

Of course, if the writing device 30 supports both contact and non-contact (as described in detail below), it is not recommended to dispose the camera 308 at one end of the body 300, because if the writing device 30 is used in contact, the camera 308 cannot capture a clear picture.

Regarding the laser emitting unit 306, reference may be made to the description related to fig. 1, and only the differences from the laser emitting unit 306 in fig. 1 will be described here:

first, the laser light emitted by the laser light emitting unit 306 may be visible light such as blue light, green light, red light, etc., so that the visible light is selected for the reasons explained above.

Secondly, for the contact writing device 30, although the position of the laser emitting unit 306 is not limited in principle, it is not recommended to actually locate the laser emitting unit 306 at the front end of the pen head 300a, because the laser spot is blocked by the pen head 300a, which is disadvantageous for recognizing the writing track according to the laser spot. However, with respect to the non-contact writing apparatus 30, there is no such limitation, and if the body 300 still adopts a structure similar to that of fig. 1, since the front end of the nib 300a is already occupied by the camera 308, the laser emitting unit 306 may be disposed at the side of the connection portion 300b with reference to fig. 1; if the body 300 does not adopt a structure similar to that of fig. 1, for example, the body 300 is a cylinder, the laser emitting unit 306 and the camera 308 may be disposed together at one end of the body 300. Of course, if the writing device 30 supports both contact and non-contact, it is not recommended to dispose the laser emitting unit 306 at one end of the body 300.

The working switch 312 is disposed on the surface of the body 300, and its specific position is not limited, for example, referring to fig. 6, the working switch 312 may be disposed on the surface of the pen body 300c of the body 300, so as to facilitate the user operation. An output of the working switch 312 is connected to an input of the processing unit 318.

The operating switch 312 includes at least two states, an on state and an off state. When the working switch 312 is in the on state, the laser emitting unit 306 continuously generates the laser light for irradiating the surface of the object to be written, and it can be considered that the writing device 30 is writing at this time, for example, when the working switch 312 is switched to the on state (i.e. turned on), a trigger signal can be sent to the input end of the processing unit 318 through the output end of the working switch 312, and the processing unit 318 controls the laser emitting unit 306 to emit the laser light after receiving the signal. When the working switch 312 is in the off state, the laser emission unit 306 stops generating the laser light for irradiating the surface of the object to be written, and it can be considered that writing is finished by the writing device 30 at this time, for example, when the working switch 312 is switched to the off state (i.e., when it is turned off), another trigger signal can be sent to the input end of the processing unit 318 through the output end thereof, and the processing unit 318 controls the laser emission unit 306 to stop emitting the laser light after receiving the signal.

It can be seen that the function of the operation switch 312 is similar to that of the contact sensing unit 104 in fig. 1, except that since the contact cannot be sensed at this time, it is necessary to control the light emitting state of the laser emitting unit 306 by the user operating the operation switch 312.

Alternatively, if the laser emitting unit 306 emits laser light as the writing device 30 is in the writing mode and the laser emitting unit 306 stops emitting laser light as the writing device 30 is in the non-writing mode, the operation switch 312 may be regarded as a mode changeover switch. When the writing device 30 is in the non-writing mode, although writing trace recognition is not performed, it is still possible to perform writing trace review (through the display screen 310) and the like.

The type of the operation switch 312 is not limited, and may be, for example, a tact switch, a toggle switch, a virtual switch on a touch screen, or the like.

The processing unit 318 is configured to determine the first writing track according to the change of the position of the laser spot in the multi-frame object surface image relative to the surface of the object to be written, that is, the principle of implementing the writing track recognition by the third type of implementation of the writing device 30 is the same as that of implementing the first type of implementation of the writing device 30, and specifically, reference is made to the description related to fig. 1, and the description is not repeated.

It is noted that the principle of writing trace recognition based on the laser spot does not relate to whether the writing device 30 contacts the surface of the object to be written, that is, at least part of the implementation of the non-contact writing device 30 may be contact-type or non-contact-type, but the non-contact writing device 30 is triggered by the operation switch 312, not by the contact sensing unit, even if it is used in contact-type.

Regarding the memory unit 320, reference may be made to the relevant description of fig. 1 and will not be repeated.

Briefly summarizing the writing device 30 in fig. 6, the writing device 30 is a non-contact type, and the writing track can be identified by turning on the operation switch 312 to generate laser light for irradiating the surface of the object to be written and by changing the position of the laser spot in the image of the surface of the object collected by the camera 308 with respect to the surface of the object to be written.

Non-contact writing is more free than contact, not only is the choice of the object being written broader, but there is also no strict limitation on the distance to be written, which undoubtedly expands the applicable scenarios of the writing device 30. In addition, the writing device 30 does not need to be matched with special external equipment such as an electronic whiteboard and a signal receiver, and can obtain writing tracks, so that the use cost of a user is reduced. In addition, the writing device 30 recognizes the first writing track by the position change of the laser spot, and since the position of the laser spot in the image is very obvious, it is advantageous to improve the recognition accuracy of the writing track. In addition, since the writing device 30 is non-contact, the writing device 30 is not required to leave writing, dirt and the like on the surface of the written object, so that the written object can be kept intact, writing on various objects can be supported, the requirement of traditional writing on paper is avoided, and the environment protection trend is met.

Further, the writing device 30 further includes a storage unit 320, and in some implementations, the storage unit 320 may store the first writing track permanently, so that the writing device 30 may be used independently and record the writing track, without relying on any external device, so that the usage scenario of the writing device 30 is greatly widened, and further, the subsequent review of the writing content is also facilitated.

For the non-contact writing device 30, if the writing track is recognized directly from the object surface image captured by the camera 308, a problem in the following example may occur:

assuming that a wall has a frame of a fixed size, a word that just occupies the entire frame is written therein by the writing device 30, and although the size of the word is fixed (equal to the size of the frame), if the user writes the word separately at a distance of 10m from the wall and a distance of 1m from the writing device 30, the size of the frame in the object surface image acquired by the camera 308 is different due to the difference in the photographing range of the camera 308, so that if the writing trace is recognized directly from the original object surface image, the size of the recognized word is different (the word recognized from 1m is larger), which is likely not the result desired by the user.

This problem is caused by the different distances of the writing means 30 from the surface of the object to be written, and can be solved by:

first, the processing unit 318 calculates the distance of the writing device 30 from the surface of the object to be written based on the difference between the time at which the laser light is emitted from the laser light emitting unit 306 and the time at which the laser light is received by the camera 308. The laser emission of the laser emission unit 306 may be controlled by the processing unit 318, so that the time processing unit 318 for emitting the laser is known, and the time processing unit 318 for sensing the laser reflected by the surface of the writing object by the camera 308 may also know, so that based on the difference between the two times and the propagation speed of the light, the processing unit 318 may perform laser ranging to obtain the distance between the writing device 30 and the surface of the writing object. It should be understood that it is also possible that writing device 30 is provided with one laser ranging unit alone, where laser emitting unit 306 and camera 308 are simply multiplexed as the laser ranging unit.

The processing unit 318 then scales the multi-frame object surface image according to the distance between the writing device 30 and the surface of the object to be written, and determines the first writing track according to the change in the position of the laser spot in the scaled multi-frame object surface image relative to the surface of the object to be written.

Along the above example, for example, the object surface image acquired at the distance of 1m from the wall is taken as a reference, referred to as a reference image, the object surface image acquired at the distance of 10m from the wall is enlarged according to the calculated distance of 10m, and in the enlarged object surface image, the frame on the wall is the same as the frame on the wall in the reference image in size, so that the first writing track is recognized based on the enlarged object surface image, and the size of the obtained word is the same as the size of the word recognized from the reference image, which accords with the user's expectation.

Alternatively, in zooming the image of the object surface, not only the distance between the writing device 30 and the surface of the object to be written, but also the magnification of the camera 308 may be considered, and in particular, in the case where the camera 308 is zoomable, the magnification is a factor that must be considered.

The gyroscope and accelerometer 302, bus 322, function keys 314, display 310, communication units (including wireless communication unit 316 and wired communication unit 326), and battery 324 shown in fig. 6 may all be implemented with reference to like-named components in fig. 1, and will not be repeated.

It should be appreciated that in describing the components included in the writing device 30 of fig. 6, the parts repeated in fig. 1 or 5 may be omitted in detail for simplicity, and reference may be made to the relevant contents of fig. 1 or 5 for each of these omitted parts.

It should also be appreciated that fig. 6 is merely an example, and that even a third type of implementation of the writing instrument is contemplated, it is possible to include more components than shown in fig. 6, or to have a different configuration than that of fig. 6.

For the non-contact writing device 30, the first writing track may be theoretically determined according to the change of the position of the feature of the surface of the object to be written in the multi-frame object surface image without using the laser emitting unit 306 (the principle is described with reference to fig. 5), but such an implementation may cause the user to make it unclear whether or not he is writing by himself due to the lack of the laser indication. However, such an implementation is also possible if the user is able to see the writing content in real time (e.g. by sending the first writing track in real time to the terminal device for display via the writing means 30).

Fig. 7 is a block diagram of a writing system 40 according to an embodiment of the present application. Referring to fig. 7, the writing system 40 includes a writing device 410 and a terminal device 420 provided in an embodiment of the present application.

The writing device 410 herein may be any of the writing devices previously described and comprising a communication unit, such as the writing devices of fig. 1, 5, 6. The processing unit of the writing means 410 may read the first writing track stored in the storage unit and send it to the terminal device 420 via the communication unit.

The terminal device 420 may display the first writing track after receiving the first writing track (for example, on a display of the terminal device 420 itself), or may control other devices to display the first writing track (for example, control the projector 430 to display it), or the terminal device 420 may display the first writing track itself, and control other devices to display the first writing track. The terminal device 420 may be a PC, a notebook, a tablet, a mobile phone, a server, etc.

In addition to the advantages of the writing device, the writing system 40 may display the first writing track through the terminal device 420 and/or other devices controlled by the terminal device 420, so as to facilitate the writer and other users to observe the writing result, and achieve the communication interaction between the users.

Optionally, with continued reference to fig. 7, the writing system 40 may further include a projector 430, the projector 430 being configured to display at least the first writing trace under control of the terminal device 420.

It should be understood that fig. 7 is merely an example, and that writing system 40 may also include more devices than in fig. 7.

The above description is only an example of the present application and is not intended to limit the scope of the present application, and various modifications and variations will be apparent to those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the protection scope of the present application.

Claims

1. A writing instrument, comprising: the camera is arranged on the body, the output end of the camera is connected with the input end of the processing unit, and the input end of the storage unit is connected with the output end of the processing unit;

the camera is used for collecting multi-frame object surface images of the surface of the object to be written when the writing device writes, and sending the multi-frame object surface images to the processing unit;

the processing unit is used for obtaining a first writing track when the writing device writes according to the multi-frame object surface image, and writing the first writing track into the storage unit.

2. The writing instrument of claim 1, wherein the writing instrument further comprises: the gyroscope and the accelerometer are arranged on the body, and the output end of the gyroscope and the output end of the accelerometer are connected with the input end of the processing unit;

the gyroscope and the accelerometer are used for collecting motion data when the writing device writes and sending the motion data to the processing unit.

3. The writing instrument of claim 2, wherein the body includes a nib, the gyroscope and the accelerometer being both disposed on the nib.

4. A writing device according to any one of claims 1 to 3, wherein the body includes a nib, the writing device further comprising: the contact sensing unit is arranged on the body, the output end of the contact sensing unit is connected with the input end of the processing unit, and the input end of the camera is connected with the output end of the processing unit;

the contact sensing unit is used for triggering the camera to acquire the object surface image through the processing unit when sensing that the pen point contacts the surface of the object to be written.

5. The writing instrument of claim 4, wherein the instrument further comprises: the laser emission unit is arranged on the body, and the input end of the laser emission unit is connected with the output end of the processing unit;

the contact sensing unit is also used for triggering the laser emitting unit to generate laser irradiating the surface of the object to be written through the processing unit when the pen point is sensed to contact the surface of the object to be written.

6. The writing instrument of claim 5, further comprising: the mode change-over switch is arranged on the body, and the output end of the mode change-over switch is connected with the input end of the processing unit;

the mode switching switch is used for triggering the processing unit to switch the working mode of the writing device, and the working mode comprises a writing mode and an indicating mode;

the laser emission unit is used for generating laser irradiating the surface of the written object under the triggering of the contact sensing unit when the writing device is in the writing mode;

when the writing device is in the indicating mode, the laser emitting unit is used for continuously generating laser, and the camera does not collect the image of the object surface.

7. The writing instrument of claim 5, wherein the body further comprises: a pen body and a connecting part;

the diameter of the pen point is smaller than that of the pen body, the connecting portions are respectively connected with the pen point and the pen body, the diameter of the connecting portions gradually increases from one end connected with the pen point to one end connected with the pen body, inclined side faces of the connecting portions are formed, and the camera and the laser emitting units are arranged on the side faces of the connecting portions.

8. The writing instrument of claim 4, further comprising: the lighting unit is arranged on the body, and the input end of the lighting unit is connected with the output end of the processing unit;

the contact sensing unit is also used for triggering the illumination unit to generate illumination light for illuminating the surface of the object to be written through the processing unit when the pen point is sensed to contact the surface of the object to be written.

9. The writing instrument of claim 8, wherein the camera is disposed on a side of the nib.

10. The writing instrument of claim 8, wherein the body further comprises: a pen body and a connecting part;

the diameter of the pen point is smaller than that of the pen body, the connecting portions are respectively connected with the pen point and the pen body, the diameter of the connecting portions gradually increases from one end connected with the pen point to one end connected with the pen body, inclined side faces of the connecting portions are formed, and the illuminating units are arranged on the side faces of the connecting portions.

11. A writing device as in any one of claims 1-3, wherein the writing device further comprises: the laser emission unit is arranged on the body, the input end of the laser emission unit is connected with the output end of the processing unit, and the output end of the working switch is connected with the input end of the processing unit;

The working switch is used for triggering the laser emitting unit to generate laser irradiating the surface of the object to be written through the processing unit when being started.

12. The writing instrument of claim 11, wherein the camera is disposed at one end of the body.

13. A writing device according to any one of claims 1 to 3, wherein the video camera is a chip-scale camera module.

14. A writing device as in any one of claims 1-3, wherein the writing device further comprises: the communication unit is arranged on the body, and the input end of the communication unit is connected with the output end of the processing unit;

the processing unit is further configured to read the first writing track from the storage unit, and send the first writing track to a terminal device through the communication unit.

15. A writing system, comprising: the writing device and the terminal equipment according to any one of claims 1-13, further comprising a communication unit provided on the body of the writing device, an input of the communication unit being connected to an output of the processing unit of the writing device, an output of the communication unit being communicatively connected to the terminal equipment;

The processing unit is further configured to read a first writing track from a storage unit of the writing device, and send the first writing track to a terminal device through the communication unit;

the terminal device is used for displaying the first writing track, and/or controlling other devices except the terminal device to display the first writing track.

16. The writing system of claim 15, wherein the system further comprises: the projector is in communication connection with the terminal equipment;

the projector is used for displaying the first writing track under the control of the terminal equipment.