JPH09102013A - Handwriting character/graphic reader - Google Patents

Abstract

PROBLEM TO BE SOLVED: To highly precisely read handwriting characters/graphics mentioned in various objects by extracting a handwriting mark from an obtained time- sequential picture, recognizing the characters/graphics from the information and outputting the codes and position information. SOLUTION: A picture input part 101 image-picks up an action for writing the characters and the graphics and outputs a time-sequential picture signal. A handwriting extraction part 102 extracts only the picture elements of the handwriting marks from the respective frames of the time sequential picture signal at every moment and outputs handwriting mark information. The handwriting mark information includes position information and time information and expresses them as a coordinate string or as the group of handwriting pictures extracted at prescribed time. A character/graphic recognition part 103 recognizes the characters/graphics based on handwriting mark information from the handwriting extraction part 102 and exchanges them into code information. A code information storage part 104 stores the code information.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a handwritten character / graphic reading device.

[0002]

2. Description of the Related Art A conventional character / graphic reading device uses an input device such as a scanner or a tablet to recognize characters / graphics based on input information. That is, in the case of an OCR (optical character reader) having a scanner,
The front surface of the original is scanned with a scanner to encode the characters and figures written on the surface of the original, and then the characters and figures are recognized. Further, in a character / graphic reading device equipped with a tablet, characters written on the tablet are fetched as time-series information to perform character recognition.

There is also a device for reading characters and figures using a video camera. For example, Japanese Patent Laid-Open No. 61-17577
Japanese Patent Publication No. 6 discloses a device for capturing and coding an image using a video camera.

[0004]

In a character / graphics reading device having a scanner as an input device, the reading target is limited. That is, there is a problem that only characters / graphics written on the plane of an object that can be placed on the reading surface of the scanner or characters / graphics written on a sheet-shaped object such as paper can be read. In addition, this device has a problem that it is difficult to recognize characters and figures written in close proximity to each other. Further, this device has a problem that the device is relatively large and inferior in variability.

Further, in a character / graphic reading device having a tablet as an input device, it is more difficult to write characters / graphics than paper and the written characters / graphics are difficult to see. In addition, this device has a problem that only what is written on the tablet can be read.

On the other hand, in characters and graphics reading apparatus using an image pickup device such as a video camera, it is possible to read even characters and graphics written on what kind of object, including the blackboard or the like, still There is a problem that it is difficult to separate and recognize adjacent characters and figures.

The present invention provides a handwritten character / figure reading device capable of reading characters / figures described on any object, and capable of highly accurately recognizing handwritten characters / figures. The purpose is to provide.

[0008]

According to the present invention, an image input means for acquiring a time series image, a handwriting extraction means for extracting a handwriting from the time series image and outputting handwriting information, and the handwriting Character / figure recognition means for recognizing characters / figures from information and outputting character / figure codes and position information,
There is provided a handwritten character / graphic reading device having a character / graphic storage means for storing the character / graphic code and position information.

More specifically, according to the present invention, the handwriting extraction means monitors a pixel value of the time-series image and extracts a pixel which maintains the value for a predetermined time after the pixel value changes. A handwriting candidate pixel extraction unit that monitors the pixel value of the time-series image and extracts a pixel that changes again within a predetermined time after the pixel value changes, and the handwriting candidate pixel A handwritten character / figure reading device having a handwriting output unit that determines a true handwriting based on the outputs of the extraction unit and the writer candidate pixel extraction unit and outputs the handwriting information as the handwriting information. available.

Alternatively, the handwriting extraction means accumulates the handwriting information extracted up to one hour before and stores it as a cumulative handwritten image, and a predetermined one in order from the latest one of the time series images. An image memory that stores only the number of frames, a stable black pixel extraction unit that extracts black pixel positions whose values do not change in all images stored in the image memory, and an image obtained from the output of the stable black pixel extraction unit When,
A handwritten character / graphic reading device that obtains a difference from the cumulative handwriting image and outputs it as the handwriting information is obtained.

Alternatively, the handwriting extraction means compares the time-series image and the cumulative handwritten image with a cumulative handwritten image storage unit for accumulating the handwriting information extracted up to one hour before and storing it as a cumulative handwritten image. A change area extracting section for extracting areas having different pixel values, a writer extracting section for extracting an area representing the writing instrument and the body of the writer from the output of the change area extracting section, and an output of the change area extracting section. A handwritten character / graphic reading device is provided, which has a new handwriting extraction unit that extracts a new handwriting based on the output of the writer extraction unit and outputs it as the handwriting information.

Further, the handwriting extraction means stores a handwriting candidate image generated one hour before and a cumulative handwriting image storage unit for accumulating the handwriting information determined one hour before and storing it as a cumulative handwriting image. A handwriting candidate image storage unit to be stored, a change region extraction unit that extracts a region having different pixel values by comparing the time-series image and the cumulative writing image, and a writing instrument and a writer's output from the output of the change region extraction unit. A writer extraction unit that extracts a region representing a body, an unconfirmed handwriting extraction unit that extracts an unconfirmed handwriting trace based on the output of the change region extraction unit and the output of the writer extraction unit, and the unconfirmed Based on the output of the handwriting extraction unit and the handwriting candidate image based on the handwriting candidate image, the handwriting determination unit that outputs as the writing information, based on the output of the handwriting extraction unit and the handwriting candidate image, The writer and the A concealed handwriting extraction unit that extracts unconfirmed handwriting concealed by a writing instrument, and a handwriting candidate image that generates the handwriting candidate image based on the output of the concealed handwriting extraction unit and the output of the unconfirmed handwriting extraction unit. It is possible to obtain a handwritten character / graphic reading device characterized by having a generation unit.

Further, according to the present invention, the handwriting extraction unit accumulates the handwriting information and stores it as a cumulative handwriting image, and a two-dimensional position of the tip of the writing instrument from the time series image. A writing instrument tracking unit that extracts a change in the pixel corresponding to the position of the writing instrument extracted by the writing instrument tracking unit, a pixel change detection unit that detects based on the cumulative handwriting image and the time-series image, and the pixel It is characterized by having a mode determination unit that determines whether or not writing is performed based on the output of the change detection unit, and outputs the position of the writing unit as the handwriting information when writing is performed. A handwritten character / graphic reader can be obtained.

[0014]

According to the handwritten character / graphic reading device of FIG. 1, it is possible to recognize characters / graphics written on a paper surface, a blackboard, etc., so that the handwritten character / graphic reading device having a tablet can be used. Easy to use. In particular, since a time-series image is obtained using a video camera, two types of information, the writing position and the writing time, can be acquired, so even if the characters and figures written by the writer are close to each other, high accuracy is achieved. Characters and figures can be separated and recognized.

In the handwriting extraction unit of FIG. 4, attention is paid to changes in pixel values in the time axis direction in a time series image obtained from a video camera. If the proof condition is constant, it is considered that the change of the pixel value in the time axis direction is caused by the handwriting of the writing instrument or the movement of the writing instrument's body or writing instrument. Therefore, among the pixels whose pixel values have changed, a line segment pixel group whose pixel values do not change again even after a lapse of a predetermined time is extracted, so that only the handwriting trace can be obtained.

The handwriting extractor in FIG. 5 compares the handwriting image extracted up to one time before with the input image at the current time to obtain the area where the pixel value has changed. Since this area should include the newly written handwriting and the writer's body, etc., it is newly written by removing the portion corresponding to the writer (obtaining the difference) from this area. Only handwriting traces can be extracted. Here, in order to obtain the change area, the difference between the time series image and the cumulative handwriting image may be obtained. Further, the writer's part can extract the writer and the writing implement by performing the contraction processing to the extent that the writing trace disappears and the expansion processing of the same number on the area where the pixel value has changed.

The handwriting extraction unit of FIG. 8 compares the handwriting image extracted up to one hour before and the input image at the current time to obtain a region in which the pixel value has changed, and a new writing location is extracted from this region. To extract candidates. By performing this processing every time and extracting the writing candidate portions found at two consecutive times (calculating the logical product), the true writing portions can be obtained. Here, in order to obtain the change area, the difference between the time-series image and the accumulated handwriting image may be obtained, as in the case of FIG. Further, the writer's part can extract the writer and the writing implement by performing the contraction processing to the extent that the writing trace disappears and the expansion processing of the same number on the area where the pixel value has changed.

In the handwriting extraction unit of FIG. 9, only the handwriting can be acquired by extracting the pixels that keep the pixel value equal to the handwriting for a predetermined time. This can extract even a relatively thick handwriting trace as compared with the handwriting extraction unit of FIG.

In the handwriting extraction section of FIG.
After obtaining the three-dimensional position, whether or not writing is in progress can be determined from the manner in which the pixel values change as the writing implement passes. Therefore, a handwriting trace can be obtained by extracting the position of the writing tool when it is determined that writing is in progress.

In the handwriting extractor of FIG. 14, regarding the two writing pixels extracted at consecutive times, points on the line segment connecting the two pixels are also treated as writing pixels.
Even if the writing speed is high, the handwriting pixels necessary for recognition can be extracted.

In the handwritten character / graphic reading apparatus of FIG. 15, it is possible to freely switch between the original drawing mode and the gesture input mode by determining that the movement of the writing instrument which is not writing indicates a gesture.

[0022]

Embodiments of the present invention will be described below with reference to the drawings. Fig. 1 shows the handwritten characters of the present invention.
1 shows an embodiment of a figure reading device. The handwritten character / graphic reading apparatus of FIG. 1 includes an image input unit 101 that acquires a time-series image of a video camera or the like, and a handwriting extraction unit 102 that extracts only pixels of a brush trajectory from the time-series image and outputs handwriting information. And a character / graphic recognition unit 103 for recognizing characters / graphics based on handwriting information to create character / graphic code information, and a code information storage unit 104 for storing character / graphic code information.

The image input unit 101 captures a motion of a writer writing a character, a graphic or the like, and outputs a time-series image signal. Here, it is assumed that the characters / figures are written on a paper surface or a blackboard (white board), and the image input unit is fixed toward the paper surface or the blackboard so that the handwriting can be captured. To do.

The handwriting extraction unit 102 extracts only the handwriting pixels from each frame of the time-series image signal input from the image input unit 101, and outputs handwriting information.
The handwriting information includes position information and time information, and is expressed as a coordinate sequence or a series of handwriting images extracted at a certain time.

The code information storage unit 104 has a memory and stores the code information from the character / graphic recognition unit 103.

The character / graphics recognizing unit 103 is a handwriting extracting unit 1.
Characters and figures are recognized based on handwriting information from 02. Then, the recognized character / graphic is converted into code information and output. Since the handwriting information from the handwriting extracting unit 102 includes the position and time information of the pixel on which the character / graphic is written, for example, two characters are in contact as shown in FIG. Then, even if it is difficult to separate them, it is possible to easily perform segmentation as shown in FIG. 3 by using time information together. That is, the handwriting information indicates that the character “contact” 301 is, for example, from time t = 1 to t.
= 2, and the character "touch" 302 includes, for example, information that the character was touched between times t = 2 and t = 3, so that these characters can be separated and recognized. In addition,
The recognition here may be performed every moment, or may be performed after the layout analysis is performed after all the writing information is obtained.

Next, a more detailed embodiment of the handwriting extraction unit 102 will be described. FIG. 4 shows a first embodiment of the handwriting extraction unit 102. The handwriting extraction unit 102 includes a handwriting candidate pixel extraction unit 401, a writer candidate pixel extraction unit 402,
And a handwriting output unit 403.

The handwriting candidate pixel extraction unit 401 writes, from the input time-series image signal, a line segment pixel group that keeps the pixel value immediately after the change for a predetermined time among the pixels whose pixel values have changed over time. It is extracted as a candidate site for traces.

Further, the writer candidate pixel extraction unit 402 selects, from the input time-series image signal, a writer or a pixel whose pixel value has changed within a predetermined time, among pixels whose pixel value has changed with time. It is extracted as a pixel candidate corresponding to the writing instrument.

The handwriting output unit 403 outputs the information output by the handwriting candidate pixel extraction unit 401 and the handwriting candidate pixel extraction unit 40.
The information output by 2 is integrated to extract only the true handwriting. The extracted true handwriting trace is the character / figure recognition unit 10
3 and the handwriting candidate pixel extraction unit 401.

If the accuracy of the handwriting extraction unit 102 is allowed to decrease, the handwriting extraction unit 102 can be downsized by only including the handwriting candidate pixel extraction unit 401.

Next, referring to FIG. 5, the handwriting extraction unit 102
The second embodiment will be described. This handwriting extraction unit 102
Includes a changed area extraction unit 501, a writer extraction unit 502, a new handwriting extraction unit 503, and a cumulative handwriting image storage unit 504.

The change area extracting section 501 is provided with the image input section 10.
The difference between the time-series image from 1 and the cumulative handwriting image stored in the cumulative handwriting image storage unit 504 is obtained, and the difference is binarized to obtain an image including a newly written portion and a writer's body part. Output.

The writer extractor 502 eliminates the handwriting trace from the output image of the change region extractor 501 based on the assumption that the pixel group corresponding to the handwriting of the writing instrument is finer than the pixel group corresponding to the body. Shrink treatment is performed as many times as necessary. After that, the expansion process is performed the same number of times to extract pixels corresponding to the body part (writing tool) of the writer.

The new handwriting extraction unit 503 extracts only new handwriting by obtaining the difference between the output image of the changed area extraction unit 501 and the output image of the writer extraction unit 501, and extracts this new handwriting. 103 and cumulative handwriting image storage unit 50
4 is output.

The cumulative handwriting image storage unit 504 generates a cumulative handwriting image in which handwritings are accumulated from the output of the new handwriting handwriting extraction unit 503 and the pixel value of the time-series image.

The operation of the handwriting extractor 102 of FIG. 5 can be easily understood with reference to FIGS. 6 and 7. That is, as shown in FIGS. 6 (a) and 7 (a), the input image input to the handwriting extraction unit 102 (output image of the image input unit 101) shows the operation of writing the hiragana “a”. Assuming that images are taken at a predetermined sampling time interval, the change area extraction unit 50 corresponding to each time (t = 0 to t = 9)
The output image of No. 1 is as shown in FIGS. 6B and 7B. That is, an image representing a difference from the accumulated handwriting image at the previous time stored in the accumulated handwriting image storage unit 504 is obtained. The writer extraction unit 502 performs contraction / expansion processing on the image and extracts an image of only the writing instrument as shown in FIGS. 6C and 7C. Then, the new handwriting extraction unit 5
03 obtains the difference between the output image of the change area extraction unit 501 and the output image of the writer extraction unit 501, and as shown in FIGS. 6D and 7D, only new handwriting Output the image. Finally, the cumulative handwriting image storage unit is shown in FIG.
An accumulated handwriting image as shown in (e) and FIG. 7 (e) is generated and stored together with the time information.

In FIG. 6 and FIG. 7, the sampling time interval is set large for the sake of convenience of explanation, so that a large number of handwriting marks newly appearing at each time are extracted.
In practice, the sampling time interval is made finer to obtain more precise handwriting information. This is because if the sampling interval is wide, the written characters are more likely to be hidden by the writer's body or the like, and the handwriting is likely to jump, as in the image at time t = 7 in FIG. 7. is there. In addition, in order to suppress the occurrence frequency of such a phenomenon, in addition to making the sampling time interval fine,
It is also important to pay attention to the camera position of the image input unit 101 so that the handwriting trace is not hidden by the writer's shadow. Even if the handwriting is skipped, as can be seen from the image at time t = 8 in FIG. 7, the pixels that have been skipped are found over time, so the handwriting information is corrected so that the jump does not occur. It is also possible to recognize characters / figures using an off-line character recognition method.

If the handwriting once written is erased or noise is erroneously recognized as handwriting, the handwriting is extracted again from the new handwriting extraction unit 503 at the time when the handwriting is deleted and the time when the noise disappears. However, by referring to the pixel value in the cumulative handwriting image and the pixel value in the input time-series image, it is possible to determine whether or not the pixel is written. If the pixel disappears, the cumulative handwriting image storage unit 504 may erase the corresponding pixel, and the character / graphic recognition unit 103 may discard the corresponding character / graphic recognition result.

Next, a third embodiment of the handwriting extractor 102 will be described with reference to FIG. This handwriting extractor 1
Reference numeral 02 designates a handwriting extract more accurately than that shown in FIG. 5. The change region extracting unit 801, the writer extracting unit 802, the unconfirmed handwriting extracting unit 803, and the handwriting determining unit 80.
4, concealed handwriting extraction unit 805, handwriting candidate image generation unit 80
6, a cumulative handwriting image storage unit 807 and a handwriting candidate image storage unit 808. The operation of the handwriting extractor 102 will be described below.

First, the change area extraction unit 801 newly writes from the time series image input from the image input unit 101 and the cumulative handwriting image extracted from the cumulative handwriting image storage unit 807 one hour before. Outputs an image that includes the selected part and the body of the writer.

The writer extraction unit 802 is a change area extraction unit 8
The output image of 01 is subjected to the contraction processing for the number of times such that the handwriting disappears, and then the expansion processing is performed for the same number of times to extract only the pixels corresponding to the body part.

The undecided handwriting extraction unit 803 obtains the difference between the output image of the writer extraction unit 802 and the output image of the changed area extraction unit 701 and extracts only new writing handwriting.

The handwriting confirmation unit 804 is an unconfirmed handwriting extraction unit 8
From 03, it is assumed that the image contains a slight noise component, and in order to extract only the handwriting more accurately, the handwriting candidate image extracted one time ago and stored in the handwriting candidate image storage unit 808 and the undetermined A logical product with the output image is obtained from the handwriting extraction unit 803, and the resulting image is output to the character / graphic recognition unit 103 and the cumulative handwriting image storage unit 807 as a new handwriting at this time.

As described above, the handwriting determination unit 804 determines the handwriting extracted at two consecutive times as the true handwriting. For this reason, if the handwriting extracted at the previous time is hidden behind the writer or the like at the current time, it will not be determined as a true handwriting. Therefore, the concealed handwriting extraction unit 805 uses the handwriting candidate image storage unit 8
Based on the handwriting candidate image stored in 08 and the image from the writer extraction unit, the pixels of the handwriting hidden by the writer at the current time are extracted. Then, the handwriting candidate image generation unit 806 obtains the sum of the image from the concealed handwriting extraction unit 805 and the image from the unconfirmed handwriting extraction unit 803, and determines it as the handwriting candidate image at the current time (used at the next time). , To the handwriting candidate image storage unit 808. Thus,
Even if there are handwriting pixels hidden in the shadow of the writer, the confirmation process can be postponed to the next time in sequence, so it should be confirmed as a true handwriting when the pixels appear again. You can

The operation of the handwriting extractor 102 of FIG. 8 can be easily understood with reference to FIGS. 9 and 10. Here, FIG. 9A and FIG.
01 is an input image, FIGS. 9B and 10B are output images of the changed area extracting unit 801, FIGS. 9C and 10B.
9C is an output image of the writer extraction unit 802, FIGS. 9D and 10D are output images of the undetermined handwriting extraction unit 803,
9 (e) and 10 (e) are output images of the handwriting determination unit 804, FIGS. 9 (f) and 10 (f) are output images of the concealed handwriting extraction unit 805, FIGS. 9G shows the output image of the handwriting candidate image generation unit 806, and FIGS. 9H and 10H show the cumulative handwriting image stored in the cumulative handwriting image storage unit 808, respectively.

With the passage of time, FIG. 9A and FIG.
When the image shown in FIG. 9A is input, the changed area extraction unit 801 obtains the difference from the accumulated handwritten image one time before, and the change area extraction unit 801 shown in FIG.
The images shown in (b) and FIG. 10 (b) are output. The writer extraction unit 802 performs contraction / expansion processing on each of the images in FIGS. 9B and 10B, and an image of only the writing instrument, that is, the images shown in FIGS. 9C and 10C. To get The undetermined handwriting extraction unit 803 displays an image showing the difference between FIGS. 9B and 10B and FIGS. 9C and 10C at each time, that is, FIG. The image shown in FIG. 10D is output. The handwriting determination unit 804 displays the current time of FIG.
The logical product of the images shown in (d) and FIG. 10 (d) and the images shown in FIG. 9 (g) and FIG. ) Output the image shown in. The concealed handwriting extraction unit 805 displays the images shown in FIGS. 9B and 10B at the current time and the image one time before that.
(G) and the image shown in FIG.
The images shown in FIGS. 9F and 10F are output. The handwriting candidate image generation unit 806 is shown in FIGS.
The sum of the image of (d) and the images of FIGS. 9 (f) and 10 (f) is obtained, and the images shown in FIGS. 9 (g) and 10 (g) are output. The cumulative handwriting image storage unit 807 sequentially accumulates the images of FIGS. 9E and 10E to obtain the images of FIGS.
The image of 0 (h) is output.

As described above, in the handwriting extraction unit 102 of FIG. 8, it takes two hours to set the handwriting, and therefore the handwriting is set one hour later than the actual writing time.

In the handwriting extraction unit 102 as well,
The process when noise is erroneously determined as handwriting or the handwriting is deleted can be realized by the same method as the handwriting extracting unit in FIG.

Next, a fourth embodiment of the handwriting extraction unit 102 will be described with reference to FIG. This handwriting extractor 1
02 is an image memory 1001 and a stable black pixel extraction unit 100.
2, a difference image generation unit 1003, and a cumulative handwriting image storage unit 1004.

The image memory 1001 has an image input section 101.
Sequentially receives images from. The received images are all stored until a predetermined number is reached, and thereafter, the oldest image is erased each time a new image is input.

The stable black pixel extraction unit 1002 extracts black pixels whose values do not change in all the images stored in the image memory 1001. Here, a black pixel whose value does not change means
A change in value does not exceed the threshold. By extracting only black pixels whose values do not change in this way, objects that move over time such as the writer's arm or writing instrument are removed, and objects that do not move over time, that is, characters, figures, etc. Images can be obtained. However, here, the image input unit 101 is set so that the writing trace by the writing tool is observed as a dark pixel. For example, when reading characters written in black on a blackboard, the image input unit 101 performs black-and-white inversion processing.

The difference image generation unit 1003 obtains the difference between the image from the stable black pixel extraction unit 1002 and the cumulative handwriting image stored in the cumulative handwriting image storage unit, and extracts a new handwriting portion. Then, the extracted handwriting image storage unit 9 stores the extracted images.
04 to the character / figure recognition unit 103.

FIG. 12 shows an input image and an image stored in the cumulative handwriting image storage unit 1004 when the image memory 1001 can store images for four frames.

The handwriting extraction unit 102 utilizes that the value of the pixel corresponding to the handwriting is stable, unlike the pixel corresponding to the moving body part. Therefore, if the sampling time interval is set too long, the handwriting may be hidden by the writer, and if it is too short, the handwriting may be erroneously extracted as the handwriting. Therefore, depending on the installation position of the video camera, a sampling time interval of 1 second to 10 seconds is appropriate.

Next, referring to FIG. 13, the handwriting extraction unit 102
The fifth embodiment will be described. The handwriting extraction unit 102 includes a writing instrument tracking unit 1301 and a pixel change detection unit 13
02, a mode determination unit 1303, and a cumulative handwriting image storage unit 1304.

The writing implement tracking unit 1301 identifies the two-dimensional position of the writing implement in the input image based on the characteristics such as the shape and color of the writing implement, and outputs the position coordinates.

The pixel change detection unit 1302 detects a change in pixel value corresponding to the writing tool tip position in the image. At this time, the pixel change detection unit 1302 refers to the cumulative handwriting image stored in the cumulative handwriting image storage unit 1304 as the pixel before the writing tool passes. In addition, as the pixel after the writing instrument passes, one hour later, the writing instrument tracking unit 130
Reference the image obtained via 1. Then, the pixel change detection unit 1302 classifies the change in the pixel value at the tip position of the writing section into the following four patterns, and outputs the coordinate value of the tip position of the writing tool and the pattern number indicating the pixel value change as a pair. . Note that the four patterns are a change from a pixel determined to be unwritten to a pixel determined to be unwritten (no change), a change from a pixel determined to be unwritten to a pixel determined to be written, A change from a pixel determined to be written to a pixel determined to be written (no change), and a change from a pixel determined to be written to a pixel determined to be unwritten.

The mode determination unit 1303 determines the writing mode at the current time based on the output of the pixel change detection unit 1302. Here, the above can be specified as being in the state of not being written, being being written, and being being erased, but cannot be specified. The situation of 1 represents the case where the writing instrument crosses the line already drawn, etc., and it is necessary to judge from the pixel value whether the writing instrument is currently writing or is simply moving upward. Because you can't. Therefore, the mode determination unit 1303
When the pixel change pattern at the current time is, the pixel change pattern one time before is inherited. That is, when the pixel change pattern at the previous time is, it is determined that the writing is not in progress. If the pixel change pattern at the previous time is, it is determined that writing is still in progress. At this time, even if a judgment error occurs, the written line segment is very thin when viewed from the entire image, and the length of the line segment written in one sampling time is also limited. It is considered that very few pixels receive. Also, if the writing tool only crosses over the line segment that has already been written, the writing tool immediately moves to an unwritten pixel, and the correct judgment can be made, which affects the recognition of characters and figures. There is no such thing. However, in this embodiment, it is necessary to pay attention to the position of the video camera so that the tip of the writing instrument is not hidden by the writing instrument itself or the body.

In this way, the mode determination unit 1303 determines that writing is in progress only in the case of and and when the previous time is, and the tip position coordinate of the writing tool is determined by the character / graphic recognition unit 10.
3 and the accumulated handwriting image storage unit 1304.

The cumulative handwriting image storage unit 1304 adds the input tip position of the writing tool as a new writing position to the cumulative handwriting image to be stored, and generates and stores a new cumulative handwriting image.

Next, referring to FIG. 14, the handwriting extraction unit 102
A sixth embodiment will be described. This handwriting extraction unit 102 is similar to the handwriting extraction unit of FIG.
Take the configuration that is connected.

The handwriting interpolation unit 1401 has a mode decision unit 13
Based on the tip position information of the writing instrument output from 03, information is generated in which the pixels located on the straight line connecting the tip positions of the writing instrument at two consecutive times are also the writing pixels. Cumulative handwriting image storage unit 1
Output to 304. According to this configuration, the writing implement moves at high speed (there is a long sampling time interval), and even if the detected tip position of the writing implement is not continuous, it is possible to approximate the handwriting during that time, making it easy to recognize characters and figures. To

Next, another embodiment of the present invention will be described with reference to FIG. This handwritten character / graphic reading apparatus includes an image input unit 1501 having a video camera, a handwriting extraction unit 1502 for extracting the tip position of a writing instrument from a self-series image input from the image input unit 1501, and a handwriting extraction unit 1502. A character / figure recognition unit 1503 that recognizes a character / figure from the output handwriting information, a gesture recognition unit 1504 that recognizes a gesture from the handwriting information output by the handwriting extraction unit 1502, and code information storage that stores character / graphic code information. Section 1505.

After extracting the writing tool tip position, the handwriting extraction unit 1502 outputs the writing tool tip position to the character / graphic recognition unit 1503 when it recognizes that writing is in progress, and when it recognizes that writing is not in progress. , And outputs the position of the writing instrument to the gesture recognition unit 1504. This allows the writer to freely perform writing and gestures without performing operations such as mode switching. It is also possible to input a command as shown in FIG. 16 by a gesture. However, it is necessary to set the gesture command so that the unconscious motion during writing is not recognized as the gesture command.

[0066]

According to the present invention, it is possible to highly accurately recognize a cutout of a character or a figure by extracting a change point of a pixel value from a time-series image or extracting a movement of a writing instrument every moment. You can As a result, characters written on paper, a blackboard, or the like can be read using the time information.

Further, it is possible to automatically judge whether or not writing is in progress by checking whether or not the pixel value of the pixel where the tip of the writing utensil has passed above has changed before and after passing through the writing utensil. You can

[Brief description of the drawings]

FIG. 1 is a block diagram showing an embodiment of the present invention.

FIG. 2 is a manuscript diagram for explaining the operation of the handwriting extractor in FIG.

FIG. 3 is an output conceptual diagram of the handwriting extraction unit of FIG. 1 when the document of FIG. 2 is read.

FIG. 4 is a block diagram showing a detailed first embodiment of a handwriting extraction unit in FIG.

5 is a block diagram showing a detailed second embodiment of the handwriting extraction unit in FIG. 1. FIG.

6 is a conceptual diagram showing a relationship between an output image of each unit and time for explaining the operation of the handwriting extraction unit of FIG.

FIG. 7 is a conceptual diagram showing the relationship between the output image of each unit and time for explaining the operation of the handwriting extraction unit of FIG.

8 is a block diagram showing a detailed third embodiment of the handwriting extraction unit in FIG. 1. FIG.

9 is a conceptual diagram showing a relationship between an output image of each unit and time for explaining the operation of the handwriting extraction unit of FIG.

FIG. 10 is a conceptual diagram showing a relationship between an output image of each unit and time for explaining the operation of the handwriting extraction unit of FIG.

11 is a block diagram showing a detailed fourth embodiment of the handwriting extraction unit in FIG. 1. FIG.

FIG. 12 is a conceptual diagram showing the relationship between an input image and an output image for explaining the operation of the handwriting extraction unit in FIG.

13 is a block diagram showing a detailed fifth embodiment of the handwriting extraction unit in FIG. 1. FIG.

FIG. 14 is a block diagram showing a detailed sixth embodiment of the handwriting extraction unit in FIG.

FIG. 15 is a block diagram showing another embodiment of the present invention.

16 is a diagram showing an example of a gesture command used in the handwritten character / graphic reading device of FIG.

[Explanation of symbols]

 101 Image Input Unit 102 Handwriting Extraction Unit 103 Character / Graphic Recognition Unit 104 Code Information Storage Unit 401 Handwriting Candidate Pixel Extracting Unit 402 Writing Candidate Pixel Extracting Unit 403 Writing Hand Output Unit 501 Change Area Extracting Unit 502 Writing Hand Extracting Unit 503 New Handwriting Extraction unit 504 Cumulative handwriting image storage unit 801 Change area extraction unit 802 Written person extraction unit 803 Undetermined handwriting extraction unit 804 Handwriting confirmation unit 805 Concealed handwriting extraction unit 806 Handwriting candidate image generation unit 807 Cumulative handwriting image storage unit 808 Handwriting candidate image Storage unit 1001 Image memory 1002 Stable black pixel extraction unit 1003 Difference image generation unit 1004 Cumulative handwriting image storage unit 1301 Writing instrument tracking unit 1302 Pixel change detection unit 1303 Mode determination unit 1304 Cumulative handwriting image storage unit 1401 Handwriting interpolation unit 1501 Image input unit 1502 Handwriting extraction 1503 characters and graphics recognition unit 1504 gesture recognition unit 1505 code information storage unit

Claims (20)

[Claims] 1. An image inputting means for acquiring a time-series image, a handwriting extracting means for extracting a handwriting from the time-series image and outputting handwriting information, and a character / figure for recognizing a character / figure from the handwriting information. A handwritten character / graphic reading device comprising a character / graphic recognition means for outputting a graphic code and position information, and a character / graphic storage means for storing the character / graphic code and position information. 2. The handwriting extracting means monitors a pixel value of the time-series image and determines a pixel which maintains the value for a predetermined time after the pixel value changes as a pixel corresponding to the handwriting. The handwritten character / graphic reading device according to claim 1. 3. A handwriting candidate pixel extraction unit, wherein the handwriting extraction means monitors a pixel value of the time-series image and extracts a pixel that maintains the value for a predetermined time after the pixel value changes; A writer candidate pixel extraction unit that monitors the pixel value of the series image and extracts a pixel that changes again within a predetermined time after the pixel value changes, the handwriting candidate pixel extraction unit, and the writer candidate pixel extraction unit The handwritten character / graphic reading apparatus according to claim 1 or 2, further comprising: a handwriting output unit that determines a true handwriting based on the output of the handwriting and outputs the handwriting information as the handwriting information. 4. A cumulative handwriting image storage unit for accumulating the handwriting information extracted up to one hour before and storing the handwriting information as a cumulative handwriting image, and a predetermined one in order from the latest one of the time series images. An image memory that stores only the number of frames, and a stable black pixel extraction unit that extracts black pixel positions whose values do not change in all images stored in the image memory,
3. The handwritten character / graphic reading device according to claim 1, wherein a difference between an image obtained from the output of the stable black pixel extraction unit and the cumulative handwriting image is obtained and is output as the handwriting information. 5. The cumulative handwriting image storage unit that the handwriting extraction means accumulates the handwriting information extracted up to one hour before and stores it as a cumulative handwritten image, and compares the time series image and the cumulative handwritten image. A change area extraction unit that extracts areas having different pixel values, and a writer extraction unit that extracts an area representing the writing instrument and the writer's body from the output of the change area extraction unit,
The handwriting extractor according to claim 1, further comprising: a new handwriting extracting unit that extracts a new handwriting based on the output of the change area extracting unit and the output of the writer extracting unit and outputs the new handwriting as the writing information. Character / graphic reader. 6. When the change area extracting unit compares the time-series image with the cumulative handwriting value image, if the difference in pixel value between corresponding pixels is less than a predetermined threshold value. 6. If the value is “0”, the value is “1” if it is equal to or more than the threshold value, and the difference image is output.
Handwritten character / graphic reader 7. The writing instrument and the writing instrument, wherein the writer extraction unit performs a predetermined number of contraction processes and an equal number of expansion processes on an output image from the change region extraction unit or an image obtained from the output. The handwritten character / graphic reading device according to claim 5, wherein an area representing a person's body is extracted. 8. The new handwriting information section generates a difference image between an image obtained from the output of the change area extraction section and an image obtained from the output of the writer extraction section, and the value is within a predetermined range. 6. The handwritten character / graphics reading apparatus according to claim 5, wherein the position information of the pixel is output as the writing information. 9. The accumulated handwriting image storage unit acquires position information of a new handwriting portion from the handwriting information from the new handwriting extraction unit, and a pixel value of a pixel indicated by the position information from the time series image. 6. The handwritten character / graphics reading device according to claim 5, wherein the handwritten character / graphic reading device acquires the characters and sequentially adds them to the cumulative image. 10. A cumulative handwriting image storage unit that the handwriting extraction means accumulates the handwriting information determined up to one hour before and stores it as a cumulative handwritten image, and stores a handwriting candidate image generated one time ago. A handwriting candidate image storage unit, a change region extraction unit that extracts a region having different pixel values by comparing the time-series image and the cumulative writing image, and a writing instrument and a writer's body from the output of the change region extraction unit. And an unconfirmed handwriting extraction unit that extracts an unconfirmed handwriting trace based on the output of the change region extraction unit and the output of the writer extraction unit, and the unconfirmed handwriting. Based on the output of the extraction unit and the handwriting candidate image based on the handwriting determination unit that determines a new handwriting location based on the handwriting candidate image, the handwriting determination unit that outputs the handwriting information, and the handwriting extraction unit For the writer and the writing instrument Therefore, the concealed handwriting extraction unit that extracts the concealed unconfirmed handwriting, and the handwriting candidate image generation unit that generates the handwriting candidate image based on the output of the concealed handwriting extraction unit and the output of the unconfirmed handwriting extraction unit. The handwritten character / graphic reading device according to claim 1, further comprising: 11. When the change area extracting unit compares the time-series image with the cumulative handwriting value image, if the difference in pixel value between corresponding pixels is less than a predetermined threshold value. If "0" is given, if it is above the threshold value, "1" is given,
11. The handwritten character / graphic reading device according to claim 10, wherein the handwritten character / graphic reading device outputs the difference image. 12. The writing instrument and the writing instrument, wherein the writer extraction unit performs a predetermined number of contraction processes and an equal number of expansion processes on an output image from the change region extraction unit or an image obtained from the output. 11. The handwritten character / graphic reading device according to claim 10, wherein an area representing a person's body is extracted. 13. The undetermined handwriting extraction unit outputs a difference image between an image obtained from the output of the change region extraction unit and an image obtained from the output of the writer extraction unit. 10 handwritten character / graphic reading devices. 14. The handwriting determination unit obtains a logical product of pixels corresponding to each other between the image obtained from the output of the undetermined handwriting extraction unit and the handwriting candidate image, and the value is “1”. 11. The handwritten character / graphic reading device according to claim 10, wherein information indicating a pixel position is output as the handwriting information. 15. The concealed handwriting extraction unit obtains a logical product of pixels corresponding to each other between the handwriting candidate image and the image obtained from the output of the writer extraction unit. 10 handwritten character / graphic reading devices. 16. The handwriting candidate image generation unit is a logical sum of pixels corresponding to each other between an image obtained from the output of the undetermined handwriting extraction unit and an image obtained from the output of the concealed handwriting extraction unit. 11. The handwritten character / graphic reading device according to claim 10, wherein: 17. The cumulative handwriting image storage unit acquires position information of a fixed handwriting portion from the handwriting information from the handwriting fixing unit, and a pixel value of a pixel indicated by the position information from the time series image. 11. The handwritten character according to claim 10, characterized in that
Graphic reader. 18. A handwriting extracting unit, wherein the handwriting extracting unit accumulates the handwriting information and stores it as a cumulative handwriting image, and a writing implement tracking unit that extracts the two-dimensional position of the tip of the writing implement from the time-series image. A pixel change detection unit that detects a change in a pixel corresponding to the position of the writing implement extracted by the writing implement tracking unit based on the cumulative handwriting image and the time-series image; and based on the output of the pixel change detection unit. The handwritten character according to claim 1, further comprising: a mode determination unit that determines whether or not writing has been performed and outputs the position of the writing unit as the handwriting information when writing is performed. -Figure reader. 19. The handwriting interpolation unit, wherein the handwriting extraction unit interpolates pixels on a straight line connecting two positions output at two consecutive times from the mode determination unit as handwriting and outputs the new handwriting information. The handwritten character / graphic reading device according to claim 18, further comprising: 20. A gesture recognition unit that detects an input of a control command by the gesture of the writer based on the handwriting information, and the handwriting extraction unit detects tip position information of the writing instrument when the handwriting pixel is detected. 19. The character / figure recognition unit is output to the gesture / recognition unit, and tip position information of the writing unit is output to the gesture recognition unit when the handwriting pixel is not detected.
9 handwritten character / graphic reading device.