JP7035852B2 - Writing selection program, writing selection method and information processing device - Google Patents

Description

The present invention relates to a writing selection program, a writing selection method, and an information processing apparatus.

A user interface for inputting the locus of a handwritten stroke such as a finger or an electronic pen as writing is known. By selecting a part of the writing that has been handwritten in this way, it is possible to use functions such as copying, character recognition, and Web search using the text that has been recognized.

For example, for the selection of writing, a method is used in which the entire writing is surrounded by handwriting or two points are specified so that the range for selecting the writing can be specified.

Japanese Unexamined Patent Publication No. 2013-25409 Japanese Unexamined Patent Publication No. 2005-267648

However, with the above technique, it may not be possible to easily select writing.

For example, in addition to the electronic white board, a user interface (hereinafter referred to as "spatial UI (User Interface)") that digitizes the entire space by giving the entire room such as a desk, board, or wall a display or touch panel function. A user interface may be used. When such a large-screen user interface is used, the range for selecting writing is wide, so it is necessary to move the hand greatly, and it is difficult to easily select writing.

In one aspect, it is an object of the present invention to provide a writing selection program, a writing selection method, and an information processing apparatus capable of easily performing a writing selection.

In one aspect, the writing selection program accepts the designation of the reference area as the reference for the writing search on the screen on which the writing with the handwritten input is displayed, and in the search direction in which the writing search is performed in the reference area. A first search area adjacent to the reference area is set, and a second search area adjacent to the reference area in the search direction and included in the first search area is set, and the first search area is set. When writing is included, the first search area is reset adjacent to the search direction side of the writing, and the end of the writing on the search direction side included in the first search area and the first. When the difference from the end of the writing on the search direction side included in the search area 2 is within a predetermined threshold, the end of the second search area on the search direction side is the end of the first search area. If the writing is not included in the first search area that has been reset to the same position as the end on the search direction side and has been set or reset, it is included in the first search area in which the writing has been searched. When the difference exceeds the predetermined threshold value, the writing is selected, and the writing included in the second search area in which the writing has been searched is selected, and the processing is executed by the computer.

Writing choices can be made easily.

FIG. 1 is a block diagram showing an example of a functional configuration of the information processing apparatus according to the first embodiment. FIG. 2 is a diagram showing an example of initial settings. FIG. 3 is a diagram showing a setting example of the writing search area S and the blank search area E. FIG. 4 is a diagram showing an example of resetting the writing search area S. FIG. 5 is a diagram showing an example of expansion of the confirmed region F. FIG. 6A is a diagram showing an example of the difference between the searched area D and the confirmed area F. FIG. 6B is a diagram showing an example of the difference between the searched area D and the confirmed area F. FIG. 7 is a diagram showing an example of resetting the blank search area E. FIG. 8 is a diagram showing an example of writing selection. FIG. 9 is a flowchart showing the procedure of the writing selection process according to the first embodiment. FIG. 10A is a diagram showing an example of a work screen. FIG. 10B is a diagram showing an example of a work screen. FIG. 10C is a diagram showing an example of a work screen. FIG. 10D is a diagram showing an example of a work screen. FIG. 11A is a diagram showing an operation example 1 of writing selection. FIG. 11B is a diagram showing an operation example 1 of writing selection. FIG. 12A is a diagram showing an operation example 2 of writing selection. FIG. 12B is a diagram showing an operation example 1 of writing selection. FIG. 13 is a flowchart showing the procedure of the writing selection process according to the second embodiment. FIG. 14A is a diagram showing an example of the central blank search area EC. FIG. 14B is a diagram showing an example of the upper blank search region EU. FIG. 14C is a diagram showing an example of the lower blank search region ED. FIG. 15A is a diagram showing an example of the threshold value Th2. FIG. 15B is a diagram showing an example of the threshold value Th3. FIG. 15C is a diagram showing an example of the threshold value Th4. FIG. 16 is a diagram showing an operation example of a multi-line search. FIG. 17 is a diagram showing an example of a frame division method. FIG. 18 is a flowchart showing the procedure of the extraction process according to the second embodiment. FIG. 19A is a diagram showing an example of extracting a character frame. FIG. 19B is a diagram showing an example of extracting a character frame. FIG. 19C is a diagram showing an example of extracting a character frame. FIG. 20A is a diagram showing an example of a mark as a candidate for a character frame. FIG. 20B is a diagram showing an example of a mark as a candidate for a character frame. FIG. 21A is a diagram showing an example of a mark as a candidate for a character frame. FIG. 21B is a diagram showing an example of a mark as a candidate for a character frame. FIG. 22 is a diagram showing an example of designating a rectangle in the vertical direction. FIG. 23 is a diagram showing a hardware configuration example of a computer that executes the writing selection program according to the first to third embodiments.

The writing selection program, the writing selection method, and the information processing apparatus according to the present application will be described below with reference to the attached drawings. It should be noted that this embodiment does not limit the disclosed technology. Then, each embodiment can be appropriately combined as long as the processing contents do not contradict each other.

FIG. 1 is a block diagram showing an example of a functional configuration of the information processing apparatus according to the first embodiment. The information processing device 10 shown in FIG. 1 provides a space UI that digitizes the entire space by providing the functions of a display and a touch panel to the entire room such as a desk, a board, and a wall. Here, an example of providing a spatial UI will be described as an example of a use case, but it may be another user interface such as an electronic whiteboard or a touch panel.

As a part of the function of such a spatial UI, the information processing apparatus 10 specifies an operation of selecting a part of the writing by handwriting input, hereinafter, "writing selection operation" in the minimum range. It provides a writing selection function realized by accepting.

The information processing device 10 is a computer that provides the above-mentioned spatial UI function.

As one embodiment, the information processing apparatus 10 can be implemented as package software or online software by installing a spatial UI program that realizes a function corresponding to the above spatial UI on an arbitrary computer. For example, the information processing apparatus 10 may be implemented on-premises as a computer that provides the above-mentioned spatial UI function, or may be implemented as a cloud that provides the above-mentioned spatial UI function by outsourcing.

As shown in FIG. 1, an output device 1a and an environment sensor 1b are connected to the information processing device 10. The connection form between the output device 1a and the environment sensor 1b and the information processing device 10 may be wired or wireless. For example, it may be connected via an arbitrary network, or may be connected via a cable or the like.

The output device 1a is a device that performs various outputs.

As an example, the output device 1a can be mounted as a projector that outputs an image on a flat surface such as a desk, a board, or a wall. For example, the output device 1a can project a work screen on a plane such as a desk, a board, or a wall that accepts handwriting input or gestures.

The environment sensor 1a is a sensor installed in an environment that provides a spatial UI.

As an example, the environment sensor 1a can be mounted as a camera that captures a work screen projected on a plane by the output device 1a. In this way, handwriting input and gestures can be detected from the captured image of the work screen. For example, when an electronic pen with infrared light is used, handwriting input or gesture can be detected by tracking the trajectory of infrared light from an image captured by a camera.

The spatial UI function mentioned here is only one aspect, and does not prevent other known spatial UI functions from being provided by the information processing apparatus 10. For example, in addition to the above work screen, the information processing apparatus 10 may display a digital sticker in space, a user terminal (not shown), media content such as an image or a moving image stored in an arbitrary computer, or the information processing device 10. Documents can be displayed in space, and operations on digital stickers, media contents, and documents can be accepted.

Although the example in which one set of output devices 1a and the environment sensor 1b is used has been described here as an example, two or more sets of output devices 1a and the environment sensor 1b may be used. In addition, the output device 1a and the environment sensor 1b may be provided by hardware other than the projector and the camera. For example, by installing the touch panel on a flat surface such as a desk, a board, or a wall, it is possible to realize a function corresponding to the output device 1a and the environment sensor 1b.

As shown in FIG. 1, the information processing apparatus 10 includes a communication I / F (InterFace) unit 11, a storage unit 13, and a control unit 15. In FIG. 1, a solid line showing the relationship between data transfer is shown, but for convenience of explanation, only the minimum part is shown. That is, the input / output of data relating to each processing unit is not limited to the illustrated example, and the input / output of data other than those shown in the illustration, for example, between the processing unit and the processing unit, between the processing unit and the data, and between the processing unit and the outside. Data may be input and output between the devices.

The communication I / F unit 11 is an interface that controls communication with other devices such as an output device 1a and an environment sensor 1b.

As one embodiment, the communication I / F unit 11 corresponds to a network interface card such as a LAN (Local Area Network) card. For example, the communication I / F unit 11 receives an image captured by the work screen from the environment sensor 1b, and outputs display data such as writing performed on the work screen and the work screen by handwriting. Output to.

The storage unit 13 includes data used in various programs such as an OS (Operating System) executed by the control unit 15, an application program, for example, the above-mentioned spatial UI program, or a writing selection program referred to by the spatial UI program as a library. It is a functional part that memorizes.

As one embodiment, the storage unit 13 can be implemented as an auxiliary storage device in the information processing device 10. For example, an HDD (Hard Disk Drive), an optical disk, an SSD (Solid State Drive), or the like can be adopted as the storage unit 13. The storage unit 13 does not necessarily have to be mounted as an auxiliary storage device, and may be mounted as a main storage device in the information processing device 10. In this case, various semiconductor memory elements such as RAM (Random Access Memory) and flash memory can be adopted for the storage unit 13.

The storage unit 13 stores the writing data 13a in which the writing performed by handwriting input is recorded on the work screen as an example of the data used in the program executed by the control unit 15. In addition to the written data 13a, other electronic data such as digital sticky notes, media contents, and documents to be displayed on the work screen may be stored in the storage unit 13.

The control unit 15 is a processing unit that controls the entire information processing apparatus 10.

As one embodiment, the control unit 15 can be implemented by a hardware processor such as a CPU (Central Processing Unit) or an MPU (Micro Processing Unit). Here, a CPU and an MPU are exemplified as an example of a processor, but any processor regardless of general-purpose type or specialized type, for example, GPU (Graphics Processing Unit), DSP (Digital Signal Processor), or GPGPU (General) -Can be implemented by Purpose computing on Graphics Processing Units). In addition, the control unit 15 may be realized by hard-wired logic such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array).

Here, since the above-mentioned spatial UI function is an existing technology, the above-mentioned writing selection function will be described in detail below.

For example, the control unit 15 has a writing selection program referred to as a library in the above spatial UI program or a writing selection added to the above spatial UI program on a work area of a RAM mounted as a main storage device (not shown). By expanding the program, the following processing unit is virtually realized.

As shown in FIG. 1, the control unit 15 includes a reception unit 15a, a setting unit 15b, a first determination unit 15c, a first resetting unit 15d, a second determination unit 15e, and a second resetting unit 15f. And 15 g of a writing selection unit.

The reception unit 15a is a processing unit that receives a reference area that serves as a reference for searching the writing area.

As one embodiment, the reception unit 15a accepts an operation of designating a writing circumscribed rectangle corresponding to one character. The circumscribed rectangle specified in this way is accepted as the "reference area B". For example, the "reference area B" may be accepted by describing the four sides of the rectangle with an electronic pen or the like, or by designating the start point and the end point of the rectangle. In addition, some gesture may be performed with an electronic pen and the circumscribed rectangle thereof may be calculated as the reference region B. Further, instead of the rectangle itself designated by the user, an area expanded by adding a play area in a predetermined range may be used as the reference area B.

After accepting the reference area B in this way, the reference area B is set as the initial values of the searched area D and the confirmed area F. The "searched area D" referred to here refers to an area in which the written search has been completed. Further, the "fixed area F" refers to an area in which the writing included in the area has been confirmed to be selected.

Here, as an example, when the direction of the character string in which handwritten input is performed on the work screen, that is, the so-called writing direction is horizontal writing, the left and right directions of the searched area D are the search directions for searching for writing or blanks. It becomes. Hereinafter, as one aspect, a specific example will be described by taking as an example the case where the left direction of the searched area D is the search direction. When the search direction is the right direction, writing can be selected in the same manner as when the search direction is the left direction by inverting the left and right sides of each process and executing the process, so the description thereof will be omitted.

FIG. 2 is a diagram showing an example of initial settings. FIG. 2 shows the writing of "Nana" as an example of the writing in which the handwriting input is performed on the work screen. Further, FIG. 2 shows an example in which the rectangle surrounding the rightmost "na" writing among the writings corresponding to the three "na" is accepted as the reference area B. In this case, the reference area B is set as the initial value of the searched area D and the confirmed area F.

The setting unit 15b is a processing unit that sets the writing search area S and the blank search area E.

As one embodiment, the setting unit 15b sets a rectangular area adjacent to the search direction of the searched area D as the writing search area S, and sets a partial area of the writing search area S as the blank search area E.

FIG. 3 is a diagram showing a setting example of the writing search area S and the blank search area E. FIG. 3 shows an example in which the written search area S and the blank search area E are set under the situation where the reference area B, the searched area D, and the confirmed area F shown in FIG. 2 are initially set. .. As shown by the alternate long and short dash line in FIG. 3, the setting unit 15b has the same width and height as the width and height of the reference region B in a state where the left side of the searched area D and the right side of the written search area S coincide with each other. The writing search area S is set. Further, as shown by a broken line in FIG. 3, the setting unit 15b has the same width as the writing search area S in a state where the left side of the searched area D includes the right side of the blank search area E, and the writing search area S has the same width. A blank search area E having a height shorter than the height of is set.

More specifically, the coordinates of the upper left vertex of the rectangle of the searched region D are (x1, y1), the coordinates of the lower right vertex are (x2, y2), the width is w, and the height is h. It is assumed that the magnitude relationship between x1 and x2 is "x2> x1" and the magnitude relationship between y1 and y2 is "y2> y1". At this time, a rectangle having the coordinates of the upper left vertex (x1-w, y1) and the coordinates of the lower right vertex (x2-w, y2) is set in the writing search area S. Further, in the blank search area E, a rectangle having the coordinates of the upper left vertex (x1-w, y1 + 0.2 * h) and the coordinates of the lower right vertex (x2-w, y2-0.2 * h) is set. Will be done.

The first determination unit 15c is a processing unit that determines whether or not there is writing in the writing search area S.

As one embodiment, the first determination unit 15c searches the writing data 13a stored in the storage unit 13 for writing including a part or all of the writing search area S. Then, the first determination unit 15c determines whether or not the midpoint of the circumscribed rectangle of the writing is included in the writing search area S for each writing obtained as the search result of the writing data 13a. At this time, when the midpoint of the circumscribed rectangle of writing is included in the writing search area S, it is identified that the writing exists in the writing search area S. Then, the first determination unit 15c determines that the writing exists in the writing search area S when any of the midpoints of the circumscribing rectangles of the writing obtained as the search result of the writing data 13a is included in the writing search area S. ..

Here, an example is given in which the writing is identified as existing in the writing search area S depending on whether or not the midpoint of the circumscribed rectangle of the writing is included in the writing search area S. Can be determined. For example, the presence or absence of writing can be determined depending on whether or not the dot at the right end of writing is included in the writing search area S.

The first resetting unit 15d is a processing unit that resets the writing search area S.

As one embodiment, when the writing is present in the writing search area S, the first resetting unit 15d expands the searched area D so that the writing set in the writing search area S is included. Then, the first resetting unit 15d resets the written search area S adjacent to the searched area D.

FIG. 4 is a diagram showing an example of resetting the writing search area S. FIG. 4 shows the writing of "Noruna" as an example of the writing in which the handwriting input is performed on the work screen. Of these, the first "no" from the left and the second "no" from the left are smaller in size than the other writing "ru" and "na", and the position is also located above the upper side. Of such writing "Noruna", in FIG. 4, the first writing "na" from the right is set as the initial value of the searched area D, and is written at a position adjacent to the left side thereof. An example of resetting the writing search area S under the situation where the search area S is set is shown.

As shown in FIG. 4, the writing search area S includes the second writing "no" from the left and the second writing "ru" from the right. When writing exists in the writing search area S in this way, the first resetting unit 15d sets the left end of the searched area D to the position where the leftmost dot of the writing set existing in the writing search area S exists, that is, The searched area D is expanded by replacing it with the position at the left end of the writing "no", which is the second from the left. Further, the first resetting unit 15d resets the new writing search area S by shifting the writing search area S to the left at a position where the right side of the writing search area S overlaps with the left side of the searched area D. ..

The second determination unit 15e is a processing unit that determines whether or not there is writing in the blank search area E.

As one embodiment, when the writing is present in the writing search area S, the second determination unit 15e searches the writing data 13a stored in the storage unit 13 for the writing including a part or all of the blank search area E. .. Then, the second determination unit 15e determines whether or not the midpoint of the circumscribed rectangle of the writing is included in the blank search area E for each writing obtained as the search result of the writing data 13a. At this time, if the midpoint of the circumscribed rectangle of the writing is included in the blank search area E, it is identified that the writing exists in the blank search area E. Then, the second determination unit 15e determines that the writing exists in the blank search area E when any of the midpoints of the circumscribed rectangles of the writing obtained as the search result of the writing data 13a is included in the blank search area E. ..

The second resetting unit 15f is a processing unit that resets the blank search area E.

As one embodiment, the second resetting unit 15f expands the determined area F so that the writing set in the blank search area E is included when the writing exists in the blank search area E. FIG. 5 is a diagram showing an example of expansion of the confirmed region F. Similar to FIG. 4, FIG. 5 shows the writing of “Noruna” as an example of the writing in which the handwriting input is performed on the work screen, and the writing search area S shown in FIG. 4 is reset. The initial values of the blank search area E and the confirmed area F at the time point are shown. As shown in FIG. 5, the blank search area E includes the second writing “ru” from the right. When writing exists in the blank search area E in this way, the second resetting unit 15f sets the left end of the confirmed area F at the position where the left end dot of the writing set existing in the blank search area E exists, that is, The confirmed area F is expanded by replacing it with the position at the left end of the writing "ru", which is the second from the right.

After the confirmed region F is expanded in this way, the second resetting unit 15f determines whether or not the difference between the searched region D and the confirmed region F is equal to or greater than the predetermined threshold value Th1.

6A and 6B are diagrams showing an example of the difference between the searched region D and the confirmed region F. In FIG. 6A, the coordinates of the left end of the searched area D and the confirmed area F under the condition that the searched area D shown in FIG. 4 is expanded and the confirmed area F shown in FIG. 5 is expanded are performed. The difference from the coordinates at the left end of is shown. Further, in FIG. 6B, the coordinates of the left end of the searched area D and the confirmed area F under the situation where the searched area D is further expanded after the confirmed area F shown in FIG. 5 is expanded are shown in FIG. The difference from the coordinates at the left end of is shown.

Here, as an example, assuming that the threshold value Th1 is the width w × 1.25 of the reference region B, an example of comparing the difference between the coordinates of the left end of the searched region D and the coordinates of the left end of the confirmed region F is compared. explain. As shown in FIG. 6A, at the stage immediately after the expansion of the confirmed region F shown in FIG. 5, the difference between the coordinates of the left end of the searched region D and the coordinates of the left end of the confirmed region F is the threshold value Th1. It becomes (= w × 1.25) or less. On the other hand, as shown in FIG. 6B, at the stage where the searched area D is further expanded after the confirmed area F shown in FIG. 5 is expanded, the coordinates at the left end of the searched area D are used. The difference from the coordinates at the left end of the fixed region F exceeds the threshold value Th1 (= w × 1.25).

Here, when the difference between the coordinates of the left end of the searched area D and the coordinates of the left end of the confirmed area F is the threshold value Th1 or less, the second resetting unit 15f keeps the coordinates of the upper and lower ends of the blank search area E as they are. By replacing the coordinates of the right end of the blank search area E with the coordinates of the left end of the confirmed area F and replacing the coordinates of the left end of the blank search area E with the coordinates of the left end of the written search area S, the blank search area E is replaced with the coordinates of the left end of the written search area S. Reset.

FIG. 7 is a diagram showing an example of resetting the blank search area E. FIG. 7 shows a writing search area S, a blank search area E, and a confirmed area F at the time when it is determined that the difference shown in FIG. 6A is equal to or less than the threshold value Th1. As shown in FIG. 7, the second resetting unit 15f identifies a portion of the blank search region E that overlaps with the confirmed region F, and a region shown by hatching with a downward slope in FIG. 7. Subsequently, the second resetting unit 15f performs the same calculation as the blank search area E shown in FIG. 3 from the written search area S and the area Q in which the portion overlapping with the confirmed area F is excluded from the blank search area E. The obtained partial region P is specified. Then, the second resetting unit 15f resets the circumscribed rectangle including the previously specified region Q and the partial region P as a new blank search region E'.

When the blank search area E'is reset in this way, the process of the first determination unit 15c is executed again. After that, the first determination unit 15c determines that there is no writing in the writing search area S, or the second resetting unit 15f determines the difference between the leftmost coordinate of the searched area D and the leftmost coordinate of the confirmed area F. Is repeatedly executed by the first determination unit 15c, the first resetting unit 15d, the second determination unit 15e, and the second resetting unit 15f until it is determined that the value exceeds the threshold value.

The writing selection unit 15g is a processing unit for selecting writing.

As one aspect, when the first determination unit 15c determines that there is no writing in the writing search area S, the writing selection unit 15g selects the writing existing in the searched area D. FIG. 8 is a diagram showing an example of writing selection. FIG. 8 shows the searched area D at the time when the resetting of the writing search area S is executed to the limit in the writing “Nana” shown in FIG. That is, as shown in FIG. 8, when the writing is not included in the writing search area, the writing selection unit 15g selects the writing “Nana” included in the searched area D.

As another aspect, the writing selection unit 15g is determined when the second resetting unit 15f determines that the difference between the coordinates of the left end of the searched area D and the coordinates of the left end of the confirmed area F exceeds the threshold value Th1. Select the writing that exists in the finished area F. That is, as in the example shown in FIG. 6B, when the difference between the coordinates of the left end of the searched region D and the coordinates of the left end of the confirmed region F exceeds the threshold value Th1, the writing selection unit 15g is the confirmed region F. Select the writing "Luna" included in.

The writing selected in this way, hereinafter "writing in the selected state", can be displayed in a display form different from the display form of the unselected writing, as an example. For example, the writing selection unit 15g can highlight the writing in the selected state or the circumscribed rectangle including the writing in the selected state, or display it with a fluorescent marker or the like. In addition, the writing selection unit 15g has an icon for copying the selected writing in the vicinity of the extrinsic rectangle including the selected writing or the selected writing on the work screen, and an icon for executing character recognition for the selected writing. It is also possible to display an icon or the like for Web search using the text in which the character is recognized in the written writing in the selected state.

[Processing flow]
FIG. 9 is a flowchart showing the procedure of the writing selection process according to the first embodiment. This process is executed, for example, when the reference area B is accepted. As shown in FIG. 9, when the reception unit 15a receives the reference area B, the reception unit 15a sets the reference area B as the initial values of the searched area D and the confirmed area F (step S101).

Subsequently, the setting unit 15b sets a rectangular area adjacent to the search direction of the searched area D initially set in step S101 as the writing search area S, and sets a partial area of the writing search area S as a blank search area. It is set as E (step S102).

Then, the first determination unit 15c determines whether or not the writing exists in the writing search area S (step S103). At this time, when writing exists in the writing search area S (step S103Yes), the first resetting unit 15d expands the searched area D so that the writing set in the writing search area S is included (step S104). .. Then, the first resetting unit 15d resets the writing search area S adjacent to the searched area D (step S105).

After that, the second determination unit 15e determines whether or not the writing exists in the blank search area E (step S106). At this time, when writing exists in the blank search area E (step S106Yes), the second resetting unit 15f expands the confirmed area F so that the writing set in the blank search area E is included (step S107). .. If there is no writing in the blank search area E (step S106 No), step S107 is skipped and the process proceeds to step S108.

Then, the second resetting unit 15f determines whether or not the difference between the searched region D and the confirmed region F is equal to or greater than a predetermined threshold value Th1 (step S108). At this time, when the difference is equal to or less than the threshold value Th1 (step S108Yes), the second resetting unit 15f has the area Q in which the portion overlapping with the confirmed area F is excluded from the blank search area E and the writing search area S. The blank search area E is reset based on the partial area P (step S109).

After that, as long as writing exists in the writing search area S and the difference is equal to or less than the threshold value Th1 (step S103Yes and step S108Yes), the processes of step S103 to step S109 are repeatedly executed.

Here, when there is no writing in the writing search area S (step S103No), the writing selection unit 15g selects the writing existing in the searched area D (step S110), and ends the process. Further, when the difference between the leftmost coordinate of the searched area D and the leftmost coordinate of the confirmed area F exceeds the threshold value Th1 (step S108No), the writing selection unit 15g selects the writing existing in the confirmed area F. (Step S111), and the process is terminated.

In the writing selection process shown in FIG. 9, when the writing direction of the work screen is horizontal writing, the processes of the above steps S101 to the above step S111 are executed for each of the two search directions of the left direction and the right direction. To. Then, the written union obtained for each search direction is selected.

[One aspect of the effect]
As described above, according to the writing selection process shown in FIG. 9, writing can be selected by designating the size equivalent to one character of the writing displayed on the work screen as the reference area, so that the writing can be selected on the large screen. Even if a user interface is provided, the writing selection operation can be realized by accepting a minimum range of specifications. Therefore, it is possible to easily select writing.

In addition to being able to improve the convenience of the writing selection operation in the user interface of a large screen in this way, according to the writing selection process shown in FIG. 9, a multi-user environment in which writing is input by a plurality of users. You can realize a very robust writing selection.

[One aspect of issues in a multi-user environment]
That is, in the above-mentioned multi-user environment, writing of various sizes may be mixed, the positions of writing performed by different users may be close to each other, or writing over multiple lines may be input by handwriting. Free writing is performed, such as the writing direction of writing varies not only vertically and horizontally but also diagonally. While the spatial UI provides an environment in which writing can be freely performed in this way, it is difficult to select writing without mixing the writings of different users.

10A to 10D are views showing an example of a work screen. FIG. 10A shows a written “chat” in which handwriting input is performed by the user A. When selecting the writing of the user A from the work screen shown in FIG. 10A, it is desired to select the writings "ya" and "tsu" corresponding to the sokuon yoon without omission. Further, FIG. 10B shows the writing "" Tokyo "" in which the handwriting input is performed by the user B. When selecting the writing of user B from the work screen shown in FIG. 10B, it is desired to select the writing "" and "" "corresponding to the key brackets without omission. Thus, some symbols, such as sokuon yoon or brackets, are smaller in size than hiragana and katakana, which correspond to kanji, vowels, and consonants. Even when the size of the characters varies in this way, it is preferable to select the writings that have been handwritten by the same user without omission.

Further, in FIG. 10C, the writing "Nakahara Ward, Kawasaki City" is handwritten by the user C, and the writing "Nakahara Ward, Kawasaki City" is smaller than the writing "Nakahara Ward, Kawasaki City" starting from the upper right end. The writing "as is" is handwritten by the user D in the size. Further, in FIG. 10D, the writing "in this case" is handwritten by the user E, and the writing "in this case" is started from the lower right end by the user F in a size larger than the writing "in this case". The written "Nana Nana" is handwritten.

For example, when the intention is to select the writing "Nakahara-ku, Kawasaki City" of the user C from the work screen shown in FIG. 10C, the writing "as is" of the user D is desired to be ignored. Further, when the intention is to select the writing "Nana Nana" of the user F from the work screen shown in FIG. 10D, it is desired to ignore the writing "in this case" of the user E.

Here, as the writing performed by the same user by handwriting, while selecting the writing corresponding to some symbols such as the sokuon or the key bracket shown in FIGS. 10A and 10B, the writing of another user When writing is mixed, the writing of the user who accepted the designation of the range corresponding to one character as the reference area is separated from the writing of other users shown in FIGS. 10C and 10D, for example, "as is" or "in this case". There are some aspects that are difficult to choose for writing.

That is, even in the existing technology, if it is extended to other fields such as OCR and number plate recognition, there is a process for determining the continuity of characters. However, the determination of the continuity of characters in the existing technology is limited to extracting character strings of the same size side by side or vertically. For example, when the writing corresponding to some symbols such as the sokuon and the brackets shown in FIGS. 10A and 10B is recognized as a series of writings by the determination of the continuity of the characters, the writing "as is" shown in FIG. 10C or It is inevitable that the writing "in this case" shown in FIG. 10D is also recognized as a series of writing.

As described above, the multi-user-compatible writing selection function, which is difficult to realize with the existing technology, can be realized by the writing selection process shown in FIG. This point will be described below with reference to FIGS. 11A, 11B, 12A, and 12B.

(1) Operation example 1
11A and 11B are diagrams showing operation example 1 of writing selection. 11A and 11B show an example in which the part of "ta" in the writing "become" is accepted as the reference area B and the writing is selected according to the reference area B.

As shown in FIG. 11A, at the time of t1, the reference region B is set as the initial values of the searched region D and the confirmed region F (step S101_loop 1). At the time of t2 following t1, as shown by the alternate long and short dash line in the figure, the width and height of the reference region B are the same as the width and height of the reference region B in a state where the left side of the searched region D and the right side of the written search region S coincide with each other. The writing search area S having a height is set. Further, as shown by the broken line in the figure, the left side of the searched area D includes the right side of the blank search area E, has the same width as the written search area S, and is shorter than the height of the written search area S. A blank search area E having a height is set (step S102_loop 1).

Here, it is determined whether or not the writing is included in the writing search area S (step S103_loop 1). Here, since the writing search area S includes the writing “tsu”, the branch in step S103 proceeds to Yes. Then, at the time of t3, the searched area D is expanded so that the writing “tsu” in the writing search area S is included (step S104_loop 1). Next, at the time of t4, the writing search area S adjacent to the searched area D is reset (step S105_loop 1).

After that, it is determined whether or not the blank search area E includes writing (step S106_loop 1). Here, since the blank search area E does not include writing, the branch in step S106 proceeds to No. In this case, the expansion of the fixed area F in step S107 is skipped. Then, at the time of t5, it is determined whether or not the difference between the coordinates of the left end D _LEFT of the searched area D and the coordinates of the left end F _LEFT of the confirmed area F is equal to or less than the threshold value Th1 (step S108_loop 1). ..

Here, since the difference between the coordinates of the left end D _LEFT of the searched area D and the coordinates of the left end F _LEFT of the confirmed area F is the threshold value Th1 or less, the branch in step S108 proceeds to Yes. Subsequently, at the time of t6, the circumscribed rectangle including the region Q in which the portion overlapping with the confirmed region F is excluded from the blank search region E and the partial region P of the written search region S is re-used as the blank search region E. It is set (step S109_loop 1).

After the step S109_loop 1 is executed in this way, the process proceeds to the second loop. Then, it is determined whether or not the writing is included in the writing search area S reset in step S105_loop 1 (step S103_loop 2). Here, since the writing “na” is included in the writing search area S reset in step S105_loop 1, the branch of step S103_loop 2 proceeds to Yes. Then, as shown in FIG. 11B, at the time of t7, the searched area D is expanded so as to include the writing “na” in the writing search area S (step S104_loop 2). Next, at the time of t8, the writing search area S adjacent to the searched area D is reset (step S105_loop 2).

After that, it is determined whether or not the writing is included in the blank search area E reset in step S109_loop 1 (step S106_loop 2). Here, since the blank search area E includes the writing “na”, the branch of step S106_loop 2 proceeds to Yes. Subsequently, at the time of t9, the confirmed region F is expanded so that the writing set “na” in the blank search region E is included (step S107_loop 2).

Then, it is determined whether or not the difference between the coordinates of the left end D _LEFT of the searched area D and the coordinates of the left end F _LEFT of the confirmed area F is equal to or less than the threshold value Th1 (step S108_loop 2). Here, since the coordinates of the left end D _LEFT of the searched area D and the coordinates of the left end F _LEFT of the confirmed area F match, the branch of step S108 proceeds to Yes. Subsequently, at the time of t10, the circumscribed rectangle including the region Q in which the portion overlapping with the confirmed region F is excluded from the blank search region E and the partial region P of the written search region S is re-used as the blank search region E. It is set (step S109_loop 2).

After the step S109_loop 2 is executed in this way, the process proceeds to the third loop. Then, it is determined whether or not the writing is included in the writing search area S reset in step S105_loop 2 (step S103_loop 3). Here, since the writing “na” is included in the writing search area S reset in step S105_loop 1, the branch of step S103_loop 2 proceeds to No. Next, at the time of t11, the written “become” included in the searched area D and the hatched portion with a downward slope in the figure is selected (step S110).

As described above, by executing the writing selection process shown in FIG. 9, the writing "tsu" corresponding to the sokuon yoon is not leaked when the writing "Nata", which was handwritten by the same user, is selected. You can choose.

(2) Operation example 2
12A and 12B are diagrams showing operation example 2 of writing selection. In FIGS. 12A and 12B, along with the writing "Other" manually input by one user, another user with a size larger than the writing "Other" starting from the upper right end of the writing "Other". An example is shown in which the writing "ta" manually entered by is included in the work screen. An example is shown in which the "ta" part of the writing included in such a work screen is accepted as the reference area B, and the writing is selected according to the reference area B.

As shown in FIG. 12A, at the time of t1, the reference region B is set as the initial values of the searched region D and the confirmed region F (step S101_loop 1). At the time of t2 following t1, as shown by the alternate long and short dash line in the figure, the width and height of the reference region B are the same as the width and height of the reference region B in a state where the left side of the searched region D and the right side of the written search region S coincide with each other. The writing search area S having a height is set. Further, as shown by the broken line in the figure, the left side of the searched area D includes the right side of the blank search area E, has the same width as the written search area S, and is shorter than the height of the written search area S. A blank search area E having a height is set (step S102_loop 1).

Here, it is determined whether or not the writing is included in the writing search area S (step S103_loop 1). Here, since the writing search area S includes the writing “no” and “other”, the branch of step S103 proceeds to Yes. Then, at the time of t3, the searched area D is expanded so that the writing “no” and the writing “other” in the writing search area S are included (step S104_loop 1). Next, at the time of t4, the writing search area S adjacent to the searched area D is reset (step S105_loop 1).

After that, it is determined whether or not the blank search area E includes writing (step S106_loop 1). Here, since the blank search area E does not include writing, the branch of step S106_loop 1 proceeds to No. In this case, the expansion of the fixed area F in step S107 is skipped. Then, at the time of t5, it is determined whether or not the difference between the coordinates of the left end D _LEFT of the searched area D and the coordinates of the left end F _LEFT of the confirmed area F is equal to or less than the threshold value Th1 (step S108_loop 1). ..

Here, since the difference between the coordinates of the left end D _LEFT of the searched area D and the coordinates of the left end F _LEFT of the confirmed area F is the threshold value Th1 or less, the branch in step S108 proceeds to Yes. Subsequently, at the time of t6, the circumscribed rectangle including the region Q in which the portion overlapping with the confirmed region F is excluded from the blank search region E and the partial region P of the written search region S is re-used as the blank search region E. It is set (step S109_loop 1).

After the step S109_loop 1 is executed in this way, the process proceeds to the second loop. Then, it is determined whether or not the writing is included in the writing search area S reset in step S105_loop 1 (step S103_loop 2). Here, since the writing “so” is included in the writing search area S reset in step S105_loop 1, the branch of step S103_loop 2 proceeds to Yes. Then, as shown in FIG. 12B, at the time of t7, the searched area D is expanded so that the writing “so” in the writing search area S is included (step S104_loop 2). Next, at the time of t8, the writing search area S adjacent to the searched area D is reset (step S105_loop 2).

After that, it is determined whether or not the writing is included in the blank search area E reset in step S109_loop 1 (step S106_loop 2). Again, since the blank search area E does not include writing, the branch of step S106_loop 2 proceeds to No. In this case, the expansion of the fixed area F in step S107 is skipped.

Then, at the time of t9, it is determined whether or not the difference between the coordinates of the left end D _LEFT of the searched area D and the coordinates of the left end F _LEFT of the confirmed area F is equal to or less than the threshold value Th1 (step S108_loop 2). .. Here, since the coordinates of the left end D _LEFT of the searched area D and the coordinates of the left end F _LEFT of the confirmed area F exceed the threshold value Th1, the branch of step S108_loop 2 proceeds to No. Subsequently, at the time of t10, the written “ta” included in the confirmed region F and the hatched portion with a downward slope in the figure is selected (step S111).

As described above, by executing the writing selection process shown in FIG. 9, the writing "ta" of the user who has accepted the designation of the range corresponding to one character as the reference area is referred to as the writing "other" of another user. You can select writing separately.

Although the embodiments relating to the disclosed apparatus have been described so far, the present invention may be implemented in various different forms other than the above-described embodiments. Therefore, another embodiment included in the present invention will be described below.

[Reference area]
In the first embodiment, the case where the writing selection process shown in FIG. 9 is executed assuming that the range corresponding to one character is designated as the reference area is illustrated. However, for example, the information processing apparatus 10 has a reference area. It is also possible to determine whether or not the writing included in B is a single character. As an example, the information processing apparatus 10 can use the aspect ratio of the reference area to determine whether the writing included in the reference area is one character or a plurality of characters. For example, if the ratio obtained from the aspect ratio of the reference area is within a predetermined range from "1" corresponding to the square, it is determined as one character, while the ratio obtained from the aspect ratio of the reference area corresponds to "1". If it is out of the predetermined range, it is determined that there are multiple characters. At this time, when the writing included in the reference area B is one character, the information processing apparatus 10 executes the writing selection process shown in FIG. 9, and when the writing included in the reference area B is a plurality of characters, the reference. You can also select the writing set contained in the area as it is. As a result, it is possible to switch between the selection of the entire writing of the same user and the selection of a part of the writing.

[Use of multiple blank search areas]
In the first embodiment described above, an example in which one type of blank search area E is used has been described, but a writing selection process can also be performed using a plurality of blank search areas. For example, the information processing apparatus 10 has three blank search regions, a central blank search region, an upper blank search region, and a lower blank search region, and corresponding confirmed regions such as a central fixed region, an upper fixed region, and a lower portion. When the difference is determined for each of the searched area, the central confirmed area, the upper confirmed area, and the lower confirmed area using the three confirmed areas, and all the differences are within a predetermined threshold value, the difference is determined. It can be implemented to repeat the search.

FIG. 13 is a flowchart showing the procedure of the writing selection process according to the second embodiment. This process is executed when the reference area B is received, as in FIG. As shown in FIG. 13, when the reception unit 15a receives the reference area B, the reception unit 15a sets the reference area B as the initial values of the searched area D and the confirmed area F (step S201).

Subsequently, the setting unit 15b sets a rectangular area adjacent to the search direction of the searched area D initially set in step S201 as the writing search area S, and centers the three partial areas of the writing search area S. It is set as the blank search area EC, the upper blank search area EU, and the lower blank search area ED (step S202).

For example, the central blank search area EC, the upper blank search area EU, and the lower blank search area ED can be set as follows. For example, the search direction is to the left, the coordinates of the upper left vertex of the rectangle of the searched area D are (x1, y1), the coordinates of the lower right vertex are (x2, y2), the width is w, and the height. Is h, the magnitude relationship between x1 and x2 is "x2> x1", and the magnitude relationship between y1 and y2 is "y2> y1". At this time, in the central blank search area EC, as an example, the coordinates of the upper left vertex (x1-w, y1 + 0.15 * h) and the coordinates of the lower right vertex (x2-w, y2-0.15 * h). A rectangle with is set. Further, in the upper blank search area EU, as an example, a rectangle having the coordinates of the upper left vertex (x1-w, y1) and the coordinates of the lower right vertex (x2-w, y2-0.4 * h) is set. Will be done. Further, in the lower blank search area ED, a rectangle having the coordinates of the upper left vertex (x1-w, y1 + 0.4 * h) and the coordinates of the lower right vertex (x2-w, y2) is set.

These central blank search area EC, upper blank search area EU, and lower blank search area ED are schematically shown in FIGS. 14A to 14C. FIG. 14A is a diagram showing an example of the central blank search area EC. FIG. 14B is a diagram showing an example of the upper blank search region EU. FIG. 14C is a diagram showing an example of the lower blank search region ED. As shown in FIG. 14A, the upper side of the central blank search area EC is 0.15 * h lower than the upper side of the searched area D. Further, the lower side of the central blank search area EC is 0.15 * h higher than the lower side of the searched area D. The height of such a central blank search area EC is 0.7 * h. Further, as shown in FIG. 14B, the upper side of the upper blank search area EU is at the same height as the upper side of the searched area D. On the other hand, the lower side of the upper blank search region EU is 0.4 * h higher than the lower side of the searched region D. The height of such an upper blank search region EU is 0.6 * h. Further, as shown in FIG. 14C, the upper side of the lower blank search area ED is 0.4 * h lower than the upper side of the searched area D. Further, the lower side of the lower blank search area ED is at the same height as the lower side of the searched area D. The height of such a lower blank search region ED is 0.6 * h.

Subsequently, the first determination unit 15c determines whether or not writing exists in the writing search area S (step S203). At this time, when writing exists in the writing search area S (step S203Yes), the first resetting unit 15d expands the searched area D so that the writing set in the writing search area S is included (step S204). .. Then, the first resetting unit 15d resets the written search area S adjacent to the searched area D (step S205).

After that, the second determination unit 15e determines whether or not the writing exists in the central blank search area EC (step S206). At this time, when writing exists in the central blank search area EC (step S206Yes), the second resetting unit 15f expands the central fixed area FC so as to include the writing set in the central blank search area EC (step S206Yes). Step S207). If there is no writing in the central blank search area EC (step S206No), step S207 is skipped and the process proceeds to step S208.

Then, the second resetting unit 15f determines whether or not the difference between the searched region D and the central confirmed region FC is equal to or greater than a predetermined threshold value Th2 (step S208). As the threshold value Th2 used here, a value shorter than the threshold value Th1 given as an example in the above-mentioned Example 1 is used. FIG. 15A is a diagram showing an example of the threshold value Th2. As shown in FIG. 15A, the width w * 1.0 of the reference region B can be used for the threshold value Th2 to be compared with the difference between the searched region D and the central confirmed region FC. In this way, using a value shorter than the threshold value Th1 given in Example 1 above makes it difficult to select the writing even if there is writing near the upper end portion or the lower end portion of the writing search area S. To do.

At this time, if the difference between the searched region D and the central confirmed region FC is the threshold value Th2 or less (step S208Yes), the process proceeds to the next step S209. That is, the second determination unit 15e determines whether or not the writing exists in the upper blank search region EU (step S209). At this time, when writing exists in the upper blank search area EU (step S209Yes), the second resetting unit 15f expands the upper fixed area FU so as to include the writing set in the upper blank search area EU (step S209Yes). Step S210). If there is no writing in the upper blank search area EU (step S209No), step S210 is skipped and the process proceeds to step S211.

Then, the second resetting unit 15f determines whether or not the difference between the searched region D and the upper confirmed region FU is equal to or greater than a predetermined threshold value Th3 (step S211). As the threshold value Th3 used here, a value longer than the threshold value Th1 and the threshold value Th2 is used. FIG. 15B is a diagram showing an example of the threshold value Th3. As shown in FIG. 15B, the width w * 1.5 of the reference region B can be used for the threshold value Th3 to be compared with the difference between the searched region D and the upper confirmed region FU. The reason why a value longer than the threshold value Th1 and the threshold value Th2 is used in this way is that there is a possibility that the writing corresponds to a part of the word in the upper part of the writing search area S, so that the writing is easily selected. Is.

At this time, if the difference between the searched area D and the upper confirmed area FU is the threshold value Th3 or less (step S211Yes), the process proceeds to the next step S212. That is, the second determination unit 15e determines whether or not the writing exists in the lower blank search area ED (step S212). At this time, when writing exists in the lower blank search area ED (step S212Yes), the second resetting unit 15f expands the lower fixed area FD so as to include the writing set in the lower blank search area ED (step S212Yes). Step S213). If there is no writing in the lower blank search area ED (step S212 No), step S213 is skipped and the process proceeds to step S214.

Then, the second resetting unit 15f determines whether or not the difference between the searched region D and the lower confirmed region FD is equal to or greater than a predetermined threshold value Th4 (step S214). As the threshold value Th4 used here, a value longer than the threshold value Th1 and the threshold value Th2 is used. FIG. 15C is a diagram showing an example of the threshold value Th4. As shown in FIG. 15C, the width w * 1.5 of the reference region B can be used for the threshold value Th4 to be compared with the difference between the searched region D and the downward confirmed region FD. The reason why a value longer than the threshold value Th1 and the threshold value Th2 is used in this way is that there is a possibility that the writing corresponds to a part of the word in the lower part of the writing search area S, so that the writing is easily selected. Is.

Here, when the difference between the searched area D and the lower confirmed area FD is the threshold value Th4 or less (step S214Yes), the three blank areas of the central blank search area EC, the upper blank search area EU, and the lower blank search area ED are displayed. The setting is reset (step S215), and the process proceeds to the process of step S203. The resetting of the three blank areas can be performed from any of the central blank search area EC, the upper blank search area EU, and the lower blank search area ED, respectively, as the central confirmed area FC, the upper confirmed area FU, or the lower fixed area FD. Reconfigured based on the region QC, QUA or QD excluding the overlap with and the partial region PC, PU or PD located at the height of FIGS. 14A, 14B or 14C in the writing search region S. ..

After that, as long as writing exists in the writing search area S and the three differences are equal to or less than the threshold Th2, the threshold Th3, and the threshold Th4 (step S203Yes, step S208Yes, step S211Yes, and step S214Yes), the above steps S203 to the above steps. The process of S215 is repeatedly executed.

Here, when there is no writing in the writing search area S (step S203No), the writing selection unit 15g selects the writing existing in the searched area D (step S216), and ends the process. Further, when the difference between the coordinates of the left end of the searched area D and the coordinates of the left end of the central confirmed area FC exceeds the threshold value Th2 (step S208No), the writing selection unit 15g is the writing existing in the central confirmed area FC. Is selected (step S217), and the process ends. Further, when the difference between the coordinates of the left end of the searched area D and the coordinates of the left end of the upper confirmed area FU exceeds the threshold value Th3 (step S211No), the writing selection unit 15g is the writing existing in the upper confirmed area FU. Is selected (step S218), and the process ends. Further, when the difference between the coordinates of the left end of the searched area D and the coordinates of the left end of the lower confirmed area FD exceeds the threshold value Th4 (step S214No), the writing selection unit 15g is the writing existing in the lower confirmed area FD. Is selected (step S219), and the process ends.

As described above, by executing the writing selection process shown in FIG. 13, it is possible to exclude the writing biased upward and the writing biased downward from the search.

In Japanese, when writing horizontally, such as voiced sound mark, semi-voiced sound mark, punctuation mark, and yoon pronunciation, it may be written unevenly upward and downward. However, since the sequence of these small characters is usually within two characters, by setting the width of the search range for detecting upper and lower blanks to be larger than two lowercase letters, continuous writing and unrelated writing can be separated. .. In the above embodiment, the threshold value of the central blank search area check character is w * 1.0, and the threshold value at the time of checking the upper blank search area and the lower blank search area is as large as w * 1.5. , This is to select without excluding the equivalent of two consecutive lowercase letters. Also, regarding the alphabet, although there are characters such as apostrophes and double quotes that are written with a bias toward the upper limit, they are usually within two characters.

In addition, a rectangle may explicitly indicate in which of the writing search area S, the central blank search area EC, the upper blank search area EU, and the lower blank search area ED the stop condition is satisfied. In addition, a reference area, a final confirmed area, or a final selected area may be explicitly indicated.

[Multiple line search]
For example, the information processing apparatus 10 can calculate the aspect ratio of the rectangular area received as the reference area B, and if the aspect ratio is vertically long to a predetermined degree, it is possible to search for a plurality of lines. As a result, it is possible to select one line when enclosing one character and select multiple lines when enclosing it vertically.

Further, in order to realize such a plurality of line selection, when the writing direction of the writing is horizontal writing, the information processing apparatus 10 is not in the left-right direction corresponding to the searching direction of the writing selection, but in the search direction of the writing selection. Rows are searched in the vertical direction orthogonal to each other, a rectangular area for each character is extracted from the area containing the selected writing set according to a predetermined line search algorithm, and each rectangle extracted for each character is used as a reference area. , Select a writing set divided for each row by performing a search in the left-right direction for each reference area.

FIG. 16 is a diagram showing an operation example of a multi-line search. FIG. 16 shows an example of a work screen in which the writing "Okubo" is included in the first line, the writing "Nishi-Shimbashi" is included in the second line, and the writing "Kisarazu" is included in the third line. As shown in FIG. 16, the information processing apparatus 10 accepts the writing "large" of the writing "Okubo" in the first line in step S1 as a reference area, and searches in the vertical direction of the writing "large" in step S2. Writing Select the writing set of "Large", "West" and "Tree". Then, in step S3, the rectangular area "large", the rectangular area "west", and the rectangular area "tree" for each character are extracted from the writing set, and each rectangular area is searched in the left-right direction for each line. Select the writing set "Okubo", "Nishi-Shimbashi" and "Kisarazu".

It is also possible to apply a predetermined process or a process specified by the user to the writing set selected for each line in this way. For example, for each selected writing set "Okubo", "Nishi-Shimbashi", and "Kisarazu", character string recognition is performed, and the recognition result text can be used to perform an image search and display an icon corresponding to the displayed command. can.

Here, the process of step S2 shown in FIG. 16 will be specifically described. FIG. 17 is a diagram showing an example of a frame division method. FIG. 17 shows an example of extracting a reference region corresponding to a character in each line from a writing set in the vertical direction of writing “large”, “west”, and “tree”. As shown in FIG. 17, the information processing apparatus 10 uses a constant N, for example, "10", in which the height of the rectangle of the reference area B is determined at the beginning of the calculation step of extracting the rectangular area for each character as the reference area. The height per frame is obtained by dividing, and the area including the selected writing set is divided in the vertical direction by a plurality of frames having the height per frame. For example, in the example of FIG. 17, the region below the reference region B is divided into 30 frames. In the example of FIG. 17, since the first character is already specified in the reference area given by the user, only the writing below the reference area is targeted for division.

Next, the information processing apparatus 10 calculates the number of writings existing in the frame for each divided frame. For each frame, the number of blank frames adjacent in the upward direction and the number of blank frames adjacent in the downward direction are calculated. As shown in FIG. 17, the calculated information is information of the number of frames × 3. Hereinafter, a frame having a writing count of 1 or more may be referred to as a “writing frame”, a frame having a writing count of 0 may be referred to as a “blank frame”, and a sequence of writing frames and blank frames may be referred to as a “frame array”.

In the case of handwritten writing, the size of the characters is not uniform, the size of the line spacing also varies, and there are also characters with many spaces above and below the writing, such as the Chinese numerals "1" and "2". do. In order to extract a character frame from such handwritten writing, in this embodiment, a frame array is sequentially searched, and one character is used by using four rules of extraction rule, confirmation rule, semi-extraction rule 1, and semi-extraction rule 2. Extract the estimated area.

FIG. 18 is a flowchart showing the procedure of the extraction process according to the second embodiment. The extraction process shown in FIG. 19 loops until the index i of the frame array becomes N. As shown in FIG. 19, the information processing apparatus 10 determines whether or not a continuous writing frame exists (step S301). The "continuous writing frame" referred to here is one or more continuous frames starting with a writing frame, and is data for which the following four rules are applicable or not to be determined.

At this time, if the continuous writing frame does not yet exist (step S301No), the information processing apparatus 10 determines whether or not the frame corresponding to the index i of the frame array is the writing frame (step S302).

Then, when the frame corresponding to the index i of the frame array is a writing frame (step S302Yes), the information processing apparatus 10 defines a continuous writing frame having the frame corresponding to the index i of the frame array as the first element (step S302Yes). Step S303). After that, the information processing apparatus 10 increments the index i of the frame array by one, and proceeds to the process of step S301. If the frame corresponding to the index i of the frame array is not a writing frame (step S302No), step S303 is skipped and the process proceeds to step S301.

On the other hand, when a continuous writing frame exists (step S301Yes), the information processing apparatus 10 adds a frame corresponding to the index i of the frame array to the continuous writing frame (step S304).

After that, the information processing apparatus 10 determines whether or not the extraction rule can be applied to the continuous writing frame (step S305). At this time, if the extraction rule is applicable (step S305Yes), the information processing apparatus 10 applies the extraction rule to the continuous writing frame (step S306).

Here, as an example, the above-mentioned "extraction rule" sets the number of continuous writing frames to be M or more and the number of blank frames at the left and right ends for a predetermined number M or less of N. The condition is that the total added is N or more. When this extraction rule is satisfied, the information processing apparatus 10 extracts an area for N frames centering on a continuous writing frame as a character frame.

19A to 19C are diagrams showing an example of extracting a character frame. 19A to 19C show an example of extracting a character frame when an extraction rule is applied. In FIGS. 19A-19C, blank frames are indicated by white blocks, while writing frames are indicated by gray blocks. In the examples of FIGS. 19A to 19C, an example in the case where N = 10 and M = 7 is shown. In the example shown in FIG. 19A, the number of continuous writing frames is 8 ≧ 7, and the smaller number of left and right blanks is 2, which is 8 + 2 + 2 ≧ 10, which meets the conditions. Further, in the example shown in FIG. 19B, if the number of continuous writing frames is 7 and an odd number, 10 frames including 1.5 minutes of left and right blank frames are extracted as character frames. Since characters may require a space between writing, when adding a frame to a continuous writing frame in the flowchart of FIG. 18, check whether it is a writing frame or a blank frame. Not. Further, as shown in FIG. 19C, a blank frame may be included in the continuous writing frame.

If the extraction rule is not applicable (step S305No), the information processing apparatus 10 determines whether or not the determination rule is applicable to the continuous writing frame (step S307). At this time, if the definite rule is applicable (step S307Yes), the information processing apparatus 10 applies the definite rule to the continuous writing frame (step S308).

Here, the above-mentioned "determination rule" is conditional on the number of consecutive writing frames being N and the existence of character frame candidates. When this confirmation rule is satisfied, the corresponding character frame candidate is extracted as a character frame, the character frame candidate is cleared, and the index of the loop is set at the final end of the corresponding character frame candidate. If there are no character frame candidates, N consecutive writing frames are extracted as they are as character frames. If the continuous writing frame reaches the end of the frame array and there is a character frame candidate, the corresponding character frame candidate is extracted as a character frame, the character frame candidate is cleared, and the loop index is the final of the character frame candidate. Set to the edge. If there is no character frame candidate, the current continuous writing frame is extracted as a character frame.

Further, when the determination rule is not applicable (step S307No), the information processing apparatus 10 determines whether or not the quasi-extraction rule 1 is applicable to the continuous writing frame (step S309). At this time, when the quasi-extraction rule 1 is applicable (step S309Yes), the information processing apparatus 10 applies the quasi-extraction rule 1 to the continuous writing frame (step S310).

Here, the above-mentioned "quasi-extraction rule 1" is a rule for dealing with a character having many upper and lower spaces such as "1", and is the number of continuous writing frames for a predetermined number M or less of N. Is less than M, and the total of the continuous writing frames and the number of blank frames at the left and right ends is N or more. When this quasi-extraction rule 1 is satisfied, the information processing apparatus 10 marks an area for N frames centered on a continuous writing frame as a character frame candidate.

20A and 20B are diagrams showing an example of a mark as a candidate for a character frame. 20A and 20B show an example of marking a character frame candidate when the quasi-extraction rule 1 is applied. The examples of FIGS. 20A and 20B also show an example in the case where N = 10 and M = 7. In the example shown in FIG. 20A, there are three or more blanks on the left and right sides of each of the four continuous writing frames, which satisfies the condition. The quasi-extraction rule 1 is only the extraction of candidates, and by further searching, there is a possibility that a more appropriate pattern matching the extraction rule will appear as shown in the example shown in FIG. 20B. Further, by proceeding with the search, if the semi-extraction rule 1 is matched again, the character frame candidate is replaced.

Further, when the quasi-extraction rule 1 is not applicable (step S309No), the information processing apparatus 10 determines whether or not the quasi-extraction rule 2 is applicable to the continuous writing frame (step S311). At this time, when the quasi-extraction rule 2 is applicable (step S311Yes), the information processing apparatus 10 applies the quasi-extraction rule 2 to the continuous writing frame (step S312).

Here, the above-mentioned "quasi-extraction rule 2" is a rule for extracting characters written in a narrow line spacing, which is narrower than the reference area given by the user. For example, it is a condition that the total of the number of continuous writing frames and the number of blank frames on the left and right is N'or more than the predefined number N'of N or less and M or more. When this quasi-extraction rule 2 is satisfied, the information processing apparatus 10 marks an area for N'frames centering on a continuous writing frame as a character frame candidate.

21A and 21B are diagrams showing an example of a mark as a candidate for a character frame. 21A and 21B show an example of marking a character frame candidate when the quasi-extraction rule 2 is applied. The examples of FIGS. 21A and 21B also show examples of cases where N = 10, M = 7, and N'= 8. In the example shown in FIG. 21A, the number of continuous writing frames is 5, and there are 2 blanks on each side, so 5 + 2 × 2 = 9 ≧ 8, which matches the condition. The quasi-extraction rule 2 is only the extraction of candidates, and by further searching, there is a possibility that a more appropriate pattern matching the extraction rule will appear as shown in FIG. 21B. Also, by proceeding with the search, if the semi-extraction rules 1 and 2 are matched again, the character frame candidates are replaced.

After that, when either the extraction rule or the confirmation rule is applied, the information processing apparatus 10 clears the continuous writing frame (step S313), increments the index i of the frame array by one, and proceeds to the process of step S301. do.

If neither the semi-extraction rule 1 nor the semi-extraction rule 2 is applicable, the index i of the frame array is incremented by one, and the process proceeds to step S301.

As described above, by executing the extraction process shown in FIG. 18, it is possible to extract the reference area corresponding to each line even when the Chinese character "1", the Chinese character "2", a small character, or the line spacing varies.

[User specification]
By extracting a rectangular area for each character from the writing set selected in the vertical search, and using each rectangle extracted for each character as a reference area, a horizontal search is performed for each reference area. When selecting a writing set divided for each line, an extra line may be extracted by searching in the vertical direction. An example thereof is shown in FIG. FIG. 22 is a diagram showing an example of designating a rectangle in the vertical direction. As shown in FIG. 22, when writing a plurality of lines, the same writer may write information different from the information to be selected with the same character size. In the example of FIG. 22, comments that are not place names are described in the lines immediately below the three lines of the three place names. If you specify a reference area and perform a search in the vertical direction with the intention of selecting three lines of place names, there is a possibility that even lines that do not need to be selected will be selected.

From the aspect of avoiding such excessive selection of rows, the information processing apparatus 10 may allow the user to specify an area to be selected without performing a search when selecting a writing set in the vertical direction. For example, by having the user specify a reference area for one character (step S1), accepting a downward drag operation from the user (step S2), and selecting a rectangular area (step S3), "large" and "west" are selected. And identifies the writing set of "trees". After specifying the writing set, the user extracts a rectangular area for each character from the writing set, uses each rectangle extracted for each character as a reference area, and searches each reference area in the left-right direction. You can choose to write in multiple lines according to your intentions.

[Writing direction]
In the above-mentioned Example 1, an example in which the writing direction is horizontal writing is given, but as a matter of course, even when the writing direction is vertical writing or diagonal writing, FIGS. 9, 13, 18, etc. Can be processed. For example, the same process can be realized by applying the search in the left direction described in the first embodiment by rotating in a predetermined search direction, for example, in an upward direction, a downward direction, or an oblique direction.

[Distributed and integrated]
Further, each component of each of the illustrated devices does not necessarily have to be physically configured as shown in the figure. That is, the specific form of distribution / integration of each device is not limited to the one shown in the figure, and all or part of them may be functionally or physically distributed / physically in any unit according to various loads and usage conditions. Can be integrated and configured. For example, a network of a reception unit 15a, a setting unit 15b, a first determination unit 15c, a first resetting unit 15d, a second determination unit 15e, a second resetting unit 15f, or a writing selection unit 15g as an external device of the information processing device 10. You may connect via. Further, another device has a reception unit 15a, a setting unit 15b, a first determination unit 15c, a first resetting unit 15d, a second determination unit 15e, a second resetting unit 15f, or a writing selection unit 15g, respectively, and has a network. By being connected and collaborating, the function of the information processing apparatus 10 may be realized.

[Writing selection program]
Further, the various processes described in the above embodiment can be realized by executing a program prepared in advance on a computer such as a personal computer or a workstation. Therefore, in the following, an example of a computer that executes a writing selection program having the same function as that of the above embodiment will be described with reference to FIG. 23.

FIG. 23 is a diagram showing a hardware configuration example of a computer that executes the writing selection program according to the first to third embodiments. As shown in FIG. 23, the computer 100 has an operation unit 110a, a speaker 110b, a camera 110c, a display 120, and a communication unit 130. Further, the computer 100 has a CPU 150, a ROM 160, an HDD 170, and a RAM 180. Each of these 110 to 180 parts is connected via the bus 140.

As shown in FIG. 23, the HDD 170 has a reception unit 15a, a setting unit 15b, a first determination unit 15c, a first resetting unit 15d, a second determination unit 15e, and a second resetting unit shown in the first embodiment. A writing selection program 170a that exhibits the same functions as the unit 15f and the writing selection unit 15g is stored. The writing selection program 170a includes a reception unit 15a, a setting unit 15b, a first determination unit 15c, a first resetting unit 15d, a second determination unit 15e, a second resetting unit 15f, and a writing selection unit 15g shown in FIG. As with each component of, it may be integrated or separated. That is, not all the data shown in the first embodiment may be stored in the HDD 170, and the data used for processing may be stored in the HDD 170.

Under such an environment, the CPU 150 reads the writing selection program 170a from the HDD 170 and develops the writing selection program 170a into the RAM 180. As a result, the writing selection program 170a functions as the writing selection process 180a, as shown in FIG. 23. The writing selection process 180a expands various data read from the HDD 170 into the area allocated to the writing selection process 180a in the storage area of the RAM 180, and executes various processes using the expanded various data. For example, as an example of the process executed by the writing selection process 180a, the process shown in FIGS. 9 and 13 is included. In the CPU 150, not all the processing units shown in the first embodiment need to operate, and it is sufficient that the processing units corresponding to the processes to be executed are virtually realized.

The writing selection program 170a may not necessarily be stored in the HDD 170 or the ROM 160 from the beginning. For example, the writing selection program 170a is stored in a "portable physical medium" such as a flexible disk inserted into the computer 100, that is, a so-called FD, CD-ROM, DVD disk, magneto-optical disk, or IC card. Then, the computer 100 may acquire and execute the writing selection program 170a from these portable physical media. Further, another computer or server device connected to the computer 100 via a public line, the Internet, a LAN, a WAN, or the like stores the writing selection program 170a, and the computer 100 acquires the writing selection program 170a from these. You may try to execute it.

The following additional notes will be further disclosed with respect to the embodiments including the above embodiments.

(Appendix 1) Accept the designation of the reference area as the reference for the writing search on the screen where the writing that has been handwritten is displayed.
A second search area of the reference area adjacent to the search direction in which the written search is performed is set, and a second search area adjacent to the search direction of the reference area and included in the first search area is included. Set the search area of
When the writing is included in the first search area, the first search area is reset so as to be adjacent to the search direction side of the writing.
When the difference between the end of the writing included in the first search area on the search direction side and the end of the writing included in the second search area on the search direction side is within a predetermined threshold value. The end of the second search area on the search direction side is reset to the same position as the end of the first search area on the search direction side.
If the first search area that has been set or reset does not contain writing, select the writing that is included in the first search area that the writing has already searched for, and the difference exceeds the predetermined threshold. In the case, the writing included in the second search area in which the writing has been searched is selected.
A writing selection program that lets a computer perform processing.

(Appendix 2) The process described in Appendix 1 is characterized in that the first search area and the second search area are set when the writing included in the reference area corresponds to one character. Written selection program.

(Appendix 3) The writing selection program according to Appendix 1, wherein the accepting process accepts designations of four sides corresponding to a rectangle in the reference area.

(Appendix 4) In the process to be set, a plurality of second search areas that do not match each other are set.
In the resetting process, the end portion of the second search region on the search direction side is reset for each of the plurality of second search regions.
In the process of selecting, any one of the differences obtained for each of the plurality of second search areas from the end of the writing included in the first search area on the search direction side is the predetermined value. The writing selection program according to Appendix 1, wherein when the threshold value is exceeded, the writing included in the second search area in which the difference exceeds the predetermined threshold value is selected.

(Appendix 5) The process to be set is one character from the reference area when the ratio of the size of the reference area in the search direction and the size of the reference area in the direction orthogonal to the search direction satisfy a predetermined condition. The writing selection program according to Appendix 1, wherein the rectangular area extracted for each is used as the reference area, and the first search area and the second search area are set for each reference area.

(Appendix 6) Accept the designation of the reference area as the reference for the writing search on the screen where the writing that has been handwritten is displayed.
A second search area of the reference area adjacent to the search direction in which the written search is performed is set, and a second search area adjacent to the search direction of the reference area and included in the first search area is included. Set the search area of
When the writing is included in the first search area, the first search area is reset so as to be adjacent to the search direction side of the writing.
When the difference between the end of the writing included in the first search area on the search direction side and the end of the writing included in the second search area on the search direction side is within a predetermined threshold value. The end of the second search area on the search direction side is reset to the same position as the end of the first search area on the search direction side.
If the first search area that has been set or reset does not contain writing, select the writing that is included in the first search area that the writing has already searched for, and the difference exceeds the predetermined threshold. In the case, the writing included in the second search area in which the writing has been searched is selected.
A written selection method in which the computer performs the process.

(Supplementary Note 7) The process described in Appendix 6 is characterized in that the first search area and the second search area are set when the writing included in the reference area corresponds to one character. Writing selection method.

(Supplementary Note 8) The writing selection method according to Supplementary Note 6, wherein the accepting process accepts designation of four sides corresponding to a rectangle of the reference area.

(Appendix 9) In the process to be set, a plurality of second search areas that do not match each other are set.
In the resetting process, the end portion of the second search region on the search direction side is reset for each of the plurality of second search regions.
In the process to be selected, any one of the differences obtained for each of the plurality of second search areas from the end of the writing included in the first search area on the search direction side is the predetermined value. The writing selection method according to Appendix 6, wherein when the threshold value is exceeded, the writing included in the second search area in which the difference exceeds the predetermined threshold value is selected.

(Appendix 10) The process to be set is one character from the reference area when the ratio of the size of the reference area in the search direction and the size of the reference area in the direction orthogonal to the search direction satisfy a predetermined condition. The writing selection method according to Appendix 6, wherein the rectangular area extracted for each is used as the reference area, and the first search area and the second search area are set for each reference area.

(Appendix 11) A reception unit that accepts the designation of the reference area used as the reference for the writing search on the screen on which the writing that has been handwritten is displayed.
A second search area of the reference area adjacent to the search direction in which the written search is performed is set, and a second search area adjacent to the search direction of the reference area and included in the first search area is included. The setting part that sets the search area of
When the writing is included in the first search area, the first resetting unit for resetting the first search area adjacent to the search direction side of the writing, and the first resetting unit.
When the difference between the end of the writing included in the first search area on the search direction side and the end of the writing included in the second search area on the search direction side is within a predetermined threshold value. A second resetting unit that resets the end of the second search area on the search direction side to the same position as the end of the first search area on the search direction side.
If the first search area that has been set or reset does not contain writing, select the writing that is included in the first search area that the writing has already searched for, and the difference exceeds the predetermined threshold. In the case, a writing selection unit that selects a writing included in the second search area in which the writing has been searched, and a writing selection unit.
An information processing device characterized by having.

(Supplementary Note 12) The information according to Supplementary Note 11, wherein the setting unit sets the first search area and the second search area when the writing included in the reference area corresponds to one character. Processing equipment.

(Supplementary Note 13) The information processing apparatus according to Supplementary Note 11, wherein the reception unit receives designations of four sides corresponding to a rectangle of the reference area.

(Appendix 14) The setting unit sets a plurality of second search areas that do not match each other.
The second resetting unit resets the end portion of the second search region on the search direction side for each of the plurality of second search regions.
In the writing selection unit, any one of the differences obtained between the plurality of second search areas and the end portion of the writing included in the first search area on the search direction side is the predetermined value. The information processing apparatus according to Appendix 11, wherein when the threshold value is exceeded, the writing included in the second search region in which the difference exceeds the predetermined threshold value is selected.

(Appendix 15) When the ratio of the size of the reference region in the search direction to the size of the reference region in the direction orthogonal to the search direction satisfies a predetermined condition, the setting unit shall be used for each character from the reference region. 11. The information processing apparatus according to Appendix 11, wherein the rectangular region extracted from the above is used as the reference region, and the first search region and the second search region are set for each reference region.

1a Output device 1b Environment sensor 10 Information processing device 11 Communication I / F unit 13 Storage unit 13a Written data 15 Control unit 15a Reception unit 15b Setting unit 15c 1st judgment unit 15d 1st resetting unit 15e 2nd judgment unit 15f 2nd Reset section 15g Writing selection section

Claims (7)

Accepts the specification of the reference area to be the reference for the writing search on the screen where the writing with handwriting input is displayed.
A second search area of the reference area adjacent to the search direction in which the written search is performed is set, and a second search area adjacent to the search direction of the reference area and included in the first search area is included. Set the search area of
When the writing is included in the first search area, the first search area is reset so as to be adjacent to the search direction side of the writing.
When the difference between the end of the writing included in the first search area on the search direction side and the end of the writing included in the second search area on the search direction side is within a predetermined threshold value. The end of the second search area on the search direction side is reset to the same position as the end of the first search area on the search direction side.
If the first search area that has been set or reset does not contain writing, select the writing that is included in the first search area that the writing has already searched for, and the difference exceeds the predetermined threshold. In the case, the writing included in the second search area in which the writing has been searched is selected.
A writing selection program that lets a computer perform processing. The writing selection program according to claim 1, wherein the processing to be set sets the first search area and the second search area when the writing included in the reference area corresponds to one character. .. The writing selection program according to claim 1 or 2, wherein the accepting process accepts designations of four sides corresponding to the rectangle of the reference area. In the process to be set, a plurality of second search areas that do not match each other are set.
In the resetting process, the end portion of the second search region on the search direction side is reset for each of the plurality of second search regions.
In the selected process, any one of the differences obtained between the plurality of second search regions and the end of the writing included in the first search region on the search direction side is the predetermined one. The writing selection program according to claim 1, 2 or 3, wherein when the threshold value is exceeded, the writing included in the second search area in which the difference exceeds the predetermined threshold value is selected. The process to be set is extracted character by character from the reference area when the ratio of the size of the reference area in the search direction to the size of the reference area in the direction orthogonal to the search direction satisfies a predetermined condition. The writing selection according to any one of claims 1 to 4, wherein the rectangular area is set as the reference area, and the first search area and the second search area are set for each reference area. program. Accepts the specification of the reference area to be the reference for the writing search on the screen where the writing with handwriting input is displayed.
A second search area of the reference area adjacent to the search direction in which the written search is performed is set, and a second search area adjacent to the search direction of the reference area and included in the first search area is included. Set the search area of
When the writing is included in the first search area, the first search area is reset so as to be adjacent to the search direction side of the writing.
When the difference between the end of the writing included in the first search area on the search direction side and the end of the writing included in the second search area on the search direction side is within a predetermined threshold value. The end of the second search area on the search direction side is reset to the same position as the end of the first search area on the search direction side.
If the first search area that has been set or reset does not contain writing, select the writing that is included in the first search area that the writing has already searched for, and the difference exceeds the predetermined threshold. In the case, the writing included in the second search area in which the writing has been searched is selected.
A written selection method in which the computer performs the process. A reception unit that accepts the designation of the reference area to be the reference for the writing search on the screen where the writing with handwriting input is displayed.
A second search area of the reference area adjacent to the search direction in which the written search is performed is set, and a second search area adjacent to the search direction of the reference area and included in the first search area is included. The setting part that sets the search area of
When the writing is included in the first search area, the first resetting unit for resetting the first search area adjacent to the search direction side of the writing, and the first resetting unit.
When the difference between the end of the writing included in the first search area on the search direction side and the end of the writing included in the second search area on the search direction side is within a predetermined threshold value. A second resetting unit that resets the end of the second search area on the search direction side to the same position as the end of the first search area on the search direction side.
If the first search area that has been set or reset does not contain writing, select the writing that is included in the first search area that the writing has already searched for, and the difference exceeds the predetermined threshold. In the case, a writing selection unit that selects a writing included in the second search area in which the writing has been searched, and a writing selection unit.
An information processing device characterized by having.

Images