JP2018128850A - Information processing program, information processing method, and information processing device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To reduce a work burden for associating contents with a letter-string information recognized from a trace of a handwriting input.SOLUTION: An information processing program causes a computer to execute processes of: recognizing first letter-string information from a trace of a handwriting input; receiving an operation of associating contents with the recognized first letter-string information among multiple kinds of contents; recognizing second letter-string information from a trace of a handwriting input; specifying respective positions where the recognized first and second letter-string information are handwritten; and when the relationship between the respective specified positions where the handwriting input have been made satisfies a preset condition, associating the specified contents with the second letter-string information on the basis of the recognized second letter-string information and on the kind of the contents associated with the first letter-string information.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an information processing program, an information processing method, and an information processing apparatus.

Large screen displays and projectors are increasingly used in meetings and brainstorming, where many participants work together to create new things and ideas. It is not limited to business situations, and not limited to the use of large screen displays and projectors.
・ In order to decide where to go for school excursions at school class meetings, enumerate potential destinations on electronic whiteboards ・ In history classes, teachers enumerate names of historical figures on electronic whiteboards In addition, it is also possible to enumerate cooking candidates on a notebook PC (Personal Computer).

  In such a place, the idea is written down on the screen by handwriting input by the user. At this time, it is desired that content such as an image corresponding to the character string input by handwriting is displayed together. For example, it is expected that the image of “XX tower” is easier to understand than the case of only handwritten input “XX tower”, and a meeting or the like is effective. Since a technique for performing character recognition from handwritten input has been established, it is possible to acquire related content from a character recognition result by Web search or image search.

  On the other hand, a technique is disclosed that aims to reduce a user's work burden of recalling an operation sequence to be executed according to the state of the system (see, for example, Patent Document 1). In addition, a technique for assigning highly accurate input candidates and realizing quick input with a simple operation is disclosed (see, for example, Patent Document 2). In addition, a technique for accurately searching and presenting necessary information without disturbing an input operation of a person in charge is disclosed (see, for example, Patent Document 3). In addition, a technique is disclosed that aims to prevent missing of information representing the relationship between keywords and to enable highly accurate information retrieval by effectively using conceptual information (for example, Patent Document 4). See). However, none of them considers a situation in which a plurality of candidates are listed by handwriting input, and contents are acquired and associated based on character string information recognized from a handwritten input locus.

JP-A-10-149246 JP 2009-223905 A JP 2011-053791 A JP-A-11-259524

  As described above, conventionally, when a user enumerates a plurality of candidates for discussion or the like, the user performs handwriting input, acquires content based on character string information recognized from a locus input by handwriting, and associates the content. There was a problem that the operation had to be repeated and the operation was complicated.

  Therefore, an object of one aspect is to reduce the work burden for associating content with character string information recognized from a trajectory input by handwriting.

  In one embodiment, the first character string information is recognized from a trajectory input by handwriting, and an operation for associating the content with the recognized first character string information is received from a plurality of types of content, and is input by handwriting. The second character string information is recognized from the trajectory, the respective positions where the recognized first and second character string information are input by handwriting are specified, and the respective positional relationships where the specified handwriting input is performed When the predetermined condition is satisfied, the identified content is determined based on the recognized second character string information and the content type associated with the first character string information. Causes the computer to execute a process corresponding to the character string information.

  It is possible to reduce the work load for associating content with character string information recognized from a handwritten input locus.

It is a figure which shows the function structural example of the information processing apparatus concerning one Embodiment. It is a figure which shows the example of document data. It is a figure which shows the example of a word attribute table. It is a figure which shows the example of a search process table. It is a figure which shows the example of line processing data. FIG. 11 is a diagram illustrating another functional configuration example of the information processing apparatus. It is a figure which shows the hardware structural example of information processing apparatus. It is a flowchart which shows the 1st process example. It is a figure which shows the example of the state in which the handwriting input was performed. It is a figure which shows the example of range specification. It is a figure which shows the example of calculation of reference | standard coordinate information. It is a figure which shows the example of selection of a search process. It is a figure which shows the example of a display of a search result. It is a figure which shows the example of the predetermined | prescribed conditions considered that the locus | trajectory group of the handwritten input immediately before from the arrangement | positioning coordinate of the content by a user, and the said content comprise one element of enumeration. It is a figure which shows the example of the new handwriting input after determining with a package. It is a figure which shows the example by which range selection was carried out by timeout. It is a figure which shows the example of the conditions of whether it is the object of a repetition process. It is a figure which shows the example of calculation of the coordinate value of automatic arrangement | positioning, and a guide display. It is a figure which shows the example of search process automatic execution and search result automatic arrangement | positioning. It is a flowchart which shows the 2nd process example. It is a figure which shows the example of selection of a search process. It is a figure which shows the example of a display of a search result. It is a figure which shows the example by which the search result was arrange | positioned by the user. It is a figure which shows the example of selection of a search process. It is a figure which shows the example of a display of a search result. It is a figure which shows the example by which the search result was arrange | positioned by the user. It is a figure which shows the example of line processing data. It is a figure which shows the example of the new handwriting input after determining with a package. It is a figure which shows the example by which range selection was carried out by timeout. It is a figure which shows the example of search process automatic execution and search result automatic arrangement | positioning. It is a figure which shows the example in case enumeration is performed about a some theme. It is a flowchart which shows the 3rd processing example.

  In the following, preferred embodiments of the present invention will be described with reference to the drawings.

<Configuration>
FIG. 1 is a diagram illustrating a functional configuration example of an information processing apparatus 1 according to an embodiment, which is a stand-alone type example. In FIG. 1, an information processing device 1 is a device such as an electronic white board device, a personal computer, or a tablet. The information processing apparatus 1 includes an input receiving unit 101, a display unit 102, a document management unit 103, and document data 104. The information processing apparatus 1 also includes a writing set selection unit 105, a reference coordinate calculation unit 106, a handwritten character recognition unit 107, an attribute determination unit 108, a word attribute table 109, a search processing determination unit 110, a search processing table 111, and a process selection execution. A section 112 and an arrangement section 113 are provided. In addition, the information processing apparatus 1 includes a row processing determination unit 114, a row processing data 115, a coordinate matching determination unit 116, an automatic processing application unit 117, and an automatic placement unit 118.

  The input receiving unit 101 has a function of receiving operations such as handwriting input (writing input), text input, menu selection, and the like by a user (user) and transmitting the received input operation content to the document management unit 103. Yes. The display unit 102 has a function of displaying a document screen, an operation menu, and the like on a monitor such as a panel type display or a projector.

  The document management unit 103 has a function of managing, as document data 104, a document in which multimedia contents such as contents of handwritten input by a user and images and texts associated therewith are arranged. A document screen is displayed by the display unit 102 based on the document data 104. FIG. 2 is a diagram showing an example of the document data 104, which includes handwritten input data (writing input data) and content data with a display position on the screen. The handwritten input data includes coordinate information of a locus input by handwriting.

  Returning to FIG. 1, the writing set selection unit 105 has a function of automatically selecting a part of a locus of handwritten input data in a document by a range designation operation or time-up by a user. The reference coordinate calculation unit 106 has a function of calculating reference coordinate information indicating the positional characteristics (for example, coordinates of a circumscribed rectangle) of the handwritten region from the locus of the handwritten input data selected by the writing set selection unit 105. ing. The handwritten character recognizing unit 107 has a function of performing handwritten character recognition from the locus of the handwritten input data selected by the writing set selecting unit 105 and converting it into a text character string.

  The attribute determination unit 108 has a function of referring to the word attribute table 109 from the text character string converted by the handwritten character recognition unit 107 and extracting an attribute list corresponding to the text character string. FIG. 3 is a diagram showing an example of the word attribute table 109, in which attribute lists are associated with various words. For example, the attribute list {place name, prefecture name, city name} is associated with the word “Okinawa”.

  Returning to FIG. 1, the search processing determination unit 110 has a function of referring to the search processing table 111 based on the attribute list extracted by the attribute determination unit 108 and extracting a search processing list corresponding to each attribute in the attribute list. Have. FIG. 4 is a diagram showing an example of the search processing table 111, in which search processing is associated with attributes. For example, the attribute “prefecture name” is associated with search processing “prefectural office location”, “prefectural governor”, “population”, “area”, and the like, and a collection of these is a search processing list.

  Returning to FIG. 1, the process selection execution unit 112 causes the user to select a desired search process from the search process list extracted by the search process determination unit 110 and selects a text string recognized from the user's handwritten input. Has a function of executing the retrieved processing. The arrangement unit 113 has a function of causing a user to arrange content based on multimedia content data obtained as a result of the executed search process on a document managed by the document management unit 103. The arranged result is held as the display position of the corresponding content data in the document data 104.

  The line processing determination unit 114 has a function of determining whether or not a predetermined grouping condition that the content of the immediately preceding handwritten input and the content constitutes one element of the enumeration is determined based on the arrangement coordinates of the content by the user. . When the condition is satisfied, a set of “reference coordinate information” and “arranged coordinate information, selected search process, attribute corresponding to the search process” is stored in the row processing data 115. “Reference coordinate information” is information indicating the positional characteristics (for example, coordinates of a circumscribed rectangle) of the handwritten region where handwritten input is performed. “Arranged coordinate information” is information indicating the position where the content is arranged by the user. “Selected search process” is information indicating the search process selected by the user from the search process list in the acquisition of content data. The search process indicates how the content as a search result has been acquired, and represents the type of content. The “attribute corresponding to the search process” is an attribute corresponding to the text string of the handwritten input used in the extraction of the search process list.

  FIG. 5 is a diagram showing an example of the row processing data 115. The reference coordinate information “x0, y0, x1, y1”, the attribute “prefecture name”, the search processing “image search”, and the arrangement coordinates “x2, y2, x3”. “y3” indicates a recorded state. In addition, when preparing a plurality of conditions that are not satisfied at the same time as conditions for the position of the content arranged for the user (for example, conditions for enumerating in the horizontal direction, conditions for enumerating in the vertical direction, etc.) Information indicating the satisfied condition is included in the row processing data 115. This satisfied condition is referred to in the position calculation when the content is automatically arranged later.

  Returning to FIG. 1, the coordinate matching determination unit 116 operates when there is a record in the row processing data 115 at the time when a new handwriting input is performed. That is, the coordinate matching determination unit 116 compares the reference coordinate information calculated from the newly performed handwriting input with the reference coordinate information recorded in the line processing data 115, and determines in advance whether or not it is an object of repeated processing. It has a function of determining whether or not a predetermined condition is satisfied. The automatic process application unit 117 performs an operation when the coordinate matching determination unit 116 determines that a predetermined condition is satisfied. That is, when the line processing data 115 has an attribute that matches the attribute corresponding to the newly-written text string of the handwritten input, the automatic processing application unit 117 performs the recorded search process with the new text of the handwritten input. It has a function that applies to character strings. The automatic arrangement unit 118 has a function of calculating the arrangement coordinates of the content obtained as a result of applying the search process by the automatic process application unit 117 and arranging it on the document managed by the document management unit 103.

  FIG. 6 is a diagram illustrating another example of the functional configuration of the information processing apparatus 1. The information processing apparatus 11 responsible for screen display and input reception from the information processing apparatus 1 (FIG. 1) and information for performing other processes. The processing apparatus 12 is separated. The information processing apparatus 12 may be a server apparatus that is directly or wiredly connected to the information processing apparatus 11 or may be a server apparatus that performs processing as a cloud service or the like via a network such as the Internet.

  6, among the components of the information processing apparatus 1 shown in FIG. 1, the input receiving unit 101 and the display unit 102 are provided in the information processing device 11, and other components are provided in the information processing device 12. . The functions of each component are the same as those described above, but a data transmission / reception function is added between the components separated into the information processing apparatus 11 and the information processing apparatus 12. Various modifications are possible, and some of the components arranged in the information processing apparatus 12 may be arranged on the information processing apparatus 11 side.

  FIG. 7 is a diagram illustrating a hardware configuration example of the information processing apparatuses 1, 11, and 12. In FIG. 7, the information processing apparatus 1 or the like includes a CPU (Central Processing Unit) 1001, a ROM (Read Only Memory) 1002, a RAM (Random Access Memory) 1003, and an HDD (Hard Disk Drive) connected to each other via a bus 1007. ) / SSD (Solid State Drive) 1004. Further, the information processing apparatus 1 and the like include a connection I / F (Interface) 1005 and a communication I / F 1006. In the case of the information processing apparatuses 1 and 11 having functions of screen display and input reception, a monitor 1008 and a pointing device 1009 are connected to the connection I / F 1005. The monitor 1008 includes a projector in addition to a panel type display. The pointing device 1009 includes a mouse, a touch panel, and the like.

  The CPU 1001 controls the operations of the information processing apparatuses 1, 11, and 12 by executing a program stored in the ROM 1002 or the HDD / SSD 1004 using the RAM 1003 as a work area. The functions of the information processing apparatuses 1, 11, and 12 described with reference to FIGS. 1 and 6 are realized by a predetermined program being executed by the CPU 1001. The program may be acquired via a recording medium, may be acquired via a network, or may be embedded in a ROM.

<First processing example>
FIG. 8 is a flowchart showing a first processing example of the above embodiment.

  In FIG. 8, when the process is started, the input receiving unit 101 receives handwritten input (step S101), determines whether or not the range is specified (step S102), and if the range is not specified (NO in step S102). ) Returns to accepting handwritten input (step S101). FIG. 9 shows a state where the letters “Okinawa” are handwritten on an electronic white board. The locus by handwriting input is transferred from the input receiving unit 101 to the document management unit 103 and held in the document data 104 as handwritten input data.

There are multiple implementation methods for range specification,
-There are things that allow you to specify a range by using a surrounding gesture. The enclosing gesture is an operation input that encloses the input characters with a line. When the pointing device is operated so as to surround a character, it is regarded as a gesture command instead of handwritten input, and the enclosed range is designated. FIG. 10 shows an example in which the range around the character “Okinawa” by handwriting input is specified by a surrounding gesture.

  In the rectangle selection mode, the range is specified by switching from the writing mode to the rectangle selection mode, and specifying the coordinates of the two points with a pointing device such as a mouse. The method specified. Mode switching is performed by a pop-up menu or the like.

  For time-outs, the time between handwriting input is measured, a time-out event is issued when the time exceeds a certain threshold, and the locus of handwriting input from the previous time-out to the new time-out is specified It is a method that is supposed to be. Alternatively, a method may be used in which the beginning of a set of trajectories is designated by timeout, and only the end of the set of trajectories is selected from a menu. Alternatively, a method may be used in which the beginning of a set of trajectories is designated by a timeout, and the end of the set of trajectories is determined based on whether a line feed has been performed. Regarding the line feed determination of written data, for example, there is a method described in Japanese Patent No. 3897999.

  Returning to FIG. 8, when it is determined that the range has been designated (YES in step S <b> 102), the writing set selection unit 105 acquires the handwritten input data of the locus of the designated range from the document data 104 via the document management unit 103. . Then, the reference coordinate calculation unit 106 calculates reference coordinate information (step S103). FIG. 11 is a diagram showing an example of calculation of reference coordinate information, and four values (x0, y0, x1, x1) representing the coordinates of the upper left corner and the lower right corner of the circumscribed rectangle of the locus set input by handwriting within the designated range. This is an example in which y1) is used. The reference coordinate information is not limited to this, but may be other definitions.

  Returning to FIG. 8, the handwritten character recognizing unit 107 then performs handwritten character recognition on the locus set selected by the range specification, and acquires a text string recognition result character string (step S <b> 104). Handwritten character recognition uses a dictionary in which trajectory features that make up each character are defined for each character, and identifies characters from the similarity of trajectory features. A known method is used. In the example of FIG. 10, the text character string “Okinawa” is acquired as the recognition result character string.

  In FIG. 8, handwritten character recognition (step S104) is performed following calculation of the reference coordinate information (step S103). However, the processing order may be reversed or executed in parallel. May be.

  Next, the attribute determination unit 108 refers to the word attribute table 109 from the recognition result character string acquired by the handwritten character recognition unit 107 and extracts an attribute list corresponding to the text character string (step S105). For example, the attribute list {place name, prefecture name, city name} is extracted from the text character string “Okinawa” by referring to the word attribute table 109 of FIG.

  Returning to FIG. 8, next, the row processing determination unit 114 determines whether or not there is a record in the row processing data 115 (step S106), and the processing branches. When it is determined that there is no record in the row processing data 115 (NO in step S106), the search processing determination unit 110 refers to the search processing table 111 based on the attribute list extracted by the attribute determination unit 108. Then, a union of search processes corresponding to each attribute in the attribute list is extracted as a search process list (step S107). For example, in the case of the attribute list {place name, prefecture name, city name}, the search processing table 111 in FIG. 4 is referred to, and from the attribute “prefecture name”, “prefectural office location” “prefectural governor” “population” "Area" etc. are extracted. Other attributes are extracted if there is a search process registered in the search process table 111.

  Returning to FIG. 8, the process selection execution unit 112 then displays the search process list extracted by the search process determination unit 110 and allows the user to select a search process (step S108). FIG. 12 is a diagram showing an example of search processing selection. In FIG. 12A, an attribute selection menu is displayed, and a desired attribute is selected by the user. For example, when the attribute “Okinawa (prefecture name)” is selected, a search process selection menu corresponding to the attribute “Okinawa (prefecture name)” is displayed as shown in FIG. Is selected. FIG. 12 shows an example in which selection is performed in two stages, that is, attribute selection and search process selection. This can reduce the number of search process lists to be presented, but the search process can be performed in one stage. It may be selected. In addition, when there are a wide variety of attributes and search processes, selection may be performed in multiple stages by grouping attributes and search processes.

  Returning to FIG. 8, the process selection execution unit 112 then executes the selected search process on the text character string recognized from the user's handwritten input (step S <b> 109). Then, the arrangement unit 113 arranges the multimedia content data obtained as a result of the executed search process on the document managed by the document management unit 103 (step S110). That is, the placement unit 113 temporarily displays the search result content at a predetermined position, and accepts a display position moving operation (such as a drag operation with a mouse) by the user. The search process includes a process that includes an operation for selecting a result desired by the user from a search result list, such as an image search, and a process that uniquely determines a search result from a character string, such as a prefectural office location search. Exists. FIG. 13 is a diagram showing an example of display of search results, and FIG. 13A shows a state where a search result list searched for images by the text character string “Okinawa” is displayed. FIG. 13B shows a state in which one search result is selected from the search result list and arranged side by side to the right of “Okinawa” by handwriting input with the intention of being associated with “Okinawa” by the user. Yes.

Returning to FIG. 8, the line processing determining unit 114 then determines whether or not a predetermined grouping condition that the content of the immediately preceding handwritten input and the content constitute one element of the enumeration from the content arrangement coordinates by the user is satisfied. (Step S111). FIG. 14 is a diagram showing an example of a predetermined condition in which the immediately preceding handwritten input trajectory group and the content are regarded as one element of the enumeration from the content arrangement coordinates by the user. That is, “the left side of the arrangement data is to the right of the right side of the handwritten data circumscribing rectangle of the reference coordinate information, and the Y coordinate of the horizontal center line of the arrangement data is the upper and lower sides of the handwritten data circumscribing rectangle of the reference coordinate information. It is an example of “between”. If expressed by mathematical formulas, the coordinates of the upper left and lower right corner points of the handwritten data circumscribed rectangle are (x0, y0) (x1, y1), and the coordinates of the upper left and lower right corner points of the arrangement data are (x2, y2) ( x3, y3), as shown in FIG.
X2> x1 and (y3 + y2) / 2> y0 and (y3 + y2) / 2 <y1
Is required to be satisfied. The arrangement relationship in FIG. 14A shows a case where the above condition is satisfied. This example means that the search result is arranged in a horizontal position with the handwritten data, but a condition corresponding to the case where the search result is arranged in the vertical direction or the case where the search result is arranged in the oblique direction may be provided separately. For example, as a condition of being arranged in the vertical direction, “the upper side of the arrangement data is below the lower side of the handwritten data circumscribing rectangle of the reference coordinate information, and the X coordinate of the left side of the arrangement data is the handwriting of the reference coordinate information The left side of the data circumscribed rectangle is within 50 pixels ".

  Returning to FIG. 8, when it is determined that a predetermined condition considered to constitute one element of the enumeration is satisfied (YES in step S <b> 111), the row processing determination unit 114 records the row processing data 115. That is, the row processing determination unit 114 records a set of “reference coordinate information” and “arranged coordinate information, selected search processing, attribute corresponding to the search processing” in the row processing data 115 (step S112). . As a predetermined condition considered to constitute one element of the enumeration, when a plurality of conditions to be satisfied at the same time are provided, information indicating the satisfied condition is also included in the row processing data. The row processing data 115 recorded as a result of the example shown in FIGS. 9 to 14 is as shown in FIG. That is, the reference coordinate information “x0, y0, x1, y1”, the attribute “prefecture name”, the search process “image search”, and the arrangement coordinates “x2, y2, x3, y3” are recorded as the row process data 115.

  Returning to FIG. 8, after recording the row processing data 115 (step S112), the processing is repeated from acceptance of handwriting input (step S101). When it is determined that the predetermined condition considered to constitute one element of the enumeration is not satisfied (NO in step S111), the line processing data 115 is not recorded (step S112), and handwriting input is accepted (step S101). The process is repeated.

  On the other hand, if it is determined that there is a record in the row processing data 115 (YES in step S106), it means that it is determined that the immediately preceding handwriting input and the content arrangement constitute one element of the enumeration. FIG. 15 illustrates a state in which “Hiroshima” is newly input by handwriting after it is determined that “Okinawa” input by handwriting and the image of the search result constitute one element of the enumeration. FIG. 16 shows a state in which “Hiroshima” has been designated as a range for handwritten character recognition (range selection) due to timeout.

Returning to FIG. 8, the coordinate matching determination unit 116 then compares the reference coordinate information calculated from the newly performed handwriting input with the reference coordinate information recorded in the row processing data 115 (step S <b> 113). Then, the coordinate matching determination unit 116 determines whether or not a predetermined condition as to whether or not it is an object of repetition processing is satisfied (step S114). FIG. 17 is a diagram showing an example of a condition for determining whether or not to subject to repeated processing.
“The upper side of the circumscribed rectangle of the reference coordinate information of the newly performed handwriting input is below the lower side of the circumscribed rectangle of the reference coordinate information of the line processing data 115,
And the distance is within 300 points,
In addition, the difference between the left side of the circumscribed rectangle of the reference coordinate information newly written and the left side of the reference coordinate information of the row processing data 115 is within 50 points.
An example is shown. If expressed by mathematical formulas, the reference coordinate information of the row processing data 115 is (x0, y0, x1, y1), and the newly performed reference coordinate information of handwritten input is (x0 ′, y0 ′, x1 ′, y1 ′). Then, as shown in FIG.
y0 ′> y1 and y0′−y1 ≦ 300 and
| x0′−x0 | ≦ 50
Is required to be satisfied. In FIG. 17A, a new handwritten input “Hiroshima” satisfies the above conditions.

  Returning to FIG. 8, when it is determined that the condition of whether or not to be the target of repeated processing is satisfied (YES in step S <b> 114), the automatic processing application unit 117 has an attribute corresponding to the newly-written text string of handwritten input. The attributes of the list and the row processing data 115 are compared (step S115). Then, the automatic process application unit 117 determines whether there is a matching attribute (step S116).

  If it is determined that there is a matching attribute (YES in step S116), the automatic process application unit 117 performs the text processing obtained by the handwritten character recognition unit 107 from the new handwritten input on the search process recorded in the line processing data 115. It is applied to the character string and executed (step S117).

  Next, the automatic arrangement unit 118 calculates new coordinate values from the new handwritten input reference coordinate information, the reference coordinate information of the line processing data 115, and the arrangement coordinate information (step S118), and automatically retrieves the search result data as the coordinate values. Arrange (step S119). Thereafter, the processing is repeated from acceptance of handwriting input (step S101).

  For example, in FIG. 18A, the coordinate values for automatically arranging the search result data are the y coordinate y0 ′ representing the upper side of the circumscribed rectangle of the handwritten input of the second row to be repeated, and the search result of the first row. Can be the coordinates (x2, y0 ′) represented by the x coordinate x2.

  Prior to handwriting input on the second line, for example, as shown in FIG. 18B, the Y coordinate ya of the upper side of the second line to be repeated is set to ya = y1 + (y1−y0) / 2. , (X0, ya), (x1, ya + y1-y0) are calculated, and handwritten input on the second line to be repeated can be facilitated by guide display with rectangles using the calculated corner points. Further, the display position of the search result when the iterative process is performed can be predicted by the user by the guide display with a rectangle having the coordinates (x2, ya) and (x3, ya + y3-y2) as corner points. You can also.

  FIG. 19 is a diagram illustrating an example of automatic execution of search processing and automatic arrangement of search results. When “Hiroshima” input by handwriting under “Okinawa” is determined to be an object of repetition processing and has matching attributes, An example in the case of determination is shown. Here, since an image search is executed as the search process, a search result list is displayed as shown in FIG. When a desired search result is selected by the user from among them, as shown in FIG. 19B, the selected search result is the second-line handwritten input “Hiroshima” and the first-line search. The result is automatically placed at a position aligned with both.

  Returning to FIG. 8, when it is determined that a predetermined condition as to whether or not the object is an object of repeated processing is not satisfied (NO in step S114), or when it is determined that there is no matching attribute (NO in step S116), The following processing is performed. That is, the coordinate matching determination unit 116 or the automatic process application unit 117 deletes the record of the row processing data 115 via the row processing determination unit 114 (step S120). Then, the processing is continued from extraction of the search processing list (step S107). That is, the same processing as when the row processing data 115 is not recorded is performed.

<Second processing example>
In the first processing example described above, one search process is performed in one element of the list. However, in the second process example, a plurality of search processes are allowed to be performed.

  FIG. 20 is a flowchart showing a second processing example. 20, steps S201 to S207 are the same as steps S101 to S107 of the first processing example shown in FIG. For example, the handwriting input “Okinawa” is performed in the same manner as in FIG. 9, the range is specified in the same manner as in FIG. 10, and the reference coordinate information is calculated in the same manner as in FIG. In addition, an attribute list {place name, prefecture name, city name} is extracted from the text string of “Okinawa” recognized by handwritten characters, and “prefectural office location”, “prefectural governor”, “ Population, area, etc. are extracted.

  In FIG. 20, thereafter, a loop process is performed until the process selection by the user is completed (steps S208 to S214). In the loop process, the process selection execution unit 112 displays the search process list extracted by the search process determination unit 110 and causes the user to select a search process (step S209). FIG. 21 is a diagram showing an example of search processing selection. In FIG. 21A, an attribute selection menu is displayed, and a desired attribute is selected by the user. For example, when the attribute “Okinawa (prefecture name)” is selected, a search process selection menu corresponding to the attribute “Okinawa (prefecture name)” is displayed as shown in FIG. Selected. FIG. 21 shows an example in which selection is performed in two stages, that is, attribute selection and search process selection. This can reduce the number of search process lists to be presented, but the search process can be performed in one stage. It may be selected. In addition, when there are a wide variety of attributes and search processes, selection may be performed in multiple stages by grouping attributes and search processes.

  Returning to FIG. 20, the process selection execution unit 112 then executes a search process selected for the text character string recognized from the user's handwritten input (step S <b> 210). The arrangement unit 113 arranges the multimedia content data obtained as a result of the executed search process on the document managed by the document management unit 103 (step S211). That is, the placement unit 113 temporarily displays the search result content at a predetermined position, and accepts a display position moving operation (such as a drag operation with a mouse) by the user. The search process includes a process that includes an operation for selecting a result desired by the user from a search result list, such as an image search, and a process that uniquely determines a search result from a character string, such as a prefectural office location search. Exists. FIG. 22 is a diagram showing an example of display of the search result, and shows a state where the search result of the population searched by the text character string “Okinawa” is displayed at the predetermined position. FIG. 23 shows a state where population search results are arranged on the right side of “Okinawa” by handwriting input with the intention of associating with “Okinawa” by the user.

  Returning to FIG. 20, the line processing determination unit 114 then determines whether or not a predetermined grouping condition that the content of the previous handwritten input and the content constitute one element of the enumeration from the user's content arrangement coordinates is satisfied. (Step S212). The predetermined conditions are the same as those described in FIG.

  Returning to FIG. 20, when it is determined that a predetermined condition that is considered to constitute one element of the enumeration is satisfied (YES in step S212), the row processing determination unit 114 records the row processing data 115 (step S212). S213). That is, the row processing determination unit 114 records a set of “reference coordinate information” and “arranged coordinate information, selected search processing, attribute corresponding to the search processing” in the row processing data 115. If there is no reselection request from the user (for example, if there is no reselection request within a predetermined time), the loop processing is terminated (step S214), and the processing is repeated from acceptance of handwriting input (step S201).

  When there is a reselection request from the user, the same loop processing as described above is performed. The reselection request from the user is made by selection from a pull-down menu, right mouse click, or the like. FIG. 24 is a diagram showing an example of selection of search processing. In FIG. 24A, the search processing selection menu corresponding to the already selected attribute “Okinawa (prefecture name)” is redisplayed, and FIG. b) shows a state in which, for example, the search process “area” is selected. FIG. 25 is a diagram showing an example of display of the search result, and shows a state where the search result of the area searched by the text character string “Okinawa” is displayed at a predetermined position. FIG. 26 shows a state where the area search results are arranged to the right of the search results of “Okinawa” and the population by handwriting input with the intention of associating with “Okinawa” by the user. FIG. 27A shows an example of the position of handwritten input and the arrangement of content data. FIG. 27B shows line processing data recorded when these are considered to constitute one element of the enumeration. 115 examples are shown. The line processing data 115 includes handwritten input reference coordinate information “x0, y0, x1, y1”, a first search processing attribute “prefecture name”, a search processing “population search”, and arrangement coordinates “x2, y2,”. x3, y3 ". The row processing data 115 also includes the attribute “prefecture name” of the second search process, the search process “area search”, and the arrangement coordinates “x4, y4, x5, y5”. In addition, when the search process is selected three times or more and it is considered that they constitute one element of the enumeration, a set of attributes, search processes, and arrangement coordinates corresponding to the number is recorded.

  Returning to FIG. 20, among the processes after it is determined that there is a record in the row process data 115 (YES in step S <b> 206), steps S <b> 215 to S <b> 218 and S <b> 224 are the processes of the first process example illustrated in FIG. 8. This is the same as steps S113 to S116 and S120. For example, as shown in FIG. 28, after “Okinawa” input by handwriting and the population and area of the search result are determined to be one element of enumeration, “Hiroshima” is newly input by handwriting, as shown in FIG. Moreover, “Hiroshima” is designated as a range for handwritten character recognition (range selection) due to timeout. Then, when the position of “Hiroshima” is determined to be the object of repeated processing, attribute comparison is performed.

  In FIG. 20, when it is determined that there is a matching attribute (YES in step S218), loop processing is performed for the number of search processes corresponding to the matching attribute (steps S219 to S223).

  In the loop process, the automatic process application unit 117 applies the search process recorded in the line process data 115 to the text character string obtained by the handwritten character recognition unit 107 from the new handwritten input (step S220). Next, the automatic arrangement unit 118 calculates new coordinate values from the new handwritten input reference coordinate information, the reference coordinate information of the line processing data 115, and the arrangement coordinate information (step S221), and automatically retrieves the search result data as the coordinate values. Arrange (step S222). FIG. 30 shows a state in which a population search and an area search are automatically performed for “Hiroshima” and search results are automatically arranged, as in “Okinawa”. When the loop process ends, the process is repeated from accepting handwritten input (step S201).

<Third processing example>
In the first and second processing examples described above, enumeration for one theme is targeted, but in the third processing example, a case where enumeration is performed for a plurality of themes is also permitted. For example, on a large-screen electronic whiteboard, completely different discussions may occur in parallel. FIG. 31 shows a list of destination candidates for determining the destination of the school excursion on the left half, and a list on the right half for discussion of the transition of ancient dynasties in China.

  FIG. 32 is a flowchart showing a third processing example. 32, steps S301 to S312 are the same as steps S101 to S112 of the first processing example shown in FIG. In the third processing example, a plurality of sets of row processing data 115 as shown in FIG. 5 are allowed.

  In FIG. 32, when it is determined that there is a record in the row processing data 115 (YES in step S306), loop processing is performed on each record stored as the row processing data 115 (steps S313 to S316). In the loop processing, the coordinate matching determination unit 116 compares the reference coordinate information calculated from the newly performed handwriting input with the reference coordinate information recorded in the row processing data 115 (step S314). Then, the coordinate matching determination unit 116 determines whether or not a predetermined condition as to whether or not it is an object of repeated processing is satisfied (step S315). The predetermined conditions are the same as those described in FIG.

  Returning to FIG. 32, when it is determined that the predetermined condition as to whether or not to be the object of repeated processing is not satisfied (NO in step S315), the process proceeds to the next recording, and the processing is performed for all the recordings. The loop process is terminated (step S316). Then, the processing is continued from extraction of the search processing list (step S307).

  When it is determined that a predetermined condition as to whether or not it is a target of repeated processing is satisfied (YES in step S315), the automatic process application unit 117 selects an attribute list corresponding to a newly performed text string of handwritten input. And the attributes of the row processing data 115 are compared (step S317). Then, the automatic process application unit 117 determines whether there is a matching attribute (step S318). If it is determined that there is no matching attribute (NO in step S318), the process proceeds to the next recording, and if all the recordings have been processed, the loop process is terminated (step S316). Then, the processing is continued from extraction of the search processing list (step S307).

  If it is determined that there is a matching attribute (YES in step S318), the automatic process application unit 117 performs a search process recorded in the line processing data 115 by using the text obtained by the handwritten character recognition unit 107 from a new handwritten input. It is applied to the character string and executed (step S319). Next, the automatic placement unit 118 calculates new coordinate values from the new handwritten input reference coordinate information, the reference coordinate information of the line processing data 115, and the placement coordinate information (step S320), and automatically retrieves the search result data as the coordinate values. Arrange (step S321). Thereafter, the processing is repeated from acceptance of handwriting input (step S301).

  In addition, although the case where it applied to the 1st process example in which one search process is performed in one element of enumeration was demonstrated, it applies to the 2nd process example in which a plurality of search processes are permitted. You may be made to do.

<Summary>
As described above, according to the present embodiment, it is possible to reduce a work burden for associating contents with character string information recognized from a handwritten input locus.

  In the above, it demonstrated by preferred embodiment. While specific embodiments have been illustrated and described herein, it will be apparent that various modifications and changes may be made thereto without departing from the broad spirit and scope as defined in the claims. . That is, it should not be construed as being limited by the details of the specific examples and the accompanying drawings.

Regarding the above description, the following items are further disclosed.
(Appendix 1)
Recognize the first character string information from the trajectory entered by handwriting,
Accepting an operation for associating the content with the recognized first character string information from the plurality of types of content,
Recognize the second character string information from the trajectory entered by handwriting,
Identify each position where the recognized first and second character string information is input by handwriting,
When the specified positional relationship of each handwritten input satisfies a preset condition, the recognized second character string information and the type of content associated with the first character string information Based on, associate the identified content with the second string information,
An information processing program for causing a computer to execute processing.
(Appendix 2)
Accepting an operation for associating the content with the recognized second character string information from a plurality of types of content when the positional relationship of the specified handwritten input does not satisfy a preset condition;
The information processing program according to appendix 1, which causes a computer to execute processing.
(Appendix 3)
The process of accepting an operation for associating content with the first and second character string information accepts an operation for associating a plurality of different types of content.
The information processing program according to supplementary note 2, characterized by:
(Appendix 4)
The process of associating the identified content with the second character string information is performed for each positional relationship between the first and second character string information.
The information processing program according to any one of supplementary notes 1 to 3, characterized in that:
(Appendix 5)
The content associated with the second character string information is automatically arranged at a position calculated based on the positional relationship of the content associated with the first character string information.
The information processing program according to any one of appendices 1 to 4, which causes a computer to execute processing.
(Appendix 6)
Displaying the position where handwriting input corresponding to the second character string information should be performed at the position calculated based on the positional relationship of the first character string information;
6. The information processing program according to any one of appendices 1 to 5, which causes a computer to execute processing.
(Appendix 7)
Recognize the first character string information from the trajectory entered by handwriting,
Accepting an operation for associating the content with the recognized first character string information from the plurality of types of content,
Recognize the second character string information from the trajectory entered by handwriting,
Identify each position where the recognized first and second character string information is input by handwriting,
When the specified positional relationship of each handwritten input satisfies a preset condition, the recognized second character string information and the type of content associated with the first character string information Based on, associate the identified content with the second string information,
An information processing method, wherein a computer executes a process.
(Appendix 8)
Accepting an operation for associating the content with the recognized second character string information from a plurality of types of content when the positional relationship of the specified handwritten input does not satisfy a preset condition;
The information processing method according to appendix 7, wherein the processing is executed by a computer.
(Appendix 9)
The process of accepting an operation for associating content with the first and second character string information accepts an operation for associating a plurality of different types of content.
The information processing method according to appendix 8, characterized by:
(Appendix 10)
The process of associating the identified content with the second character string information is performed for each positional relationship between the first and second character string information.
The information processing method according to any one of appendices 7 to 9, characterized in that:
(Appendix 11)
The content associated with the second character string information is automatically arranged at a position calculated based on the positional relationship of the content associated with the first character string information.
The information processing method according to any one of appendices 7 to 10, wherein the processing is executed by a computer.
(Appendix 12)
Displaying the position where handwriting input corresponding to the second character string information should be performed at the position calculated based on the positional relationship of the first character string information;
The information processing method according to any one of appendices 7 to 11, wherein the processing is executed by a computer.
(Appendix 13)
A first recognition unit for recognizing first character string information from a handwritten input locus;
A reception unit that receives an operation of associating the content with the recognized first character string information from the plurality of types of content;
A second recognition unit for recognizing second character string information from a locus input by handwriting;
A identifying unit that identifies each position where the recognized first and second character string information is input by handwriting;
When the specified positional relationship of each handwritten input satisfies a preset condition, the recognized second character string information and the type of content associated with the first character string information Based on the, the associating unit for associating the identified content with the second character string information,
An information processing apparatus comprising:
(Appendix 14)
A first operation for accepting an operation for associating the content with the recognized second character string information from a plurality of types of content when each of the identified handwritten input positional relationships does not satisfy a preset condition; The second reception department,
The information processing apparatus according to attachment 13, further comprising:
(Appendix 15)
The accepting unit and the second accepting unit accept operations for associating a plurality of different types of content.
The information processing apparatus according to appendix 14, characterized by:
(Appendix 16)
The association unit performs association for each positional relationship between the first and second character string information.
The information processing apparatus according to any one of supplementary notes 13 to 15, characterized in that:
(Appendix 17)
An arrangement unit that automatically arranges the content associated with the second character string information at a position calculated based on the positional relationship of the content associated with the first character string information;
The information processing apparatus according to any one of supplementary notes 13 to 16, further comprising:
(Appendix 18)
A guide display unit for performing display indicating a position where handwriting input corresponding to the second character string information is to be performed at a position calculated based on the positional relationship of the first character string information;
The information processing apparatus according to any one of supplementary notes 13 to 17, further comprising:

  The handwritten character recognition unit 107 is an example of a “first recognition unit” and a “second recognition unit”. The input receiving unit 101 is an example of a “receiving unit”. The reference coordinate calculation unit 106 is an example of a “specification unit”. The row processing determination unit 114 is an example of “association unit”.

1, 11, 12 Information processing apparatus 101 Input reception unit 102 Display unit 103 Document management unit 104 Document data 105 Written set selection unit 106 Reference coordinate calculation unit 107 Handwritten character recognition unit 108 Attribute determination unit 109 Word attribute table 110 Search processing determination unit DESCRIPTION OF SYMBOLS 111 Search processing table 112 Process selection execution part 113 Arrangement part 114 Line process determination part 115 Line process data 116 Coordinate matching determination part 117 Automatic process application part 118 Automatic arrangement part

Claims (8)

Recognize the first character string information from the trajectory entered by handwriting,
Accepting an operation for associating the content with the recognized first character string information from the plurality of types of content,
Recognize the second character string information from the trajectory entered by handwriting,
Identify each position where the recognized first and second character string information is input by handwriting,
When the specified positional relationship of each handwritten input satisfies a preset condition, the recognized second character string information and the type of content associated with the first character string information Based on, associate the identified content with the second string information,
An information processing program for causing a computer to execute processing. Accepting an operation for associating the content with the recognized second character string information from a plurality of types of content when the positional relationship of the specified handwritten input does not satisfy a preset condition;
The information processing program according to claim 1, which causes a computer to execute processing. The process of accepting an operation for associating content with the first and second character string information accepts an operation for associating a plurality of different types of content.
The information processing program according to claim 2. The process of associating the identified content with the second character string information is performed for each positional relationship between the first and second character string information.
The information processing program according to any one of claims 1 to 3. The content associated with the second character string information is automatically arranged at a position calculated based on the positional relationship of the content associated with the first character string information.
The information processing program according to any one of claims 1 to 4, wherein the information processing program causes a computer to execute processing. Displaying the position where handwriting input corresponding to the second character string information should be performed at the position calculated based on the positional relationship of the first character string information;
The information processing program according to claim 1, wherein the processing is executed by a computer. Recognize the first character string information from the trajectory entered by handwriting,
Accepting an operation for associating the content with the recognized first character string information from the plurality of types of content,
Recognize the second character string information from the trajectory entered by handwriting,
Identify each position where the recognized first and second character string information is input by handwriting,
When the specified positional relationship of each handwritten input satisfies a preset condition, the recognized second character string information and the type of content associated with the first character string information Based on, associate the identified content with the second string information,
An information processing method, wherein a computer executes a process. A first recognition unit for recognizing first character string information from a handwritten input locus;
A reception unit that receives an operation of associating the content with the recognized first character string information from the plurality of types of content;
A second recognition unit for recognizing second character string information from a locus input by handwriting;
A identifying unit that identifies each position where the recognized first and second character string information is input by handwriting;
When the specified positional relationship of each handwritten input satisfies a preset condition, the recognized second character string information and the type of content associated with the first character string information Based on the, the associating unit for associating the identified content with the second character string information,
An information processing apparatus comprising:

Images