JP6187236B2 - Data identification program, data identification method, and information processing apparatus - Google Patents

Description

  The present invention relates to a data specifying program, a data specifying method, and an information processing apparatus.

  Conventionally, there is a service called account aggregation that aggregates and displays information on a plurality of sites on one screen. According to account aggregation, for example, information on a plurality of accounts of different financial institutions held by depositors in Internet banking or the like can be aggregated and displayed on a list screen.

  As related prior art, for example, a server cuts out an image of a rectangular area from an image, recognizes the text by OCR processing, and extracts the text having the highest degree of coincidence with the text recognized from the source of the html file. There is a technique for transmitting to a client terminal. In addition, there is a web page information extraction system for extracting a part that a user needs.

  Further, there is a technique for extracting an extracted data object from an html document acquired from a plurality of sites based on a cut-out rule, vocabulary information, and an inference operation, using a tag of the html document. Technology that captures and stores time-varying information by capturing a screen containing time-varying information as image data and using character recognition for the image data, and displays it in a specified area on the screen There is.

JP 2011-123740 A JP 2003-308275 A JP 2004-062446 A JP 2007-206908 A

  However, according to the prior art, if there is a plurality of data having the same content as the character string specified by the user in the html data of the site, the position of the information acquired from the site in the html data of the site cannot be specified. There is a case.

  In one aspect, an object of the present invention is to provide a data specifying program, a data specifying method, and an information processing apparatus that can accurately specify the position of information acquired from a site in screen information of the site.

  According to one aspect of the present invention, text data having the same content as text data obtained from image data in a range in which selection has been received on image data of a screen of a site is searched from screen information of the screen, The searched text data in the screen information is changed to different text data, and is obtained from the image data in the same range as the selection received on the screen image data based on the screen information after the change. A data specifying program, a data specifying method, and an information processing for specifying text data corresponding to a range in which the selection is accepted from screen information of the screen by determining whether text data matches the different text data A device is proposed.

  According to one aspect of the present invention, it is possible to accurately specify the position of information acquired from a site in screen information of the site.

FIG. 1 is an explanatory diagram of an example of the data specifying method according to the embodiment. FIG. 2 is an explanatory diagram illustrating a system configuration example of the system 200. FIG. 3 is a block diagram illustrating a hardware configuration example of the information processing apparatus 101. FIG. 4 is a block diagram illustrating a hardware configuration example of the server 201. FIG. 5 is an explanatory diagram showing an example of the contents stored in the account aggregation information DB 220. FIG. 6 is an explanatory diagram showing an example of the contents stored in the site-specific purpose data attribute DB 230. FIG. 7 is an explanatory diagram showing an example of the contents stored in the list information DB 240. FIG. 8 is an explanatory diagram illustrating a screen example of a list setting screen. FIG. 9 is an explanatory diagram illustrating a screen example of a region initial setting screen. FIG. 10 is an explanatory diagram illustrating a screen example of a list screen. FIG. 11 is a block diagram illustrating a functional configuration example of the information processing apparatus 101. FIG. 12 is an explanatory diagram illustrating a screen example of the area resetting screen. FIG. 13 is a flowchart illustrating an example of an information provision processing procedure of the information processing apparatus 101. FIG. 14 is a flowchart (part 1) illustrating an example of a specific processing procedure of the new registration processing. FIG. 15 is a flowchart (part 2) illustrating an example of a specific processing procedure of the new registration processing. FIG. 16 is a flowchart illustrating an example of a specific processing procedure of the list display processing. FIG. 17 is a flowchart illustrating an example of a specific processing procedure of the target data setting process. FIG. 18 is a flowchart illustrating an example of a specific processing procedure of the area reset screen display process.

  Exemplary embodiments of a data specifying program, a data specifying method, and an information processing apparatus according to the present invention will be described below in detail with reference to the drawings.

(One example of data identification method)
FIG. 1 is an explanatory diagram of an example of the data specifying method according to the embodiment. In FIG. 1, an information processing apparatus 101 is a computer having a function of collecting and outputting information on a plurality of sites on one screen. A site is a page or a set of pages, for example, a website.

  The page is information disclosed on the network, for example, a web page. The page includes an electronic document (html data, xhtml data) or image data described by html (HyperText Markup Language) or xhtml (Extensible HyperText Markup Language).

  Here, it is assumed that information on the bank site S1, the securities company site S2, and the pension site S3 is collected and output on a list screen. In this case, the user sets, for each site of the bank site S1, the securities company site S2, and the pension site S3, which part of which page of each site is to be acquired.

  As an example, it is assumed that the payment amount of the pension of the pension site S3 is acquired. In this case, for example, in the image data 110 on the screen of the pension site S3, by selecting a range including information to be displayed on the list screen (hereinafter referred to as “region T1”) by a user operation input, the list screen is displayed. The text data of the character string “18776540” to be displayed can be acquired.

  Further, it is conceivable to specify the position of the character string “18776540” in the html data 120 by specifying the tag of the html element including the text data of the character string “18776540” from the html data 120 on the screen of the pension site S3. . A tag is additional information described in a form embedded in a document using a predetermined notation.

  In html data (or xhtml data), by embedding half-width alphanumeric characters enclosed in “<” and “>” as tags in the original document, the document structure, format, character decoration, etc. are given to the browser. You can instruct and embed images and links to other documents. The html element is an element constituting html data, and includes, for example, a start tag, contents, and an end tag.

  However, the same character string as the character string “18776540” specified by the user may exist by chance in the screen of the pension site S3. In this case, the tag of the html element including the information to be extracted from the html data 120 may not be uniquely specified only by the text data of the character string “18776540” designated by the user.

  Therefore, in this embodiment, a data specifying method for accurately specifying the position of text data corresponding to a character string designated by the user in the html data of the site will be described. Hereinafter, an embodiment of the data specifying process of the information processing apparatus 101 will be described.

  (1) The information processing apparatus 101 acquires screen information of the site S with reference to the identification information of the site S recorded in advance. Here, the site S is one of a plurality of sites that collect and display information on one screen. The identification information of the site S is information for identifying the site S, and is, for example, a URL (Uniform Resource Locator) of the site S.

  More specifically, the identification information of the site S is, for example, a URL of a page including the information of the site S displayed on the list screen. The list screen is a screen that aggregates and displays information on a plurality of sites. The screen information of the site S is information for displaying a page including the information of the site S, and is, for example, html data or xhtml data of a page including the information of the site S.

  In the following description, the information on the site S displayed on the list screen may be referred to as “target data”. Further, a page including the target data of the site S may be referred to as “target page”. Further, “html data” will be described as an example of the screen information of the site S.

  Here, as an example, the target page is “pension site S3”, and the target data is “numeric string indicating the amount of pension payment”. In this case, the information processing apparatus 101 acquires the html data 120 of the pension site S3 by specifying the URL of the pension site S3 and accessing the pension site S3.

  (2) The information processing apparatus 101 searches the html data on the screen of the site S for text data having the same content as the text data obtained from the image data of the region T set on the image data on the screen of the site S. Here, the text data obtained from the image data in the region T is the text data of the target data.

  Specifically, for example, first, the information processing apparatus 101 extracts the image data 111 of the region T1 set on the image data 110 from the image data 110 of the pension site S3. Next, the information processing apparatus 101 performs character recognition processing on the image data 111 of the extracted region T1. Here, the character recognition process is a process of identifying a character from image data based on the shape of the character and converting it into character data that can be handled on a computer.

  The character recognition process is, for example, an OCR (Optical Character Recognition) process. According to the character recognition process, characters or character strings in the region T can be obtained as text data. In the example of FIG. 1, text data of a numeric string “18776540” indicating payment (yen) is obtained as text data of the target data. Then, the information processing apparatus 101 searches the html data 120 of the pension site S3 for text data having the same contents as the text data of the target data “18776540” obtained by the character recognition process.

  (3) The information processing apparatus 101 changes the searched text data in the html data on the screen of the site S to different text data. In the example of FIG. 1, text data 121 indicating “payment amount of annuity” and text data 122 indicating a part of “telephone number” are text data having the same content as text data “18776540” of target data “18776540”. Is searched.

  In this case, the information processing apparatus 101 changes any text data of the text data 121 and 122 to different text data. In the example of FIG. 1, the text data 121 in the html data 120 is changed to text data 123 indicating a predetermined character string “here?”.

  Note that, for example, the information processing apparatus 101 may execute the process (3) after the text data search for the entire html data is completed in (2). Further, for example, in (2) above, the information processing apparatus 101 searches for text data from the beginning or end of html data, and whenever the text data having the same content is searched, the processing of (3) above is performed. May be executed.

  (4) The information processing apparatus 101 determines whether the text data obtained from the image data of the region T on the image data of the screen of the site S based on the html data of the site S after the change matches the changed different text data. Determine whether. Specifically, for example, first, the information processing apparatus 101 captures the pension site S3 based on the changed html data 120 and acquires the image data 130 of the pension site S3. Note that capture means saving a screen image displayed on the display as image data.

  Then, the information processing apparatus 101 extracts the image data 131 of the region T1 on the image data 130 (the same range as the range selected on the image data 110) from the acquired image data 130 of the pension site S3. Next, the information processing apparatus 101 performs character recognition processing on the image data 131 of the extracted region T1. Then, the information processing apparatus 101 determines whether the text data obtained by the character recognition process matches the text data 123.

  (5) The information processing apparatus 101 identifies text data corresponding to the region T from the html data on the screen of the site S based on the determined determination result. Here, when the text data matches, it indicates that the location of the target data is the part of the html data that has been changed to different text data.

  For this reason, when the text data matches, the information processing apparatus 101 specifies the text data changed to different text data in the html data as the text data corresponding to the region T. In the example of FIG. 1, the text data obtained by the character recognition process matches the text data 123.

  In this case, the information processing apparatus 101 specifies the text data 121 changed to the different text data 123 among the text data 121 and 122 in the html data 120 as the text data corresponding to the region T. If the text data does not match, the text data to be changed to different text data is switched (for example, text data 122), and the series of processes (3) to (5) is repeated.

  However, in the example described above, the text data searched in (2) is two text data 121 and 122. For this reason, when the text data obtained by the character recognition process does not match the text data 123, the information processing apparatus 101 can change the text data 122 without repeating the processes (3) to (5), for example. The text data corresponding to the region T may be specified.

  Thus, according to the information processing apparatus 101, text data having the same content as the text data obtained from the html data 120 of the pension site S3 and the image data 111 of the area T1 set on the image data 110 of the pension site S3. Can be searched. Thereby, text data having the same content as the target data can be searched from the html data 120 of the pension site S3.

  Further, according to the information processing apparatus 101, when a plurality of text data 121 and 122 are searched, any text data (for example, text data) of the plurality of text data 121 and 122 in the html data 120 of the pension site S3. 121) can be changed to different text data. Further, according to the information processing apparatus 101, the text data obtained from the image data 131 of the region T1 on the image data 130 of the pension site S3 based on the html data 120 of the changed pension site S3 is different from the changed different text data. It can be determined whether or not they match.

  Further, according to the information processing apparatus 101, when the text data matches the changed different text data, the text data 121 changed to the different text data among the html data 120 of the pension site S3 is specified as the text data corresponding to the region T1. can do. Thereby, even when there are a plurality of text data having the same contents as the target data in the html data 120 of the pension site S3, the position of the target data in the html data 120 of the pension site S3 can be accurately specified. .

(System configuration example of system 200)
Next, a system configuration example of the system 200 according to the embodiment will be described.

  FIG. 2 is an explanatory diagram illustrating a system configuration example of the system 200. In FIG. 2, a system 200 includes an information processing apparatus 101 and a server 201. In the system 200, the information processing apparatus 101 and the server 201 are connected to each other via a wired or wireless network 210 so that they can communicate with each other. The network 210 is, for example, the Internet, a LAN (Local Area Network), a WAN (Wide Area Network), or the like.

  Here, the information processing apparatus 101 includes an account aggregation information DB (database) 220, a site-specific objective data attribute DB 230, and a list information DB 240. Specifically, for example, the information processing apparatus 101 is a PC (Personal Computer) installed with a browser, a notebook PC, a tablet PC, a smartphone, a mobile phone, or the like.

  The account aggregation information DB 220, the site-specific purpose data attribute DB 230, and the list information DB 240 will be described later with reference to FIGS.

  The server 201 is a computer that transmits screen information of the site S including html data and images in response to a request from the information processing apparatus 101. The information processing apparatus 101 can display the screen of the site S based on the screen information of the site S from the server 201. Specifically, for example, the server 201 is a Web server.

(Hardware configuration example of information processing apparatus 101)
FIG. 3 is a block diagram illustrating a hardware configuration example of the information processing apparatus 101. In FIG. 3, an information processing apparatus 101 includes a CPU (Central Processing Unit) 301, a memory 302, a disk drive 303, a disk 304, a display 305, an I / F (Interface) 306, a keyboard 307, a mouse. 308, a scanner 309, and a printer 310. Each component is connected by a bus 300.

  Here, the CPU 301 governs overall control of the information processing apparatus 101. The memory 302 includes, for example, a ROM (Read Only Memory), a RAM (Random Access Memory), and a flash ROM. Specifically, for example, a flash ROM or ROM stores various programs, and a RAM is used as a work area for the CPU 301. The program stored in the memory 302 is loaded into the CPU 301 to cause the CPU 301 to execute the coded process.

  The disk drive 303 controls reading / writing of data with respect to the disk 304 according to the control of the CPU 301. The disk 304 stores data written under the control of the disk drive 303. Examples of the disk 304 include a magnetic disk and an optical disk.

  A display 305 displays data such as a document, an image, and function information as well as a cursor, an icon, or a tool box. As the display 305, for example, a CRT, a TFT liquid crystal display, a plasma display, or the like can be adopted.

  The I / F 306 is connected to the network 210 via a communication line, and is connected to another computer (for example, the server 201) via the network 210. The I / F 306 controls an internal interface with the network 210 and controls data input / output from other computers. For example, a modem or a LAN adapter may be employed as the I / F 306.

  The keyboard 307 includes keys for inputting characters, numbers, various instructions, and the like, and inputs data. The keyboard 307 may be a touch panel type input pad or a numeric keypad. The mouse 308 performs cursor movement, range selection, window movement, size change, and the like.

  The scanner 309 optically reads an image and takes in image data into the information processing apparatus 101. The scanner 309 may have an OCR function. The printer 310 prints image data and document data. As the printer 310, for example, a laser printer or an ink jet printer can be employed. Note that the information processing apparatus 101 may not include the scanner 309, the printer 310, or the like among the above-described components.

(Hardware configuration example of server 201)
FIG. 4 is a block diagram illustrating a hardware configuration example of the server 201. In FIG. 4, the server 201 includes a CPU 401, a memory 402, an I / F 403, a disk drive 404, and a disk 405. Each component is connected by a bus 400.

  Here, the CPU 401 governs overall control of the server 201. The memory 402 includes, for example, a ROM, a RAM, a flash ROM, and the like. Specifically, for example, a flash ROM or ROM stores various programs, and the RAM is used as a work area of the CPU 401. The program stored in the memory 402 is loaded on the CPU 401 to cause the CPU 401 to execute the coded process.

  The I / F 403 is connected to the network 210 via a communication line, and is connected to another computer (for example, the information processing apparatus 101 shown in FIG. 2) via the network 210. The I / F 403 controls an internal interface with the network 210 and controls input / output of data from other computers. For example, a modem or a LAN adapter may be employed as the I / F 403.

  The disk drive 404 controls reading / writing of data with respect to the disk 405 according to the control of the CPU 401. The disk 405 stores data written under the control of the disk drive 404. Examples of the disk 405 include a magnetic disk and an optical disk. Note that the server 201 may include, for example, a keyboard, a mouse, a display, and the like in addition to the components described above.

(Contents stored in the account aggregation information DB 220)
Next, the contents stored in the account aggregation information DB 220 included in the information processing apparatus 101 will be described. The account aggregation information DB 220 is realized by a storage device such as the memory 302 and the disk 304 of the information processing apparatus 101 illustrated in FIG.

  FIG. 5 is an explanatory diagram showing an example of the contents stored in the account aggregation information DB 220. In FIG. 5, the account aggregation information DB 220 has fields of ID, PW, login URL, and data URL. By setting information in each field, account aggregation information (account aggregation information 500-1 to 500-7) is stored as a record.

  Here, ID (identification) is an identifier for identifying a user of the site S. PW (password) is a user password for logging in to the site S. The login URL is a URL for displaying a Web page (so-called login screen) for logging into the site S.

  The data URL is a URL for displaying a Web page including information on the site S to be displayed on the list screen. The list screen is a screen that aggregates and displays information on a plurality of sites S. Here, the data URL is a URL of a CGI (Common Gateway Interface) script.

  For example, the account aggregation information 500-1 indicates ID “12345”, PW “P111111”, login URL “A.html”, and data URL “A / 12345.cgi”.

(Storage contents of site-specific purpose data attribute DB 230)
Next, the contents stored in the site-specific purpose data attribute DB 230 of the information processing apparatus 101 will be described. The site-specific purpose data attribute DB 230 is realized by a storage device such as the memory 302 and the disk 304 of the information processing apparatus 101, for example.

  FIG. 6 is an explanatory diagram showing an example of the contents stored in the site-specific purpose data attribute DB 230. In FIG. 6, the site-specific purpose data attribute DB 230 includes fields for a data URL, a data specific html attribute, and a data attribute. By setting information in each field, site-specific target data attribute information (for example, site-specific target data attribute information 600-1 to 600-5) is stored as a record.

  Here, the data URL is a URL for displaying the target page of the site S. The data specifying html attribute is information for specifying a tag of an html element including target data. The data attribute is an attribute of the target data. Examples of data attributes include numerical values, kanji, kana, kana, and alphabet. For example, the site-specific purpose data attribute information 600-1 indicates the data URL “A / 12345.cgi”, the data specific html attribute “fourth of all td twelve”, and the data attribute “numerical value”.

(Storage contents of list information DB 240)
Next, the contents stored in the list information DB 240 included in the information processing apparatus 101 will be described. The list information DB 240 is realized by a storage device such as the memory 302 and the disk 304 of the information processing apparatus 101, for example.

  FIG. 7 is an explanatory diagram showing an example of the contents stored in the list information DB 240. In FIG. 7, the list information DB 240 has fields of data URL and list position, and sets information in each field to store list information (for example, list information 700-1 to 700-5) as a record. .

  Here, the data URL is a URL for displaying the target page of the site S. The list position is information indicating a position where the target data of the site S is displayed on the list screen. Here, the list position indicates the number of a box (for example, boxes B1 to B3 shown in FIG. 8) in the list screen. For example, the list information 700-1 indicates the data URL “A / 12345.cgi” and the list position “2”.

(Example of list setting screen)
Next, a screen example of the list setting screen displayed on the display 305 of the information processing apparatus 101 will be described. The list setting screen is a screen for setting the screen configuration of the list screen displaying the target data of the plurality of sites S and the posting contents.

  FIG. 8 is an explanatory diagram illustrating a screen example of a list setting screen. In FIG. 8, a list setting screen 800 is a screen for setting item names and display positions of target data to be displayed on the list screen. In the list setting screen 800, the item name of the target data to be displayed on the list screen can be set by the user's operation input using the keyboard 307 and the mouse 308 shown in FIG.

  In the example of FIG. 8, the item names “number of months of pension participation”, “pension receipt (expected) amount” and “bank balance of X bank” of the target data displayed on the list screen are set. The “months of pension participation” and “pension receipt (expected) amount” are information on a certain pension site. Further, “deposit balance of bank X” is information on a certain bank site.

  In the list setting screen 800, a box for displaying target data can be set by a user operation input. In the example of FIG. 8, the box B1 that displays the target data of the item name “pension enrollment months”, the box B2 that displays the target data of the item name “pension receipt (expected amount)”, and the item name “bank X deposit balance”. The box B3 for displaying the target data “is set.

(Example of initial area setting screen)
Next, a screen example of the area initial setting screen displayed on the display 305 of the information processing apparatus 101 will be described. The area initial setting screen is a screen for setting an area T including target data on the screen of the target page.

  FIG. 9 is an explanatory diagram illustrating a screen example of a region initial setting screen. In FIG. 9, a region initial setting screen 900 is a screen for setting a region T including target data on the screen of the welfare pension information page of the pension site. In the area initial setting screen 900, image data 910 of the screen of the welfare pension information page of the pension site is displayed.

  In the region initial setting screen 900, by accepting selection of a range including the target data on the image data 910 as an operation input for specifying the region by the user, the region T including the target data on the screen of the welfare annuity information page is set. Can do. In the example of FIG. 9, regions T <b> 1 and T <b> 2 are set as regions including target data on the welfare pension information page screen.

  Here, area | region T1 is an area | region containing the numerical sequence which shows the enrollment period [month] of an employee pension information page. The region T2 is a region including a numeric string indicating the annual amount (expected) [yen] of the employee pension information page. On the region initial setting screen 900, when the setting completion button 920 is clicked (pressed) by a user operation input, the setting of the region T is completed.

(Screen example of list screen)
Next, a screen example of a list screen displayed on the display 305 of the information processing apparatus 101 will be described. The list screen is a screen that aggregates and displays the target data of a plurality of sites S.

  FIG. 10 is an explanatory diagram illustrating a screen example of a list screen. In FIG. 10, a list screen 1000 is a screen that collectively displays the purpose data of the welfare pension information page of the pension site and the purpose data of the account information page of the bank site. Specifically, on the list screen 1000, the number of months of pension participation on the welfare pension information page of the pension site is displayed in box B1, and the amount of pension receipt (expected) is displayed in box B2.

  On the list screen 1000, the deposit balance on the account information page of the bank site is displayed in a box B3. According to the list screen 1000, the user can confirm at a glance the number of months of pension participation, the pension receipt (expected) amount on the welfare pension information page of the pension site, and the deposit balance on the account information page of the bank site.

(Functional configuration example of the information processing apparatus 101)
FIG. 11 is a block diagram illustrating a functional configuration example of the information processing apparatus 101. In FIG. 11, the information processing apparatus 101 includes a reception unit 1101, an acquisition unit 1102, a registration unit 1103, a display control unit 1104, a recognition unit 1105, a search unit 1106, a change unit 1107, and a determination unit 1108. The specifying unit 1109 is included. The receiving unit 1101 to the specifying unit 1109 are functions as control units. Specifically, for example, by causing the CPU 301 to execute a program stored in a storage device such as the memory 302 and the disk 304 illustrated in FIG. Alternatively, the function is realized by the I / F 306. The processing result of each functional unit is stored in a storage device such as the memory 302 and the disk 304, for example.

<When a new registration request is accepted>
First, processing contents of each functional unit when a new registration request is received will be described. The new registration request is a request for newly registering target data to be displayed on the list screen.

  The accepting unit 1101 accepts a new registration request. Specifically, for example, the reception unit 1101 receives a new registration request by a user operation input using the keyboard 307 or the mouse 308. The receiving unit 1101 may receive a new registration request by receiving a new registration request from an external computer.

  The acquisition unit 1102 acquires the ID, PW, login URL, and data URL of the site S in response to receiving a new registration request. Specifically, for example, the acquisition unit 1102 acquires the ID, PW, login URL, and data URL of the site S by a user operation input. At this time, the acquisition unit 1102 may acquire the data URL of the target page by causing a screen transition to the target page by a user operation input.

  Note that the ID, PW, login URL, and data URL of the site S may be included in the new registration request. In this case, the acquisition unit 1102 acquires the ID, PW, login URL, and data URL of the site S from the accepted new registration request.

  The registration unit 1103 registers the acquired ID, PW, login URL, and data URL of the site S in the account aggregation information DB 220. Specifically, for example, the registration unit 1103 sets the acquired site S ID, PW, login URL, and data URL in each field of the account aggregation information DB 220.

  As a result, new account aggregation information is newly registered in the account aggregation information DB 220.

  The acquisition unit 1102 acquires html data of the screen of the target page. Specifically, for example, first, the acquisition unit 1102 accesses the login screen of the site S using the acquired login URL of the site S. Then, the acquisition unit 1102 logs into the site S using the acquired ID and PW of the site S. Next, the acquisition unit 1102 acquires the html data of the target page of the site S using the acquired data URL of the site S.

  The display control unit 1104 outputs image data of the screen of the target page based on the acquired html data of the screen of the target page. Specifically, for example, first, the display control unit 1104 acquires the image data of the screen of the target page by capturing the screen of the target page based on the acquired html data. Then, the display control unit 1104 displays an area initial setting screen (for example, the area initial setting screen 900 shown in FIG. 9) including the acquired image data of the screen of the target page on the display 305.

  The accepting unit 1101 accepts selection of an area T including the target data on the image data of the output target page screen. Specifically, for example, the accepting unit 1101 selects an area T (for example, areas T1 and T2) including target data on the image data 910 (see FIG. 9) by a user operation input on the area initial setting screen 900. Accept.

  The recognition unit 1105 performs character recognition processing on the image data of the selected region T. Specifically, for example, the recognition unit 1105 first extracts the image data of the selected region T from the image data of the target page. Then, the recognition unit 1105 performs OCR processing on the extracted image data of the region T. Thereby, the text data of the target data can be acquired.

  In addition, the recognition unit 1105 acquires the position information of the selected region T. Here, the position information of the area T is information indicating the position of the area T in the image data of the target page. For example, when the region T is a rectangle, the position information of the region T is the coordinates (x coordinate, y coordinate) of two vertices of the diagonal of the rectangle. When the region T is a circle, the position information of the region T is the coordinates and radius of the center of the circle. Specifically, for example, when the region T is rectangular, the recognition unit 1105 acquires the upper left coordinates (x coordinate, y coordinate) and the lower right coordinates (x coordinate, y coordinate) of the selected region T. .

  The retrieval unit 1106 retrieves text data having the same content as the recognized character or character string from the obtained html data of the target page. That is, the search unit 1106 searches the html data of the target page for text data having the same content as the text data of the target data obtained by the OCR process.

  The changing unit 1107 changes the searched text data in the html data of the target page to different text data. Specifically, for example, when a plurality of text data are searched, the changing unit 1107 converts any text data of the plurality of text data in the html data of the target page into a predetermined character string (or character). Change to text data indicating. The predetermined character string can be arbitrarily set. For example, the predetermined character string is set to a character string that hardly appears in the html data of the page.

  The recognition unit 1105 performs character recognition processing of the image data in the region T on the image data of the target page based on the html data of the target page after the change. Specifically, for example, first, the recognition unit 1105 captures the target page by developing the changed html data in the memory 302, and acquires the image data of the target page. Next, the recognition unit 1105 extracts image data of the region T from the acquired image data of the target page based on the acquired position information of the region T. Then, the recognition unit 1105 performs character recognition processing on the image data of the extracted region T.

  The determination unit 1108 determines whether or not the text data obtained from the image data in the region T on the target page image data based on the changed target page html data matches the changed different text data. Specifically, for example, the determination unit 1108 determines whether the text data obtained by the character recognition process matches text data indicating a predetermined character string.

  The specifying unit 1109 specifies text data corresponding to the region T from the html data of the target page. Specifically, for example, when one text data is searched by the search unit 1106, the specifying unit 1109 sets the searched text data among the html data of the target page as text data corresponding to the region T. Identify.

  On the other hand, when a plurality of text data are searched, the specifying unit 1109 specifies the text data corresponding to the region T from the plurality of searched text data based on the determined determination result. For example, when the text data matches, the specifying unit 1109 specifies text data changed to different text data among the html data of the target page as text data corresponding to the region T.

  Further, the specifying unit 1109 specifies information related to the tag in the html data of the target page based on the text data corresponding to the specified region T. Here, the information regarding the tag is information for specifying the tag of the html element including the target data among the html data of the target page.

  Specifically, for example, first, the specifying unit 1109 searches for html elements including text data corresponding to the region T from html data of the target page. Then, the specifying unit 1109 specifies the data specifying html attribute of the tag of the searched html element in the html data of the target page as information related to the tag.

  The data specifying html attribute is information for specifying the position of the tag of the html element including the text data of the target data in the html data of the target page. The data specifying html attribute indicates, for example, the type of tag and the tag number from the top of all tags of the same type in the html data.

  Further, the specifying unit 1109 specifies the data attribute of the text data corresponding to the specified region T. Specifically, for example, the recognizing unit 1105 analyzes the text data corresponding to the region T to identify the data attribute (for example, numeric value, kanji, kana, kana, alphabet) of the text data.

  The receiving unit 1101 has a function of receiving a list position of target data. Specifically, for example, the accepting unit 1101 accepts a list position of the target data (for example, a box number on the list setting screen 800 shown in FIG. 8) by a user operation input.

  The registration unit 1103 records information related to the tag in the html data of the specified target page in association with the data URL of the target page. In addition, the registration unit 1103 records the data attribute of the text data corresponding to the specified region T in association with the data URL of the target page.

  Specifically, for example, the registration unit 1103 sets a data URL, a data specific html attribute, and a data attribute in each field of the site-specific purpose data attribute DB 230. As a result, new site-specific purpose data attribute information is newly registered in the site-specific purpose data attribute DB 230.

  Also, the registration unit 1103 records the list position of the received target data in association with the data URL of the target page. Specifically, for example, the registration unit 1103 sets a data URL and a list position in each field of the list information DB 240. As a result, new list information is newly registered in the list information DB 240.

  In the above description, a case has been described in which a predetermined character string for changing text data is set. However, the present invention is not limited to this. For example, there is a case where a predetermined character string set in advance is accidentally included in the html data of the target page. For this reason, for example, the changing unit 1107 searches text data indicating a predetermined character string from html data of the target page before the change, and when the text data is searched, the predetermined character string is changed to a different character string. You may decide to set it again.

<When a list display request is accepted>
Next, processing contents of each functional unit when a list display request is received will be described. The list display request is a display request for a list screen (for example, the list screen 1000 shown in FIG. 10) that aggregates and displays target data of a plurality of sites S.

  The accepting unit 1101 accepts a list display request. Specifically, for example, the reception unit 1101 receives a list display request by a user operation input. The receiving unit 1101 may receive a list display request by receiving a list display request from an external computer.

  The acquisition unit 1102 acquires html data of the screen of the target page in response to receiving the list display request. Specifically, for example, the acquiring unit 1102 first acquires account aggregation information (record) from the account aggregation information DB 220. Then, the acquisition unit 1102 accesses the login screen of the site S using the login URL of the acquired account aggregation information. Next, the acquisition unit 1102 logs in to the site S using the ID and PW of the acquired account aggregation information. Then, the acquisition unit 1102 acquires html data of the target page of the site S using the data URL of the acquired account aggregation information.

  The retrieval unit 1106 retrieves data (text data) specified by information on the tag recorded in advance in association with the data URL of the target page from the acquired html data of the target page. Specifically, for example, first, the search unit 1106 acquires site-specific purpose data attribute information (record) corresponding to the data URL of the target page from the site-specific target data attribute DB 230.

  Then, the search unit 1106 searches the data specified by the data specifying html attribute of the acquired site-specific target data attribute information from the html data of the target page. For example, when the site-specific target data attribute information 600-1 is acquired, the search unit 1106 searches for the data of the fourth td out of all td 12 from the html data of the target page.

  When the search unit 1106 does not search for data, the display control unit 1104 outputs image data of the target page screen based on the acquired html data of the target page screen. Specifically, for example, the display control unit 1104 displays an area reset screen including image data of the screen of the target page on the display 305. A screen example of the area resetting screen will be described later with reference to FIG.

  As a result, an area reset screen for resetting the area T including the target data on the screen of the target page after the change when the target page cannot be acquired due to the change in the screen configuration or posted content of the target page. It can be displayed on the display 305.

  Further, when data is retrieved by the retrieval unit 1106, the display control unit 1104 determines whether or not the data attribute of the data matches the data attribute recorded in advance in association with the information regarding the tag. Specifically, for example, the display control unit 1104 determines whether the data attribute of the searched data matches the data attribute of the site-specific target data attribute information corresponding to the data URL of the target page.

  If the data attributes of the data do not match, the display control unit 1104 may output image data of the target page screen based on the acquired html data of the target page screen. As a result, when the screen configuration or posted content of the target page is changed and the data attribute of the data of the region T is changed, the region reset for resetting the region T including the target data on the screen of the target page after the change is performed. A setting screen can be displayed on the display 305.

  The accepting unit 1101 accepts selection of an area T including the target data on the image data of the output target page screen. Specifically, for example, the accepting unit 1101 accepts selection of an area T including target data on the image data 1210 (see FIG. 12) by a user operation input on an area resetting screen 1200 described later.

  The recognition unit 1105 performs character recognition processing on the image data of the selected region T. The specific processing content of the character recognition processing is the same as when a new registration request is made.

  The specifying unit 1109 specifies information related to the tag of the html element including the target data in the html data of the target page. The specific processing content for specifying information related to the tag is the same as when a new registration is requested. Further, the specifying unit 1109 specifies the data attribute of the data recognized by the character recognition process. The specific processing content for specifying the data attribute is the same as when a new registration is requested.

  The registration unit 1103 updates the information about the tag recorded in advance in association with the URL of the target page with the information about the tag specified by the specifying unit 1109. Specifically, for example, the registration unit 1103 overwrites the specified data specifying html attribute with the data specifying html attribute of the site-specific target data attribute information in the site-specific target data attribute DB 230 corresponding to the data URL of the target page. To do. Also, the registration unit 1103 overwrites the identified data attribute with the data attribute of the site-specific purpose data attribute information.

  As a result, the site-specific purpose data attribute in the site-specific purpose data attribute DB 230 is updated in accordance with the change in the screen configuration of the target page and the posted content.

  The display control unit 1104 outputs a list screen in which the data searched by the search unit 1106 is inserted at a position recorded in advance in association with the position information of the region T on the list screen. Specifically, for example, first, the display control unit 1104 generates html data of the list screen 1000 in which the target data is not inserted, based on the html data of the list setting screen 800 (see FIG. 8).

  Next, the display control unit 1104 specifies the list position of the site-specific target data attribute information in the site-specific target data attribute DB 230 corresponding to the data URL of the target page. Then, the display control unit 1104 generates html data of the list screen 1000 in which the searched data is inserted at the specified list position and displays it on the display 305. As a result, a list screen can be displayed on the display 305 that aggregates and displays the target data of the plurality of sites S.

  In the above description, the information processing apparatus 101 has the function units 1101 to 1109. However, the server 201 may have the function units 1101 to 1109. Specifically, for example, the system 200 may be realized by SaaS (Software as a Service) that enables the information processing apparatus 101 to use a necessary function as a required service.

(Screen example of area reset screen)
Next, a screen example of the area resetting screen displayed on the display 305 of the information processing apparatus 101 will be described. The area reset screen is a screen for resetting the area T including the target data on the screen of the target page.

  FIG. 12 is an explanatory diagram illustrating a screen example of the area resetting screen. In FIG. 12, an area resetting screen 1200 is a screen for resetting an area T2 including target data on the welfare pension information page screen of the pension site. On the area resetting screen 1200, image data 1210 of the screen of the welfare pension information page of the pension site is displayed.

  In the area resetting screen 1200, as an operation input for specifying the area by the user, by accepting selection of an arbitrary range on the image data 1210, the area T2 including the target data on the screen of the welfare annuity information page can be reset. it can.

  In the example of FIG. 12, the region T2 including a numeric string indicating the annual amount (expected) [yen] on the screen of the welfare annuity information page is reset by the user's operation input. On the region reset screen 1200, when the setting completion button 1220 is clicked (pressed) by a user operation input, the resetting of the region T2 is completed.

  Thus, according to the area resetting screen 1200, the area T2 including the target data on the screen of the welfare pension information page of the pension site can be reset.

(Information provision processing procedure of information processing apparatus 101)
Next, an information provision processing procedure of the information processing apparatus 101 will be described.

  FIG. 13 is a flowchart illustrating an example of an information provision processing procedure of the information processing apparatus 101. In the flowchart of FIG. 13, first, the information processing apparatus 101 determines whether or not a new registration request has been received (step S1301).

  If a new registration request is accepted (step S1301: Yes), the information processing apparatus 101 executes a new registration process (step S1302). A series of processing by this flowchart is complete | finished. A specific processing procedure of the new registration processing will be described later with reference to the flowcharts of FIGS.

  On the other hand, if a new registration request has not been received (step S1301: No), the information processing apparatus 101 determines whether a list display request has been received (step S1303). Here, when the list display request is not received (step S1303: No), the information processing apparatus 101 returns to step S1301.

  On the other hand, when a list display request is received (step S1303: Yes), the information processing apparatus 101 executes list display processing (step S1304). A series of processing by this flowchart is complete | finished. A specific processing procedure of the list display processing will be described later with reference to a flowchart of FIG.

<Specific processing procedure of new registration processing>
Next, a specific processing procedure of the new registration processing in step S1302 shown in FIG. 13 will be described.

  14 and 15 are flowcharts showing an example of a specific processing procedure of the new registration processing. In the flowchart of FIG. 14, first, the information processing apparatus 101 acquires the ID, PW, login URL, and data URL of the site S (step S1401).

  The information processing apparatus 101 registers the acquired ID, PW, login URL, and data URL of the site S in the account aggregation information DB 220 (step S1402). As a result, new account aggregation information is newly registered in the account aggregation information DB 220.

  Next, the information processing apparatus 101 acquires html data of the target page of the site S using the acquired ID, PW, login URL, and data URL of the site S (step S1403). Then, the information processing apparatus 101 acquires the target page image data by capturing the target page based on the acquired html data (step S1404).

  Next, the information processing apparatus 101 displays a region initial setting screen including the acquired image data of the target page on the display 305 (step S1405). Then, the information processing apparatus 101 determines whether or not the region T including the target data on the image data of the target page has been selected by the user's operation input (step S1406).

  Here, the information processing apparatus 101 waits for the area T to be selected (step S1406: No). If the area T is selected (step S1406: Yes), the information processing apparatus 101 acquires the position information of the selected area T on the image data of the target page (step S1407).

  Next, the information processing apparatus 101 extracts the image data of the region T from the image data of the target page and performs the OCR process on the image data of the region T, thereby acquiring the text data of the target data (step S1408). . In the following description, text data obtained by the OCR process may be referred to as “region data”.

  The information processing apparatus 101 searches the html data of the target page for text data having the same content as the area data (step S1409). Next, the information processing apparatus 101 determines whether the number of search hits is “1” (step S1410). The number of search hits is the number of text data searched in step S1409.

  If the number of search hits is “1” (step S1410: NO), the information processing apparatus 101 proceeds to step S1508 shown in FIG. On the other hand, when the number of search hits is not “1” (step S1410: Yes), the information processing apparatus 101 determines whether the number of search hits is “0” (step S1411).

  If the number of search hits is “0” (step S1411: YES), the information processing apparatus 101 accepts an input of area data by a user operation input (step S1412) and returns to step S1409. That is, when the number of search hits is “0”, the information processing apparatus 101 determines that the target data has not been recognized, and causes the user to manually input text data (region data) of the target data.

  On the other hand, when the number of search hits is not “0” (step S1411: No), the information processing apparatus 101 proceeds to step S1501 illustrated in FIG. In the following description, the text data searched in step S1409 may be referred to as “search hit data”.

  In the flowchart of FIG. 15, first, the information processing apparatus 101 sets “m = 1” (step S1501), and converts the m-th search hit data from the top of the html data of the target page into text data indicating a predetermined character string. Change (step S1502).

  Next, the information processing apparatus 101 expands the html data of the changed target page in the memory 302, thereby capturing the changed target page and acquiring the changed target page image data (step S1503). ).

  Then, the information processing apparatus 101 extracts the image data of the area T from the image data of the target page after the change based on the position information of the area T acquired in step S1407, and performs the OCR process on the image data of the area T To obtain area data (step S1504). Next, the information processing apparatus 101 determines whether or not the area data obtained by the OCR process in step S1504 matches text data indicating a predetermined character string (step S1505).

  If the data does not match the text data indicating the predetermined character string (step S1505: No), the information processing apparatus 101 changes the changed search hit data to the search hit data before the change (step S1506). The information processing apparatus 101 increments “m” (step S1507) and returns to step S1502.

  On the other hand, in step S1505, when the data matches the text data indicating the predetermined character string (step S1505: Yes), the information processing apparatus 101 identifies the mth search hit data as the text data of the target data (step S1508). .

  Then, the information processing apparatus 101 identifies the data attribute of the area data (step S1509). Next, the information processing apparatus 101 specifies the data specification html attribute of the tag of the html element in the html data of the target page by searching the html element including the region data from the html data of the target page before the change (step). S1510).

  The information processing apparatus 101 registers the specified data specific html attribute and data attribute in the site-specific target data attribute DB 230 in association with the data URL of the target page (step S1511), and calls the new registration process. Return to. As a result, new site-specific purpose data attribute information is newly registered in the site-specific purpose data attribute DB 230.

  The list position corresponding to the data URL of the target page is accepted by the user's operation input on the list setting screen (for example, the list setting screen 800), and is associated with the data URL of the target page in the list information DB 240. Is set.

<Specific processing procedure of list display processing>
Next, a specific processing procedure of the list display processing in step S1304 shown in FIG. 13 will be described.

  FIG. 16 is a flowchart illustrating an example of a specific processing procedure of the list display processing. In the flowchart of FIG. 16, first, the information processing apparatus 101 generates html data of a list screen in which target data is not inserted (step S1601).

  Next, the information processing apparatus 101 acquires the number n of records in the account aggregation information DB 220 (step S1602) and sets “i = 1” (step S1603). The information processing apparatus 101 executes target data setting processing (step S1604). A specific processing procedure of the target data setting process will be described later with reference to a flowchart of FIG.

  Next, the information processing apparatus 101 increments “i” (step S1605), and determines whether “i” is greater than “n” (step S1606). If “i” is equal to or smaller than “n” (step S1606: NO), the information processing apparatus 101 returns to step S1604.

  On the other hand, when “i” becomes larger than “n” (step S1606: Yes), the information processing apparatus 101 displays the html data of the list screen on the display 305 (step S1607) and calls the list display process. Return to step.

  Thereby, a list screen (for example, list screen 1000) in which target data of a plurality of sites S are aggregated can be displayed on the display 305.

<Specific processing procedure of target data setting processing>
Next, a specific processing procedure of the target data setting process in step S1604 shown in FIG. 16 will be described.

  FIG. 17 is a flowchart illustrating an example of a specific processing procedure of the target data setting process. In the flowchart of FIG. 17, the information processing apparatus 101 first acquires the i-th record (hereinafter referred to as “record Ra”) of the account aggregation information DB 220 (step S1701).

  Next, the information processing apparatus 101 accesses the login screen of the site S using the acquired login URL of the record Ra, and logs in to the site S using the ID and PW of the record Ra (step S1702). Then, the information processing apparatus 101 acquires the html data of the target page of the site S using the data URL of the record Ra (step S1703).

  Next, the information processing apparatus 101 acquires a record (hereinafter referred to as “record Rb”) corresponding to the data URL of the acquired record Ra from the site-specific objective data attribute DB 230 (step S1704). Then, the information processing apparatus 101 searches the data specified by the data specification html attribute of the acquired record Rb from the acquired html data of the target page (step S1705).

  Next, the information processing apparatus 101 determines whether data is retrieved from the html data of the target page (step S1706). Here, when the data is searched (step S1706: Yes), the information processing apparatus 101 specifies the data attribute of the searched data (step S1707).

  The information processing apparatus 101 determines whether the specified data attribute matches the data attribute of the record Rb (step S1708). If the data attributes match (step S1708: YES), the information processing apparatus 101 acquires a list position corresponding to the data URL of the record Ra from the list information DB 240 (step S1709).

  The information processing apparatus 101 inserts the retrieved data into the html data on the list screen based on the identified list position (step S1710), and returns to the step that called the target data setting process. As a result, it is possible to generate html data of a list screen in which the target data of the site S is embedded at a preset list position.

  If no data is retrieved in step S1706 (step S1706: No), the information processing apparatus 101 executes region reset screen display processing (step S1711) and returns to step S1701. A specific processing procedure of the area reset screen display process will be described later with reference to the flowchart of FIG.

  If the data attributes do not match in step S1708 (step S1708: No), the information processing apparatus 101 proceeds to step S1711.

<Specific processing procedure of area reset screen display processing>
Next, a specific processing procedure of the area reset screen display process in step S1711 shown in FIG. 17 will be described.

  FIG. 18 is a flowchart illustrating an example of a specific processing procedure of the area reset screen display process. In the flowchart of FIG. 18, first, the information processing apparatus 101 acquires image data of the target page by capturing the target page based on the html data of the target page (step S1801).

  Then, the information processing apparatus 101 displays an area resetting screen including the image data of the acquired screen of the target page on the display 305 (step S1802). Next, the information processing apparatus 101 determines whether or not the region T including the target data on the image data of the target page has been selected by the user's operation input (step S1803).

  Here, the information processing apparatus 101 waits for the area T to be selected (step S1803: No). When the region T is selected (step S1803: Yes), the information processing apparatus 101 extracts the image data of the region T from the image data of the target page and performs OCR processing on the image data of the region T. Area data is acquired (step S1804).

  Next, the information processing apparatus 101 identifies the data attribute of the acquired area data (step S1805). The information processing apparatus 101 searches the html data of the target page for an html element that includes the acquired region data (step S1806). Next, the information processing apparatus 101 specifies the data specifying html attribute of the tag of the searched html element (step S1807).

  Then, the information processing apparatus 101 updates the site-specific purpose data attribute DB 230 by overwriting the specified data attribute and the data specific html attribute on the record Rb (step S1808), and calls the area reset screen display process. Return to step. Thereby, the storage content of the site-specific purpose data attribute DB 230 can be updated in accordance with the change of the screen configuration of the target page and the posted content.

  As described above, according to the information processing apparatus 101 according to the embodiment, the same content as the text data obtained from the image data of the region T set on the image data of the target page from the html data of the target page. Text data can be searched. Thereby, text data having the same content as the target data can be searched from the html data of the target page.

  Further, according to the information processing apparatus 101, when a plurality of text data is searched, any one of the plurality of text data in the html data of the target page can be changed to different text data. Further, according to the information processing apparatus 101, whether or not the text data obtained from the image data in the region T on the image data of the target page based on the html data of the target page after the change matches the changed different text data. Can be determined.

  Further, according to the information processing apparatus 101, when the text data matches the changed different text data, the text data changed to the different text data among the html data of the target page can be specified as the text data corresponding to the region T. it can. Thereby, even when there are a plurality of text data having the same content as the target data, the position of the target data in the html data of the target page can be accurately specified.

  Further, according to the information processing apparatus 101, information on the tag in the html data of the target page specified by the text data corresponding to the specified region T can be recorded in association with the data URL of the target page.

  Further, according to the information processing apparatus 101, it is possible to search the data specified by the information about the recorded tag from the obtained html data of the target page. Further, according to the information processing apparatus 101, when data is not searched, an area reset screen (for example, the area reset screen 1200) including the image data of the target page based on the acquired html data of the target page is displayed. 305 can be displayed.

  As a result, it is possible to determine whether or not information as intended by the user can no longer be acquired due to a change in the screen configuration of the target page or the posted content, depending on whether or not the data specified by the information related to the tag can be searched. Further, when it becomes impossible to acquire information as intended by the user, it is easy to re-set which part of the target page information is to be acquired, and it is possible to reduce the trouble of changing the setting.

  Further, according to the information processing apparatus 101, the information related to the recorded tag can be updated with the information related to the tag specified by the data of the region T set on the output image data of the target page. Thereby, the storage content of the site-specific purpose data attribute DB 230 can be updated in accordance with the change of the screen configuration of the target page and the posted content.

  Further, according to the information processing apparatus 101, when data is searched, it is possible to determine whether or not the data attribute of the data matches the data attribute recorded in advance in association with the information related to the tag. Further, according to the information processing apparatus 101, when the data attribute does not match the pre-recorded data attribute, the area reset screen including the acquired image data of the target page can be displayed on the display 305.

  As a result, even if the data is searched, if the data attribute of the searched data is different from the pre-recorded data attribute, it is determined that the information as intended by the user cannot be obtained, and the image data of the target page is The area reset screen including the area reset screen (for example, the area reset screen 1200) can be displayed on the display 305.

  Further, according to the information processing apparatus 101, when the data attribute matches the pre-recorded data attribute, a list screen in which the searched data is inserted is output at the list position recorded in advance in association with the information related to the tag. be able to. Thereby, a list screen (for example, list screen 1000) in which target data of a plurality of sites S are aggregated can be displayed on the display 305.

  The data specifying method described in this embodiment can be realized by executing a program prepared in advance on a computer such as a personal computer or a workstation. The data specifying program is recorded on a computer-readable recording medium such as a hard disk, a flexible disk, a CD-ROM, an MO, and a DVD, and is executed by being read from the recording medium by the computer. The data specifying program may be distributed through a network such as the Internet.

  The following additional notes are disclosed with respect to the embodiment described above.

(Supplementary note 1)
Search the screen information of the screen for text data having the same content as the text data obtained from the image data in the range of selection received on the screen data of the site,
Change the searched text data in the screen information of the screen to different text data,
Determining whether text data obtained from image data in the same range as the selected range on the screen image data based on the screen information after the change matches the different text data By specifying the text data corresponding to the range that received the selection from the screen information of the screen,
A data identification program for executing a process.

(Supplementary note 2)
When a plurality of text data is searched, the text data of any of the plurality of text data in the screen information of the screen is changed to different text data,
The process to specify is
The data specification according to appendix 1, wherein if any of the different text data matches, the text data in the screen information of the screen is specified as text data corresponding to the range in which the selection is accepted. program.

(Supplementary note 3)
A supplementary note that executes processing for recording information related to a tag in screen information of the screen specified by text data corresponding to a range in which the selected selection is received, in association with identification information of the site. The data identification program according to 1 or 2.

(Supplementary note 4)
Based on the recorded identification information of the site and information related to the tag in the screen information of the screen of the site, the data specified by the information related to the tag is searched from the acquired screen information of the screen of the site.
When the data is not searched, the image data of the site based on the acquired screen information of the site is output.
4. The data specifying program according to appendix 3, wherein the program is executed.

(Supplementary note 5)
When the data is searched, the list screen in which the searched data is inserted at the position recorded in association with the information on the tag in the list screen that aggregates and displays information of a plurality of sites is displayed. The data identification program according to appendix 4, wherein the output process is executed.

(Appendix 6)
Search the screen information of the screen for text data having the same content as the text data obtained from the image data in the range of selection received on the screen data of the site,
Change the searched text data in the screen information of the screen to different text data,
Determining whether text data obtained from image data in the same range as the selected range on the screen image data based on the screen information after the change matches the different text data By specifying the text data corresponding to the range that received the selection from the screen information of the screen,
A data identification method characterized by executing processing.

(Supplementary note 7) Text data having the same content as the text data obtained from the image data in the range of selection received on the screen image data of the site is searched from the screen information of the screen, and the search within the screen information of the screen The text data obtained from the image data in the same range as the range on which the selection is received on the image data of the screen based on the screen information of the screen after the change is changed to different text data, A controller that identifies text data corresponding to a range in which the selection is accepted from the screen information of the screen by determining whether or not the text data matches different text data;
An information processing apparatus comprising:

(Appendix 8)
Search the screen information of the screen for text data having the same content as the text data obtained from the image data in the range of selection received on the screen data of the site,
Change the searched text data in the screen information of the screen to different text data,
Determining whether text data obtained from image data in the same range as the selected range on the screen image data based on the screen information after the change matches the different text data By specifying the text data corresponding to the range that received the selection from the screen information of the screen,
A computer-readable recording medium in which a data specifying program for executing processing is recorded.

101 Information processing apparatus 200 System 201 Server 220 Account aggregation information DB
230 Site-specific purpose data attribute DB
240 List information DB
DESCRIPTION OF SYMBOLS 1101 Reception part 1102 Acquisition part 1103 Registration part 1104 Display control part 1105 Recognition part 1106 Search part 1107 Change part 1108 Determination part 1109 Identification part

Claims (7)

On the computer,
Search the screen information of the screen for text data having the same content as the text data obtained from the image data in the range of selection received on the screen data of the site,
Change the searched text data in the screen information of the screen to different text data,
Determining whether text data obtained from image data in the same range as the selected range on the screen image data based on the screen information after the change matches the different text data By specifying the text data corresponding to the range that received the selection from the screen information of the screen,
A data identification program for executing a process. The process to change is
When a plurality of text data is searched, the text data of any of the plurality of text data in the screen information of the screen is changed to different text data,
The process to specify is
2. The data according to claim 1, wherein when the text data matches the different text data, the text data in the screen information of the screen is specified as text data corresponding to the range in which the selection is accepted. Specific program. In the computer,
A process of recording information related to a tag in screen information of the screen specified by text data corresponding to a range in which the selected selection is received in association with identification information of the site is executed. Item 3. The data specifying program according to item 1 or 2. In the computer,
Based on the recorded identification information of the site and information related to the tag in the screen information of the screen of the site, the data specified by the information related to the tag is searched from the acquired screen information of the screen of the site.
When the data is not searched, the image data of the site based on the acquired screen information of the site is output.
The data specifying program according to claim 3, wherein a process is executed. In the computer,
When the data is searched, the list screen in which the searched data is inserted at the position recorded in association with the information on the tag in the list screen that aggregates and displays information of a plurality of sites is displayed. The data specifying program according to claim 4, wherein an output process is executed. Computer
Search the screen information of the screen for text data having the same content as the text data obtained from the image data in the range of selection received on the screen data of the site,
Change the searched text data in the screen information of the screen to different text data,
Determining whether text data obtained from image data in the same range as the selected range on the screen image data based on the screen information after the change matches the different text data By specifying the text data corresponding to the range that received the selection from the screen information of the screen,
A data identification method characterized by executing processing. Search the screen information of the screen for text data having the same contents as the text data obtained from the image data in the range of selection received on the image data of the screen of the site, and search for the searched text data in the screen information of the screen Change to different text data, text data obtained from image data in the same range as the selected range on the image data of the screen based on the screen information of the screen after the change, the different text data A control unit that identifies text data corresponding to a range in which the selection is accepted from the screen information of the screen by determining whether or not they match;
An information processing apparatus comprising:

Images