JP2020042427A - Information processing apparatus, control method thereof, and program - Google Patents

Abstract

To provide an information processing apparatus capable of easily inputting a text such as an explanation of an image or a comment by using a result of image analysis, a control method thereof, and a program.SOLUTION: In a text input paired with an image, a text input subject image is analyzed to extract a keyword, the extracted keyword is temporarily registered in an input dictionary, and the order of input candidates is adjusted so that the extracted keyword is displayed preferentially when the text input is performed.SELECTED DRAWING: Figure 5

Description

  The present invention relates to an information processing apparatus, a control method thereof, and a program, and more particularly to input of text associated with an image.

  2. Description of the Related Art Conventionally, a user may input text describing an image into an image to create a document, or record text together with the image. Further, in recent years, it has been common to post a post including an image to a third party via a network. For example, in a social networking service (SNS) or the like, an image and text describing the image are shared with a third party. Also, a third party who has received the disclosure can input a comment related to the image.

  With respect to an image published in this way and text such as an explanation and a comment of the image, a technique of analyzing text such as an explanation and a comment attached to the image and determining a priority of the image is disclosed. For example, in Patent Literature 1, an image-related noun and an adjective word included in a comment for each image are extracted and analyzed from comments related to a published image, and the priority of an image posted from the analyzed comment is determined. Judge and perform display control.

JP 2008-139948 A

  When inputting text or comments describing an image, an input prediction function that provides input candidates for a device or a browser, kana-to-kanji conversion, or the like may make it difficult to find a name or keyword that matches the image. In such a case, for example, it may be necessary to select a target text from the rear of the candidate, and it is troublesome to input a description of an image or a comment. However, Patent Document 1 described above describes that the priority of an image is determined by analyzing the contents of the comment of the image. However, a device for simplifying the input of a keyword suitable for the image is described. Not done.

  Therefore, an object of the present invention is to provide a means capable of easily inputting text such as a description of an image or a comment by using a result of image analysis.

  In order to achieve the above object, the present invention provides an input unit for inputting characters related to an image, a prediction unit for indicating a candidate of a word or a sentence according to an input to the input unit, and a result of image analysis of the image. And control means for controlling the prediction means to indicate candidate characters based on the prediction means.

  According to the present invention, it is possible to easily input a text such as a description of an image or a comment by using a result of the image analysis.

1 is a block diagram illustrating a configuration example of an information processing apparatus according to an embodiment of the present invention. FIG. 4 is a diagram illustrating an example of a text display screen and an edit screen according to the embodiment of the present invention. FIG. 4 is a diagram illustrating an example of an image management table and a text management table according to the embodiment of the present invention. 9 is a flowchart illustrating an image acquisition process of the information processing apparatus according to the embodiment of the present invention. 4 is a flowchart illustrating a text input process of the information processing apparatus according to the embodiment of the present invention.

  Hereinafter, preferred embodiments of the present invention will be described with reference to the drawings.

(Description of configuration example)
FIG. 1 is a block diagram illustrating a configuration example of an information processing apparatus 100 according to an embodiment of the present invention. In the present embodiment, the case where the information processing apparatus 100 is a single computer apparatus will be described as an example. However, each function may be distributed to a plurality of computer apparatuses as needed. When a plurality of computer devices are used, they are connected to each other via a communication line such as a local area network (LAN) so that they can communicate with each other.

  In FIG. 1, a control unit 101 is, for example, a Central Processing Unit (CPU), and controls the entire information processing apparatus 100. The ROM 102 is a read only memory and stores programs and parameters that do not need to be changed. The RAM 103 is a Random Access Memory, and temporarily stores work information at the time of processing and display in the apparatus. For example, it temporarily stores programs and display data such as images supplied from an external device or the like via a network. The control unit 101 executes the programs stored in the ROM 102 or the RAM 103 to implement each process of the present embodiment described later.

  The external storage device 104 is a hard disk or a memory card fixedly installed in the information processing device 100, and stores a program such as an operating system (OS). The external storage device 104 can store and read application programs, data, libraries, and the like. The document editing application program software (hereinafter, referred to as a document editing application) of this embodiment, document data, image files, and the like are stored. In the following embodiments of the present invention, a set of an image and text information added to or associated with the image, for example, a description or comment of the image, is defined and described as a “document”. The “document” may include a single image or a plurality of images, and may include a plurality of texts associated with each image.

  The input I / F 105 (input interface) receives a user operation. For example, it is connected to an input device such as a pointing device or a keyboard for inputting data.

  The BMU 106 (Bit Move Unit) controls, for example, data transfer between memories (for example, between the VRAM 107 and another memory) and data transfer between the memory and each I / O device (for example, the network interface 109).

  The VRAM 107 (Video RAM) is a memory for drawing a graphic display such as an image on a display. The display unit 111 displays an image or the like based on the received signal. The image generated by the VRAM 107 is transmitted to the display unit 111 according to a predetermined rule, and the display unit 111 displays the image.

  The NET I / F 109 (network interface) connects to a communication line 110 which is a network such as the Internet. Via the communication circuit 110, it can communicate with an external device such as a PC, a server, and the like, and can transmit and receive data such as images to and from the external device, and can perform a user operation from the external device. The system bus 112 connects the units 101 to 109 in a communicable manner.

  In the present embodiment, a case where the present invention is applied to an application of a smartphone or a personal computer will be described as an example, but the present invention is not limited to this. The present invention may be applied to an imaging device such as a digital camera. That is, the present invention is also applicable to a case where a captured image recorded on a recording medium readable by a digital camera such as a memory card is reproduced and displayed on a display such as a rear liquid crystal device of the digital camera. Furthermore, it can be applied to devices that can display contents such as images, texts, and illustrations, such as PDAs, mobile phones and portable image viewers, printers with displays, digital photo frames, music players, game consoles, and e-book readers. It is possible.

  Hereinafter, as an embodiment of the information processing apparatus of the present invention, a process of editing text related to an image such as a caption of an image according to a user operation will be described. An information processing device such as a smartphone or a personal computer starts an editing application and inputs and edits images and text.

(Example of document display and edit screen)
FIG. 2 is a diagram showing an example of a text display screen and an edit screen according to the embodiment of the present invention. FIG. 2A shows an example of a preview display screen, and FIG. 2B shows an example of an edit screen.

  In the example of the preview display screen of FIG. 2A, the display screen 200 displays a page to be edited and a page list. A preview of a page (document) to be edited is displayed in a preview area 201 of the display screen 200, and a page list of the document is displayed in a list area 206.

  In the preview area 201 where the page to be edited is displayed, the image areas 202 and 203 indicate the display area of the image arranged on the target page. A text area 204 indicates a text display area explaining an image in the image area 202, and a text area 205 indicates a text display area explaining an image in the image area 203.

  The list area 206 displays a list of a plurality of pages that can be displayed by the information processing apparatus 100, and the user can scroll the page list displayed in the list area 206 by operating a scroll bar or flicking the area. Further, a page to be edited can be selected from a plurality of pages by a touch operation or the like. The focus frame 207 marks a page displayed in the preview area 201, which is selected from among a plurality of pages displayed as a list and to be edited.

  FIG. 2B shows an example of the editing screen, particularly in the case of text editing. In the preview display screen of FIG. 2A, in response to a user's touch operation or the like, for example, an operation such as selecting an image area, selecting a text area, or performing a specific mode change operation is performed, and the edit mode is changed. When selected, a transition is made to the editing screen of FIG. In the present embodiment, particularly, a case is shown in which a mode for editing the text area 205 is entered. Separately, an “edit” button or icon may be provided. On the editing screen, input, deletion, change, and the like of images and texts are possible.

  The display screen 210 displays a preview of a page to be edited and a text editing area. A preview area 211 on the display screen 210 displays a preview of a page to be edited, and image areas 212 and 213 indicate display areas of images arranged on the page to be edited. A text area 214 indicates a text display area for explaining an image in the image area 212, and a text area 215 indicates a text display area for explaining an image in the image area 213. It can be seen that the text area to be edited is selected. Is marked. In the example of FIG. 2B, the text in the text area 215 is being edited, and the text area 215 is marked.

  The text editing area 216 indicates an operation area for actually editing text. As described above, the characters in the text area selected as the editing target are edited by performing operations such as input, deletion, and conversion on the text editing area 216. The text editing area 216 includes a character input area 217 and a word candidate area 218 in which input candidate words and sentences are displayed. The input area 217 is an area where characters, symbols, and the like can be input and deleted according to a user operation. In the word candidate area 218, when the input area 217 is input, a word or a sentence extracted as a candidate for an input word or a conversion word is displayed. Even when a word or a sentence of a desired input candidate is selected from the word candidate area 218, a word or a sentence can be input to the input area 217, and the input content of the input area 217 can be converted. is there.

  The icon 219 is a control button for terminating text input. By performing a click operation on the icon 219 by touch or other instruction means, the character string input to the input area 217 is determined as the content of the text area to be edited. In the example of FIG. 2B, the character content of the selected text area 215 is determined. Note that the method and display format of page editing and text editing in FIG. 2 are merely examples, and the present invention is not limited to this, and the present invention is applied to the case where text relating to an image is added.

(Example of image and text information management table)
With reference to FIG. 3, a table for managing image information and text information according to the present embodiment among tables for managing information on documents used by the document editing application (hereinafter, referred to as document information DB) will be described. FIG. 3 is a diagram illustrating an example of the image management table and the text management table according to the embodiment of the present invention.

  The image management table 300 shown in FIG. 3A manages information of a document image. The image management table stores an image ID 301, an original image file path 302, a thumbnail image file path 303, and a keyword list 304. A record is generated at the timing when the user generates a page including an image area, and an image ID 301 is issued.

  The image ID 301 is a unique ID assigned to each image. The original image file path 302 has a path storing the original file of the image, and the thumbnail image file path 303 has a path storing the thumbnail image. The image analysis keyword list 304 stores keywords related to the image extracted from the image by analyzing the image.

  The text management table 310 shown in FIG. 3B manages text information of a document. The text management table 310 has a text ID 311, a text content 312, an associated image ID 313, and a dictionary update flag 314. A record is generated at the timing when the user generates a page including a text area, and a text ID 311 is issued.

  The text management table 310 has a text content 312 for storing the text input by the user. Since the text area describes an image, the association of the related image is held as the image ID 313. Depending on the layout of the document, for example, when one text is associated with a plurality of images, a plurality of image IDs may be associated. Also, a dictionary update flag 314 indicating whether the dictionary information of the text input area has been updated is provided.

(Control flow when acquiring images related to document editing)
Hereinafter, a control flow at the time of image acquisition will be described with reference to FIG. The flowchart of FIG. 5 illustrates a processing procedure executed by controlling each processing block by the control unit 101 of the information processing apparatus 100. The control unit 101 reads a table or an image file storing document information used by a document editing application in the RAM 103 and performs document editing according to an instruction of a program read and started from the ROM 102.

  The control unit 101 monitors an event to the preview area 201 (for example, a touch operation or an operation by another operation member), and when an event to the image areas 202 and 203 in the preview area 201 is detected, The control flow at the time of image acquisition in FIG. 4 is started. The control unit 101 is, for example, made to select a document to be edited from an existing document list stored in the document information DB in advance or to create a new document in response to a user operation, so that a new document is created. Select a document. In the present embodiment, an explanation will be given of a case where an existing document is to be edited and an image in the image area 202 is edited.

  First, in step S401, an image file instructed by the user to be arranged in the image area 202 to be edited is temporarily stored in the RAM 103. A thumbnail image file for display is generated from the received image file. The image file that can be arranged in the preview area 202 or 203 may be data stored in the external storage device 104 or data taken in from outside via a network. As a method of selecting an image file, a method of obtaining a file list of a device or the like and selecting a specific image file from a list of a file name or a thumbnail list can be considered, but is not limited to this.

  Next, in step S402, using the image ID associated with the image area to be edited as a key, the image management table in FIG. 3A of the document information DB is searched and image information is stored. An original image file path 302 of the acquired image and a thumbnail image file path 303 for displaying the image are stored.

  In step S403, image analysis of the original image is performed. The image data stored in the original image file path 302 of the image is analyzed by using the image ID 301 of the image management table as a key in FIG. Then, as a result of the image analysis, the keywords extracted from the subject of the image are stored in the image analysis keyword list 304 of the image management table in FIG. For example, keywords that can be extracted by well-known image analysis, such as objects (persons, buildings, flowers, etc.) shown in images, recognized objects (specific people, building names, etc.), and color information of the objects, are images. It is stored in the analysis keyword list 304. A keyword may be added based on GPS information or the like added to the image file other than the image analysis. Further, when the image has date and time information such as the image shooting date and time and the image registration date and time, or when the age can be identified from the analysis of the subject, a process of converting the keyword into a wording suitable for that era may be included.

  In step S404, an image is acquired from the thumbnail image file path 303 for displaying an image by using the image ID 301 of the image management table in FIG. 3A as a key, and in the present embodiment, the image is displayed in the image area 202 and displayed. End the acquisition flow.

(Control flow for text input related to document editing)
Hereinafter, a control flow at the time of text editing will be described with reference to FIG. The flowchart of FIG. 5 illustrates a processing procedure executed by controlling each processing block by the control unit 101 of the information processing apparatus 100. The control unit 101 reads out a table or an image file in the RAM 103 which stores document information used by a document editing application and edits the document according to an instruction of a program read and started from the ROM 102.

  When an editing target page including the image areas 202 and 203 and the text areas 204 and 205 shown in FIG. 2A is displayed in the preview area 201, an event (for example, a touch operation or another operation member) to the preview area 201 is performed. Operation, etc.). When an event to the text areas 204 and 205 is detected, the text editing flow of FIG. 5 is started. In the present embodiment, a case where an event in the text area 205 is detected will be described as an example.

  First, in step S501, a transition is made to a text editing screen in response to a tap event to the text editing area 205. In other words, transition from the state shown in FIG. 2A to the state shown in FIG. 2B is performed, the text area 215 of the preview area 211 of the page to be edited is marked, and the text editing area 216 is displayed on the display screen 210. The text editing area 216 has a text input area 217 and an input candidate area 218. In the text editing area 216, a control button 219 for ending the character input of the text is displayed. The information processing apparatus 100 holds dictionary information for assisting text input in the ROM 102 or another storage unit, and the control unit 101 refers to the dictionary information when a character is input to the input area 217. Then, words / sentences which are input candidates are displayed in the input candidate area 218.

  Next, in step S502, the text management table of FIG. 3B in the document information DB is searched using the text ID linked to the text area 205 when the document was created.

  In step S503, the image management table of FIG. 3A in the document information DB is searched using the image ID stored in the searched text management table of FIG. 3B as a key.

  In step S504, the retrieved image analysis keyword list 304 in the image management table of FIG. 3A is acquired. Although the example in which the analysis process is performed during the image acquisition flow of FIG. 4 to extract the keyword has been described above, the keyword is acquired by performing the image analysis at the timing when the text input is performed (that is, step S503 of the flow). Dictionary information may be generated. The timing of image analysis is not limited. If the image analysis keyword list 304 corresponding to the image linked to the text area 205 exists, the process proceeds to step 505. If the image analysis keyword 304 cannot be extracted, the process of step 505 is skipped and the dictionary information is updated. Is not performed.

  In step S505, the information processing apparatus 100 updates the dictionary information stored in the RAM 102 or another storage area according to the text area to be input (the text area 205 in the present embodiment). That is, the contents of the image analysis keyword list 304 relating to the image associated with the text area 205 are added as dictionary information to the dictionary information held by the information processing apparatus 100 to assist text input. When the dictionary information has been updated, the dictionary update flag 314 of the text management table in FIG. 3B of the document information DB is turned ON.

  In step S506, by referring to the updated dictionary information, the display of the input candidate area 218 is updated so that the words and sentences included in the keyword list and the words and sentences related thereto are preferentially displayed. The input candidate may be a prediction input for predicting a subsequent character candidate in accordance with a key input. This input candidate may be a kana-kanji conversion candidate according to a key input.

  In step S507, the control unit 101 detects that the control button 219 for terminating text character input has been pressed, and performs a user text input completion process. The value input to the text character input area 217 is obtained and stored in the text information 312 of the text management table in FIG. 3B of the document information DB.

  In step S508, if the dictionary update flag 314 of the text management table in FIG. 3B of the document information DB is ON, the dictionary information held by the information processing apparatus 100 to assist the text input is displayed. Restore the default state before updating. Then, the dictionary update flag 314 of the text management table of FIG. 3B of the document information DB is turned off, and the display of the document is returned to the display screen 200 of FIG. 3A on the display area of the device.

  As described above, the present invention has been described in detail based on the preferred embodiments. However, the present invention is not limited to these specific embodiments, and various forms that do not depart from the gist of the present invention are also included in the present invention. included. A part of the above-described embodiments may be appropriately combined.

  Further, when a software program for realizing the functions of the above-described embodiments is supplied to a system or an apparatus having a computer capable of executing the program directly from a recording medium or using wired / wireless communication, and the program is executed. Is also included in the present invention. Therefore, the program code itself supplied and installed in the computer to implement the functional processing of the present invention by the computer also implements the present invention. That is, the present invention includes the computer program itself for implementing the functional processing of the present invention. In this case, any form of the program, such as an object code, a program executed by an interpreter, and script data to be supplied to the OS, may be used as long as the program has the function of the program. As a recording medium for supplying the program, for example, a magnetic recording medium such as a hard disk or a magnetic tape, an optical / magneto-optical storage medium, or a nonvolatile semiconductor memory may be used. As a method for supplying the program, a method in which a computer program forming the present invention is stored in a server on a computer network, and a connected client computer downloads and executes the computer program is also conceivable.

Claims (14)

Input means for inputting characters related to the image,
Prediction means for indicating a candidate for a word or a sentence according to the input to the input means,
Control means for controlling the prediction means to indicate candidate characters based on the result of image analysis of the image,
An information processing apparatus comprising: Storage means for storing a keyword extracted by image analysis for each image,
2. The information processing apparatus according to claim 1, wherein when performing text input relating to the image, the control unit indicates a stored keyword as a candidate of the prediction unit as a priority candidate. 3. Conversion means for instructing kana-kanji conversion of the character input to the input means,
3. The information processing apparatus according to claim 1, wherein when the kana-kanji conversion is instructed, the control unit indicates a conversion candidate character of the conversion unit based on a result of analyzing the image. 4. .   The control means, when the information on the date and time when the image was obtained by the image analysis of the image or the age at which the image was obtained can be specified, the candidate shown in the prediction means is a word or a sentence in that era. The information processing apparatus according to claim 1, wherein:   The control unit is characterized in that when the information on the date and time when the image was obtained by the image analysis of the image or the date when the image was obtained can be specified, the conversion candidate is set to a word or a sentence in the era. The information processing apparatus according to claim 1, wherein: A first dictionary unit that creates dictionary information as a candidate for the prediction unit based on a word or a sentence extracted for each image;
A first switching unit that selects a candidate of the prediction unit based on the dictionary information different for each associated image when inputting characters related to the image;
The information processing apparatus according to claim 1, further comprising: A second dictionary means for creating dictionary information as a conversion candidate of the kana-kanji means for words or sentences extracted for each image,
A second switching unit for selecting a conversion candidate of the kana-kanji conversion unit based on the dictionary information for each associated image when inputting characters related to the image;
The information processing apparatus according to claim 1, further comprising: Input means for inputting characters related to the image
Conversion means for instructing kana-kanji conversion of the character input to the input means,
Control means for controlling the conversion means to indicate a character of a conversion candidate based on the result of analyzing the image,
An information processing apparatus comprising: Storage means for storing a keyword extracted by image analysis for each image,
9. The information processing apparatus according to claim 8, wherein, when inputting text related to the image, the control unit indicates the saved keyword as a conversion candidate of the kana-kanji conversion unit as a priority candidate.   The control unit is characterized in that when the information on the date and time when the image was obtained by the image analysis of the image or the date when the image was obtained can be specified, the conversion candidate is set to a word or a sentence in the era. The information processing device according to claim 8 or 9, wherein Dictionary means for creating dictionary information as a conversion candidate of the kana-kanji means, the words or sentences extracted for each image,
Switching means for selecting a conversion candidate of the kana-kanji conversion means based on the dictionary information for each associated image when inputting characters related to the image,
The information processing apparatus according to any one of claims 8 to 10, comprising: Inputting characters related to the image;
Indicating a candidate word or sentence based on the result of image analysis of the image according to the input,
Indicating a candidate for kana-kanji conversion of the input character based on a result of image analysis of the image;
A method for controlling an information processing apparatus, comprising:   A computer program for causing a computer to execute the steps according to claim 12.   A computer-readable recording medium in which the program according to claim 13 is described.

Images