JP2018106481A - Display device, display controlling device and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a display device, a display controlling device and a program that have an excellent usability.SOLUTION: A display device 1 according to the present invention includes: an imaging unit 110 that images an object; a specifying unit 130 that specifies positions of multiple search subjects in a motion image imaged by the imaging unit 110; and a display unit 140 that displays multiple augmented reality image so as to allow a user to identify each of the multiple search subjects from the imaged object on the basis of the respective positions of the multiple search subjects specified by the specifying unit 130.SELECTED DRAWING: Figure 3

Description

  The present invention relates to a display device, a display control device, and a program for displaying an image obtained by photographing a subject.

  Conventionally, patent document 1 is known as this kind of technique. In Patent Literature 1, when a character is photographed with a mobile terminal and a character code corresponding to the character is acquired, the character string photographed immediately before is highlighted so that the character string to be photographed is not lost. Techniques to do this are disclosed. This highlighting is called augmented reality display and has been widely used in game machines and smartphones in recent years.

JP 2006-209599 A

  Incidentally, the usability of Patent Document 1 has room for further improvement.

  It is an object of the present invention to provide a display device, a display control device, and a program that are easier to use.

  The display device according to the present invention is based on an imaging unit that captures a subject, a specifying unit that specifies a position of a search target in an image captured by the imaging unit, and a position of the search target specified by the specifying unit. A display unit for displaying an augmented reality image so that the user can identify the search target from among the subjects.

According to the configuration of the present invention, since the position of the search target in the image obtained by photographing the subject is specified and the augmented reality image is displayed based on the result of the specification, the user can easily select the search target from the subject. Can be identified.
The “subject” of “so that the user can identify the search target from among the subjects” may be “a subject that can be seen through a transparent screen” or “a subject that appears in the image”.
In addition, the image captured by the imaging unit may be a moving image or a still image.
The specifying unit may specify the position of one detection target or may specify the positions of a plurality of detection targets. In the latter case, the display unit displays a plurality of augmented reality images, and the display control unit displays the plurality of augmented reality images.

  The display device includes an acquisition unit that acquires a search condition for searching for a search target from an image, and the specifying unit searches the search target from the image based on the search condition, and searches for the search target Based on the result, the position of the search target is specified.

According to the configuration of the present invention, the user can instruct search conditions.
Note that the acquisition unit is not limited to one and may acquire a plurality of search conditions. When a plurality of search conditions are acquired, the search target search algorithm can employ various logical operation expressions (AND, OR, NOT, NAND, NOR, EOR, XOR, etc.) or a combination of various logical operation expressions. .

  In the above display device, the acquisition unit acquires the search condition by voice recognition.

  According to the configuration of the present invention, the user can instruct search conditions by voice, which is convenient in situations where the hand cannot be used. Further, for example, in a book reading or a conference using materials, it is also convenient when it is desired to indicate a part that has been read or a sentence / paragraph to be explained as a search target.

  In the above display device, when a plurality of search target positions are specified by the specifying unit, the display unit displays a plurality of augmented reality images based on the positions of the plurality of search targets specified by the specifying unit. These are displayed in different display forms.

  According to the configuration of the present invention, since the plurality of augmented reality images are displayed in different display forms, the user can distinguish the plurality of search objects from each other.

  In the above display device, the search target is a character.

According to the structure of this invention, it is convenient when it is necessary to identify a specific character from sentences, such as a meeting using a book reading or a document, for example.
The “character” to be searched may be a single character or a plurality of characters.

  In the above display device, the specifying unit specifies a character position based on an optical character recognition result of the character.

  According to the structure of this invention, a character can be easily recognized and the position of a character can be specified using an optical character recognition technique.

  In the above display device, the subject is a printed material, and the specifying unit specifies a position of a search target printed on the printed material based on image data for printing the printed material.

  According to the configuration of the present invention, since image data for printing a printed material is used, it is not necessary to perform optical character recognition on the image. Thereby, even when the search target is small or the print quality of the printed matter is poor, the position of the search target can be specified accurately.

  In the above display device, an identification mark is printed at a predetermined position of the printed material, and the specifying unit specifies the position of the search target on the basis of the position of the identification mark printed on the printed material. Features.

  According to the configuration of the present invention, in order to specify the position of the search target with reference to the position of the identification mark printed on the printed material, the position of the character string of the printed material on which the identification mark is not printed is specified. In comparison, the position of the search target can be specified accurately.

  The display device includes a generation unit that generates image data including an image of an identification mark.

  According to the configuration of the present invention, since the image data for printing the printed matter with the identification mark is generated by the own apparatus, it is not necessary to acquire the image data from the outside.

  The display control apparatus according to the present invention includes a specifying unit that specifies a position of a search target in an image obtained by photographing a subject, and a user selects a search target as a subject based on the position of the search target specified by the specifying unit. A display control unit for displaying an augmented reality image so that the image can be identified.

According to the configuration of the present invention, since the position of the search target in the image obtained by photographing the subject is specified and the augmented reality image is displayed based on the specification result, the user can easily select the search target from the display result. Can be identified.
The display control device may have a built-in display device or may be externally attached. In the latter case, the display control device and the display device may be configured to communicate with each other via wireless communication or wired communication.

  According to the program of the present invention, the user selects a search target of the subject based on the specifying unit that specifies the position of the search target in the image obtained by photographing the subject, and the position of the search target specified by the specifying unit. It is characterized by functioning as a display control unit for displaying an augmented reality image so that it can be identified from among them.

By using this program, it is possible to realize a display device that performs display so that a user can easily identify a search target from among subjects.
Note that the display device including a computer may be physically one, or a plurality of display devices may form the display device in cooperation.

It is a top view of a portable terminal and printed matter concerning one embodiment of the present invention. It is a block diagram which shows the hardware constitutions of a portable terminal. It is a block diagram which shows the function structure of a portable terminal. It is a figure which shows an example of printed matter. It is a figure which shows the example of a display of a to-be-photographed image and an augmented reality image. It is a figure which shows the example of a display of a to-be-photographed image and an augmented reality image. It is a flowchart which shows the flow of an augmented reality display process. It is a block diagram which shows the function structure of the portable terminal which concerns on 2nd Embodiment. It is a figure which shows an example of the printed matter with the identification mark which concerns on 2nd Embodiment. It is a block diagram which shows the function structure of the display system which concerns on 3rd Embodiment.

[First Embodiment]
Hereinafter, a display device, a display control device, and a program according to an embodiment of the present invention will be described in detail with reference to the accompanying drawings. In this embodiment, the case where the display device of the present invention is applied to a mobile terminal is illustrated.

  FIG. 1 is a plan view of a portable terminal 1 and a printed material P according to an embodiment of the present invention. The printed matter P is an example of “subject”, and character strings are represented by bar lines. The mobile terminal 1 is a smartphone, for example, and includes an imaging lens 15a, a microphone 16a, and a touch panel 17 (display 17a). The figure shows a state in which the printed matter P is being photographed by the imaging lens 15 a of the mobile terminal 1. An image obtained by photographing the printed matter P is displayed on the touch panel 17. Hereinafter, the image of the printed matter P included in the image is referred to as a “subject image”.

  Note that the mobile terminal 1 is not limited to the smartphone shown in the figure, and various information processing terminals with a camera function and a display function such as smart glasses can be applied. I do.

  Next, the hardware configuration of the mobile terminal 1 will be described with reference to FIG. The mobile terminal 1 includes a central processing unit (CPU) 11, a read only memory (ROM) 12, a random access memory (RAM) 13, a terminal memory 14, a camera module 15, a microphone module 16, and a touch panel 17.

  The CPU 11 inputs and outputs signals to and from each unit in the portable terminal 1 and performs various arithmetic processes. The ROM 12 is a nonvolatile storage medium and stores an OS (Operating System) and various applications. The mobile terminal 1 of the present embodiment stores a dedicated application 50 as an application. The dedicated application 50 searches for a search target from an image obtained by photographing a subject, specifies the position of the search target, and displays an augmented reality image g (see FIGS. 5 and 6) at the specific position. The dedicated application 50 is an example of a “program”.

  The RAM 13 is a volatile storage medium and is used as a work area for the CPU 11. The terminal memory 14 is a non-volatile storage medium and stores various setting information.

  The camera module 15 includes an imaging lens 15a that forms a subject image, an imaging device that photoelectrically converts a subject image formed by the imaging lens 15a, and an A / D conversion that performs A / D conversion on an output signal output from the imaging device. And an image processing circuit for performing image processing on the A / D converted digital signal.

  The microphone module 16 includes a microphone 16a that collects (receives) a user's voice, an amplifier that amplifies the collected analog voice signal, and an A / D conversion that performs A / D conversion on the analog voice signal output from the amplifier. And an audio signal processing circuit for performing audio signal processing on the A / D converted digital audio signal.

  The touch panel 17 is an integrated display 17a and touch sensor 17b, and is used as display means and input means. The display 17a displays various information such as a captured moving image. The touch sensor 17 b detects a user operation on the touch panel 17 and outputs a detection signal to the CPU 11.

  Note that the above hardware configuration of the mobile terminal 1 is merely an example, and other hardware having the same function may be adopted. For example, instead of the touch panel 17, a display unit and an input unit may be individually mounted.

  Next, the functional configuration of the mobile terminal 1 will be described with reference to FIG. The portable terminal 1 includes a photographing unit 110, an acquisition unit 120, a specifying unit 130, and a display unit 140 as functional configurations. These functions are functions realized mainly by the CPU 11 executing the dedicated application 50. Note that the photographing unit 110 may be realized using a known camera application instead of the dedicated application 50.

  The photographing unit 110 photographs a printed matter P that is a subject, and includes the function of the camera module 15. In the present embodiment, the imaging unit 110 captures a moving image (video without sound).

  The acquisition unit 120 acquires search conditions for searching for a search target from a moving image by voice recognition, and includes the function of the microphone module 16. The acquisition unit 120 collects the user's voice, performs voice recognition based on the voice recognition algorithm, and acquires the voice recognition result as a search condition. In the present embodiment, characters include numbers and symbols, “search target” is “one or more characters”, and “search condition” is “how to read one or more characters (sound)”. That is, the specifying unit 130 to be described later searches for “one or more characters” matching the “how to read one or more characters” acquired as a search condition, and specifies the position. In the following description, “one or more characters” is referred to as a “character string”.

  Based on the search condition acquired by the acquisition unit 120, the specifying unit 130 searches for a search target from the moving images captured by the capturing unit 110, and based on the search results of the plurality of search targets, Specify the position of. That is, one or more character strings that match the character string acquired by voice recognition are searched from the moving image, and the position (position on the touch panel 17) of the one or more character strings is specified. At this time, the specifying unit 130 searches for a character string by performing optical character recognition on the moving image, and specifies the position of the character string based on the search result. The specifying unit 130 not only specifies the position of the character string, but also specifies the character size, character code, and font type of the character string. Note that the term “match” does not need to be a complete match, and may be a character string having the same meaning (including numbers, symbols, and characters replaced in another language) as necessary. For example, “1”, “1”, “1”, and “one” may all be considered to match “1”.

  The display unit 140 displays the augmented reality image g based on the position of the character string specified by the specifying unit 130, and includes the function of the touch panel 17 (display 17a). When a plurality of character strings are specified by the specifying unit 130, the display unit 140 displays each of the plurality of character strings on the subject (touch panel 17) based on the position of each of the specified character strings. A plurality of augmented reality images g are displayed so that they can be identified from among the captured subject images.

  Here, the processing of the display unit 140 will be described with reference to FIGS. FIG. 4 is a diagram illustrating an example of the printed matter P. As shown in the figure, a text is printed on the printed matter P. This document is cited from the article "Seiko Epson" updated on 09/29/2016 (Tuesday), 09:30, in the Japanese Wikipedia: Free Encyclopedia. Further, when the printed matter P is photographed by the photographing unit 110, the display unit 140 displays a moving image S including a subject as shown in FIG.

  An example will be described in which the user speaks “Seiko Instruments” to the microphone module 16 while the printed part P is photographed by the photographing unit 110. First, when the user speaks up to “Seiko”, the acquisition unit 120 acquires how to read the character string “Seiko” as a search condition (the character string “Seiko” is recognized by voice). Then, the specifying unit 130 recognizes characters printed on the printed matter P by optical character recognition from the moving image S obtained by photographing the printed matter P by the photographing unit 110, and reads a character string whose reading is “Seiko” (in the example of FIG. The position of the character string “Seiko” and the character string “Seiko”) is specified. Then, the display unit 140 displays the augmented reality image g with respect to the character string at the specified position in the moving image S in which the photographing unit 110 has photographed the printed matter P (see FIG. 5). In the example of FIG. 5, since six search objects are searched, six augmented reality images g1 to g6 are displayed. In the present embodiment, as the augmented reality image g, an image in which a character string to be searched is bolded and an underline image added to the character string to be searched are displayed. The former “image in which a character string is bolded” is an image generated according to the character size, character code, and font type specified by the specifying unit 130. As described above, since the character string read out by voice is highlighted by the bold character and the underline, the user can easily grasp which part of the printed matter P has been read out. In the present embodiment, since the moving image S is used, the position where the photographing unit 110 is photographing can vary. For this reason, even if the position where the image capturing unit 110 captures changes, the change follows the change and the augmented reality image g is always displayed on the character string to be searched.

  Next, as shown in FIG. 5, in a state where the moving image S and the augmented reality images g1 to g6 are displayed, the user speaks “instrument” so that the acquisition unit 120 can generate a character string following “Seiko”. How to read the character string “instrument” is acquired as a search condition (when the character string “instrument” is recognized by voice). The specifying unit 130 specifies a character string whose reading is “instrument”. Next, the display unit 140 leaves the augmented reality image g4 existing before the character string whose reading is “instrument” among the augmented reality images g1 to g6 that have been displayed so far, while the reading is “instrument”. A new augmented reality image g11 is displayed for the character string (see FIG. 6). Further, the display unit 140 hides the other augmented reality images g1, g2, g3, g5, and g6 that have been displayed until the character string “instrument” is acquired.

  In this way, the mobile terminal 1 (dedicated application 50) reads out the character strings sequentially and updates the display of the augmented reality image g each time the search condition is acquired. However, when the user performs an operation to end the augmented reality display process, or when acquisition of the search condition is interrupted for a long time (for example, 3 minutes), the augmented reality display process is temporarily terminated (displayed until then). All the augmented reality images g are not displayed). When the character string is read again, the augmented reality image g is displayed on the newly read character string.

  In addition, when acquisition of search conditions is interrupted for a certain period of time (longer than the time for recognizing a series of character strings and shorter than the time for temporarily ending augmented reality display processing, for example, 5 seconds), another search target It is good also as a character string. In the example of FIGS. 5 and 6, as the augmented reality image g, an image in which a character string to be searched is bolded and an underline image added to the character string to be searched are displayed. Therefore, if the character string that is the target of the augmented reality image g is continuous, the underlined image appears to be continuous, but the underlined image does not continue to another character string to be searched. Different augmented reality images g may be displayed. For example, the type of underline may be changed for each character string to be searched.

  The augmented reality image g is not limited to the examples shown in FIGS. For example, an image obtained by decorating a character string to be searched with a character decoration other than bold (such as an image in which a character color, a character size, or a character font is changed) may be superimposed and displayed. In the case of superimposing display, only the image line portion of the character may be superimposed, or it may be superimposed including the base portion as if a label is attached. In the former case, the base color of the subject image (the color of the print medium of the printed matter P) can be used. In the latter case, even if the position of the character string to be searched for the subject image and the display position of the augmented reality image g are slightly shifted, the characters printed on the subject image and the bold characters of the augmented reality image g Visibility is not impaired because the characters do not look double. In addition, a marker having transparency may be displayed superimposed on the character string to be searched. Further, a moving image may be displayed as the augmented reality image g. For example, a moving image in which a character string to be searched rises, a moving image in which an image line portion of the character string to be searched vibrates, and the like can be considered.

  Next, the flow of augmented reality display processing by the mobile terminal 1 will be described with reference to the flowchart of FIG. Here, it is assumed that the dedicated application 50 is activated, the subject (printed matter P) is photographed by the photographing unit 110, and the display unit 140 displays the moving image S including the subject. This flowchart is repeatedly executed until the dedicated application 50 is terminated.

  In this state, when the acquisition unit 120 newly acquires a search condition or there is a search condition that has already been acquired (S01: Yes), optical character recognition is performed on the newly acquired moving image S to obtain a moving image. A character string is recognized from the target frame in S (S02). Then, the specifying unit 130 searches for a search target from the target frame in the moving image S based on the acquired or already acquired search condition, and determines whether there is a search target (S03). That is, it is determined whether or not the character string acquired as a search condition (character string recognized by speech) is included in the optical character recognition result of the target frame. If there is no search condition already acquired without acquiring the search condition (S01: No), and if it is determined that there is no search target (S03: No), the process returns to S01. In S01, the search condition may be acquired every time a syllable or word is recognized (for each syllable or for each word), or the search condition may be acquired when the sound is interrupted for a short time.

  When determining “Yes” in S03, the specifying unit 130 specifies the position to be searched (S04). That is, it is specified in which position of the target frame the character string newly acquired as the search condition or the character string already acquired as the search condition exists. If a plurality of search targets are found in S03, the positions of the plurality of search targets are specified in S04. Thereafter, the display unit 140 determines whether or not the character string searched in S03 includes a continuous character string (S05). Here, the “continuous character string” refers to a character string that is newly continuous after the character string on which the augmented reality image g is already superimposed and displayed. For example, in the example shown in FIG. 6, the character string “instrument” is a continuous character string. The determination in S05 is “No” when there is no character string on which the augmented reality image g is superimposed and displayed, and therefore, “No” is always the first time after the augmented reality display process starts. Further, even when a new search condition is not acquired (when the search condition is not updated), since there is no new character string, the determination in S05 is “No”.

  If the display unit 140 determines “No” in S05, the display unit 140 expands the search target character string so that the user can identify the search target on the target frame based on the position of the search target specified in S04. The target frame is displayed by superimposing the real image g (S06). In S06, when there is an old augmented reality image that is the augmented reality image g that has been displayed so far, the augmented reality image g having a display form different from that of the old augmented reality image is superimposed and displayed.

  On the other hand, if the display unit 140 determines “Yes” in S05, a new augmented reality image is generated for the continuous character string among the character strings searched in S03 based on the position of the search target specified in S04. The new augmented reality image which is g is superimposed and the target frame is displayed (S07). That is, in S07, the new augmented reality image is not displayed for character strings that do not correspond to the continuous character string among the character strings searched in S03. In S07, the old augmented reality image existing before the continuous character string in the old augmented reality image is left and the other augmented reality image g is not displayed. That is, in the example illustrated in FIG. 6, the augmented reality image g4 is “an old augmented reality image existing before a continuous character string”, and the augmented reality image g11 is a “new augmented reality image”.

  Then, after displaying in S06 and S07, it is determined whether or not acquisition of a new search condition has been interrupted for a long time (S08). If “No” is determined in S08, that is, if there is no interruption for a long time, the next frame of the moving image S is set as the target frame (S09), and the process returns to S01. In S08, whether or not the acquisition of a new search condition has been interrupted for a long time is determined by measuring the elapsed time from the acquisition of the last search condition and determining that the acquisition has been interrupted when the elapsed time reaches a predetermined time. . In other words, the determination is not based on the elapsed time from the point of determination of S01: No. Further, the “new search condition” does not indicate “a search condition different from the previous search condition” but indicates a “newly recognized search condition”. That is, when the same sound as “Seiko” and “Seiko” is continuously recognized, the second “Seiko” becomes a “new search condition”. On the other hand, if “Yes” is determined in S08, that is, if acquisition of a new search condition is interrupted for a long time, the already acquired search condition is deleted and all augmented reality images g are not displayed ( S10). Thereafter, the next frame of the moving image S is set as a target frame (S09), and the process returns to S01.

  As described above, according to the mobile terminal 1 of the present embodiment, the position of the character string that matches the character string acquired as the search condition is identified from the moving image obtained by photographing the printed matter P that is the subject, and the identification is performed. Since the plurality of augmented reality images g are displayed based on the result, the user can easily identify the plurality of character strings to be searched from the subject images. Further, since the mobile terminal 1 acquires the search condition by voice recognition, the user can instruct the search condition even in a situation where the hand cannot be used. Furthermore, since the mobile terminal 1 according to the present embodiment executes the augmented reality display process shown in FIG. 7 in units of frames while shooting a moving image, the position of the character string to be searched is once specified. Even if the mobile terminal 1 moves without acquiring new search conditions, the augmented reality image g can be continuously displayed for the character string to be searched.

  Note that “display of a plurality of augmented reality images g” here refers to displaying the augmented reality images g1 to g6 shown in FIG. 5 and displaying the augmented reality images g4 and g11 shown in FIG. , Including both. That is, the search condition is not necessarily acquired once, and may be acquired in a plurality of times. In the above embodiment, the augmented reality images g displayed based on the search conditions acquired for the first time are the augmented reality images g1 to g6, and the augmented reality images displayed based on the search conditions acquired for the first time and the second time. g is augmented reality images g4 and g11.

In addition, the following modifications can be employed.
[Modification 1]
In the above embodiment, the display unit 140 may display the augmented reality image g in a display form different from the above. Further, although the plurality of augmented reality images g corresponding to the plurality of search targets are all displayed in the same display form, they may be displayed in different display forms. “Different display forms” include different types of decoration and decoration methods. For example, a plurality of augmented reality images g may be displayed in different colors, or different additional information (numbers 1 and 2 or the like, single underline and wave underline, round circle and cloud circle, etc.) may be added. May be displayed. Also, a plurality of augmented reality images g may be blinked and displayed at different periods. Further, the display form may be changed according to the character type (katakana, hiragana, kanji, alphabet, etc.) of the character string and the character size. For example, an augmented reality image g of a yellow marker is displayed for Katakana, and an augmented reality image g of a red marker is displayed for an alphabet.

[Modification 2]
In the above-described embodiment, the specifying unit 130 specifies a plurality of search target positions, but may not necessarily be a plurality, and may specify a single search target position. As a further modification, the position of the search target is specified when there are a plurality of search targets, and the position of the search target is not specified when there are no plurality of search targets (when there is only one search target). (In other words, the augmented reality image g is not displayed). According to this configuration, useless display control can be omitted when the necessity for displaying the augmented reality image g is low, such as when the number of characters printed on the printed matter P is small.

[Modification 3]
In the above embodiment, the imaging unit 110 captures a moving image, but may capture a still image. In this case, the augmented reality display process is executed in a state where a still image is displayed on the touch panel 17.

[Modification 4]
In the above embodiment, the specifying unit 130 searches for a character string as a search target. However, an image other than the character string may be set as a search target. Moreover, although the acquisition part 120 acquired the character string as a search condition, if the search object can be specified, you may acquire information other than a character string. For example, when the search target is a character string, the character type or character color may be acquired as a search condition. In addition, when the search target is an image other than a character string, image characteristics (color, size, brightness, shape, etc.) may be acquired as a search condition.

[Modification 5]
In the above embodiment, the acquisition unit 120 may acquire one search condition or a plurality of search conditions. When acquiring a plurality of search conditions, the search target search algorithm can employ various logical operation expressions (AND, OR, NOT, NAND, NOR, EOR, XOR, etc.) or a combination of various logical operation expressions. For example, in the case where the subject image shown in FIG. 5 is displayed, the search condition 1 is “Seiko” as the reading method of the character string, the search condition 2 is “alphabet” as the character type, and the logical operation expression adopted is “AND”. ", The specifying unit 130 may specify the position of the character string" Seiko "that satisfies these two conditions. That is, the display unit 140 may display the augmented reality image g2 and the augmented reality image g6.

[Modification 6]
In the above embodiment, the display unit 140 includes one or more character strings corresponding to the search condition, and if the one or more character strings do not include a continuous character string, the display unit 140 determines the old augmented reality for the character string to be searched. Although the augmented reality image g different from the image is superimposed and displayed (see S06 in FIG. 7), the old augmented reality image may be hidden. On the other hand, when there is one or more character strings corresponding to the search condition and the one or more character strings include a continuous character string, only the old augmented reality image before the continuous character string is left among the old augmented reality images, and the new Although the augmented reality image is superimposed and displayed (see S07 in FIG. 7), all the old augmented reality images may be left. In S07 of FIG. 7, the new augmented reality image is superimposed and displayed only on the continuous character string. However, the new augmented reality image may be superimposed and displayed on all the character strings searched in S03. When determining whether or not there is a search target in S03 of FIG. 7, even if the read-out character string does not completely match the character string in the sentence, if the consistency is equal to or higher than a predetermined ratio, “search target “Yes” may be determined.

[Modification 7]
In the above embodiment, the display unit 140 displays the augmented reality image g for the character string specified by the specifying unit 130, but the augmented reality for the predetermined range including the character string specified by the specifying unit 130. The image g may be displayed. For example, it is conceivable to display the augmented reality image g for a predetermined area centered on a character string specified by the specifying unit 130, a sentence or a paragraph including the specified character string, and the like.

[Modification 8]
In the above embodiment, the specifying unit 130 specifies the position on the moving image as the position to be searched, but may specify the position on the subject image in the moving image. In this case, the specifying unit 130 determines the search target on the moving image based on the position of the search target on the specified subject image and the position of the subject image in the moving image displayed on the touch panel 17. Identify the location.

[Modification 9]
In the above embodiment, a moving image is displayed on the touch panel 17 of the mobile terminal 1, but the moving image may be displayed on a screen having transparency (translucency). In this case, the display unit 140 does not need to display the subject image, and may display the augmented reality image g in accordance with the position of the subject visible to the user through the screen. In this case, since it is assumed that the shooting range of the shooting unit 110 is included in the user's visual field, the user's head position, face orientation, eyeball movement, and the like are detected by a sensor or the like, and the detection results thereof Based on the above, the visual field of the user viewing the subject may be specified. Then, the plurality of augmented reality images g may be displayed so that the user can identify each of the plurality of search targets from the subjects included in the field of view (subjects that can be seen through the transparent screen). As a display device provided with a screen having transparency, for example, laser projection type car navigation, see-through projection, transmissive digital signage, and the like are conceivable. Further, a head mounted display may be adopted as a display device having a transmissive screen. In this case, since the imaging range by the imaging unit 110 changes according to the movement of the user's head, detection of the user's head position and face orientation may be omitted. As a further modification, the augmented reality image g may be projected toward a screen or a wall by a document camera or a projector.

[Second Embodiment]
Next, a second embodiment of the present invention will be described with reference to FIGS. In the present embodiment, the specifying unit 130 uses image data for printing the printed material PM, and specifies the position to be searched based on the position of the identification mark M printed on the printed material PM (see FIG. 9). This is different from the first embodiment. Only differences from the first embodiment will be described below. In the present embodiment, the same components as those in the first embodiment are denoted by the same reference numerals, and detailed description thereof is omitted. Moreover, the modification applied about the component similar to 1st Embodiment is applied similarly about this embodiment.

  Since the hardware configuration of the mobile terminal 1 ′ can be the same as that of the first embodiment (see FIG. 2), description thereof is omitted. FIG. 8 is a block diagram illustrating a functional configuration of the mobile terminal 1 ′ according to the second embodiment. The mobile terminal 1 ′ according to the present embodiment has a configuration in which a generation unit 150 is added to the functional configuration of the mobile terminal 1 ′ in FIG. 1 (see FIG. 3).

  The generation unit 150 generates image data for printing the printed matter PM, and is realized mainly by the CPU 11 executing the dedicated application 50. The file format of the image data to be generated (JPEG, BMP, etc.) does not matter. The generation unit 150 also prints a second mark for printing the identification mark M on the first image data for printing document data that is information input to the portable terminal 1 ′ or information acquired from the outside. By adding the image data, image data for printing the printed matter PM is generated. Then, the image data is converted as necessary or control information is added to generate print data. If not necessary, the image data may be used as print data as it is. The generation unit 150 stores related information in which the document data is associated with the identification mark M (character string for generating the identification mark M) in a predetermined storage area in the terminal memory 14.

  FIG. 9 is a diagram illustrating an example of the printed material PM with the identification mark M. In the present embodiment, barcodes are printed as the identification marks M1 to M4. The barcode includes a printed matter code for specifying the printed matter PM and information indicating the printing position of each barcode (upper left, upper right, lower left, lower right, etc. of the paper). Note that the identification mark M further includes information indicating a print number and a page number when a plurality of pages are printed by one printing. Alternatively, a barcode including a different printed matter code may be printed on each page. Thereby, when a plurality of pages are printed by one printing, the page can be specified. In the example of FIG. 9, the identification marks M are printed at the four corners of the printed matter PM. However, the identification marks M may be printed at one place or more places.

  Returning to the description of FIG. The mobile terminal 1 ′ of the present embodiment is configured to be able to communicate with the printing apparatus 2 directly or indirectly. Although not particularly illustrated, the mobile terminal 1 ′ and the printing apparatus 2 are provided with a communication unit for communicating with each other. The printing apparatus 2 includes a printing unit 210. The printing unit 210 performs printing based on print data transmitted from the mobile terminal 1 ′, and includes a print head, a paper transport mechanism, and the like. Further, the printed matter PM is created by printing by the printing unit 210.

  With the above configuration, the specifying unit 130 ′ according to the present embodiment specifies the position of the search target (character string) printed on the printed matter PM based on the document data. Alternatively, the specifying unit 130 ′ specifies the position to be searched based on the position of the identification mark M based on the identification mark M printed on the printed matter PM and the document data. That is, the specifying unit 130 ′ refers to a predetermined storage area in the terminal memory 14, and reads the document data associated with the identification mark M from the identification mark M of the subject image included in the moving image captured by the imaging unit 110. Identify. Specifically, the barcode information is read using the barcode reader function, and the barcode data is read and decoded to identify the identification mark M, and the document data associated with the identified identification mark M is identified. To do. Further, the specifying unit 130 ′ searches the character code of the character string acquired as the search condition from the character code of the specified document data, and the position of the character string indicated by the search result and the position of the identification mark M And the position of the character string with the identification mark M as a reference. For example, the front end position of the identification mark M1 is coordinates (0, Y), the rear end position of the identification mark M2 is coordinates (X, Y), the front end position of the identification mark M3 is coordinates (0, 0), and the rear end of the identification mark M4. When the end position is the coordinate (X, 0), the leading end position and the trailing end position of the character string to be searched are specified. Then, based on the position of the character string in the image data (position of the character string with reference to the identification mark M) and the position of the identification mark M of the moving image S displayed on the touch panel 17, the moving image The position of the character string on S (that is, the position where the augmented reality image g is to be displayed) is specified. If the document data does not include a character code, the image of the document data may be searched using optical character recognition.

  On the other hand, the display unit 140 according to the present embodiment displays the augmented reality image g based on the position of the character string specified by the specifying unit 130 ′.

  As described above, according to the mobile terminal 1 ′ of the second embodiment, a moving image that is taken to identify the position of the search target printed on the printed matter PM using the image data for printing the printed matter PM. There is no need to perform optical character recognition on the image. As a result, even if it is difficult to perform optical character recognition on the captured image, such as when the characters in the captured image are small or the print quality of the printed matter PM is poor, the position of the search target can be accurately determined. Can be specified. In addition, since the position of the search target is specified based on the position of the identification mark M printed on the printed matter PM, the position of the search target is accurately compared with a configuration that does not use the position of the identification mark M as a reference. Can be identified. Further, since the mobile terminal 1 ′ itself generates image data, it is not necessary to acquire image data from the outside.

In the second embodiment, the following modifications can be employed.
[Modification 10]
In the above-described embodiment, the generation unit 150 generates image data including the image of the identification mark M, but may generate image data that does not include the image of the identification mark M. When the image of the identification mark M is not included, the search target position may be specified based on the positions of the four corners of the subject image or the top, bottom, left, and right edges of the paper.

[Modification 11]
In the above embodiment, a barcode is exemplified as the identification mark M, but other code images such as a two-dimensional code may be used. In addition to the code image, a predetermined symbol or mark may be used.

[Modification 12]
In the above embodiment, the print data is generated from the document data by the generation unit 150 of the mobile terminal 1 ′. However, a print data generation device different from the mobile terminal 1 ′ generates the print data from the document data, and the mobile terminal 1 ′. May be configured not to generate print data. In this case, the specifying unit 130 ′ acquires the document data and the related information from the print data generation apparatus, and specifies the position of the character string to be searched using the acquired document data and the related information. The print data generation device may be a personal computer, or the printing device 2 may also serve as the print data generation device.

[Third Embodiment]
Next, a third embodiment of the present invention will be described with reference to FIG. This embodiment implement | achieves the portable terminal 1 shown in 1st Embodiment by the combination of a some apparatus. Only differences from the first embodiment will be described below. In the present embodiment, the same components as those in the first embodiment are denoted by the same reference numerals, and detailed description thereof is omitted. Moreover, the modification applied about the component similar to 1st Embodiment is applied similarly about this embodiment.

  FIG. 10 is a block diagram illustrating a functional configuration of the display system SY according to the third embodiment. The display system SY includes a display control device 3, a photographing device 4, and a display device 5. The display control device 3 and the imaging device 4, and the display control device 3 and the display device 5 are configured to be communicable. The imaging device 4 includes an imaging unit 410 having the same function as the imaging unit 110 of the first embodiment. The moving image data photographed by the photographing device 4 is transmitted to the display control device 3.

  On the other hand, the display control device 3 includes an acquisition unit 120, a specification unit 130, and a display control unit 160. The display control unit 160 is a function that is realized mainly by the CPU 11 executing the dedicated application 50, and corresponds to a function that controls display control of the display unit 140 of the first embodiment. The display control unit 160 generates display data for displaying the augmented reality image g and transmits the display data to the display device 5. The display device 5 includes a display unit 510 that displays the display data transmitted from the display control device 3. The display unit 510 is a non-transmissive or transmissive display.

  As described above, according to the third embodiment, even if the display control device 3 has a simple configuration, the same effect as that of the first embodiment is obtained by cooperating with the photographing device 4 and the display device 5. be able to.

In the third embodiment, the following modifications can be employed.
[Modification 13]
This embodiment may be combined with the second embodiment. That is, the configuration may be such that the generation unit 150 of the second embodiment is added to the display control device 3 and the specifying unit 130 is replaced with the specifying unit 130 ′.
[Modification 14]
You may implement | achieve portable terminal 1,1 'with the combination of the several apparatus different from this embodiment.

In addition, the following modifications can be adopted in common with the first to third embodiments.
[Modification 15]
You may provide each component in the portable terminal 1, 1 'shown in said embodiment and the modification, or the display control apparatus 3 as a program. The program may be provided by being stored in various recording media (CD-ROM, flash memory, etc.). That is, a program for causing a computer to function as each component of the portable terminal 1, 1 'or the display control device 3, a computer-readable recording medium on which the program is recorded, and a product (printed material PM or the like) created thereby Is also included in the scope of rights of the present invention. Other modifications can be made as appropriate without departing from the scope of the present invention.

  DESCRIPTION OF SYMBOLS 1 ... Portable terminal, 110 ... Image pick-up part, 120 ... Acquisition part, 130 ... Identification part, 140 ... Display part

Claims (11)

A shooting section for shooting the subject;
A specifying unit that specifies the positions of a plurality of search targets that are reflected in the moving image shot by the shooting unit;
A display unit that displays a plurality of augmented reality images so that the user can identify each of the plurality of search targets from among the subjects based on the positions of the plurality of search targets specified by the specifying unit. A display device comprising: An acquisition unit for acquiring a search condition for searching the plurality of search objects from the moving image;
The specifying unit searches a plurality of search objects from the moving images based on the search condition, and specifies a plurality of positions of the search objects based on a plurality of search object search results. The display device according to claim 1.   The display device according to claim 2, wherein the acquisition unit acquires the search condition by voice recognition.   The display device according to claim 1, wherein the display unit displays the plurality of augmented reality images in different display forms.   The display device according to claim 1, wherein the search target is a character.   The display device according to claim 5, wherein the specifying unit specifies a position of the character based on an optical character recognition result of the character. The subject is a printed matter,
The said specific | specification part specifies the position of the said search object printed on the said printed matter based on the image data for printing the said printed matter, The one of Claim 1 thru | or 5 characterized by the above-mentioned. Display device. An identification mark is printed at a predetermined position of the printed matter,
The display device according to claim 7, wherein the specifying unit specifies the position of the search target based on a position of the identification mark printed on the printed matter.   The display device according to claim 8, further comprising a generation unit that generates the image data including an image of the identification mark. A specifying unit for specifying the positions of a plurality of search targets in a moving image obtained by shooting a subject;
Display control for displaying a plurality of augmented reality images so that the user can identify each of the plurality of search targets from among the subjects based on the positions of the plurality of search targets specified by the specifying unit. A display control device. Computer
A specifying unit for specifying the positions of a plurality of search targets in a moving image obtained by shooting a subject;
A display control unit that displays an augmented reality image so that a user can identify each of the plurality of search targets from among the subjects based on the positions of the plurality of search targets specified by the specifying unit; , A program to make it function as.

Images