JP2013243692A - Display device, display method and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To acquire and display additional information concerned with an image object in a photographed image.SOLUTION: A display device comprises: recognition means 203 for detecting an image object displayed on a display screen on the basis of object recognition information; recognition mark drawing means 205 for drawing a recognition mark corresponding to each detected image object by the size corresponding to the display size of each detected image object; image object retrieval means 207 which, when the recognition mark is drawn in a preset display determination area in the display screen by the size of a rate of a fixed value or more to the size of the display determination area, retrieves object information concerned with the image object corresponding to the recognition mark through a communication line; and image display means 210 for displaying the retrieved and acquired object information on the display screen.

Description

  The present invention relates to a technique for acquiring object information indicating additional information related to a video object included in a captured video.

A camera that captures the object to be photographed and a display screen for displaying the photographed image and video are also identified, and the video object such as text information is identified and identified from the image and video displayed on the display screen. A method for acquiring additional information related to a video object is disclosed (Patent Document 1).
Further, as a related technique for this, a method for displaying the acquired additional information of the video object in a superimposed manner on the same display screen as the video object is disclosed (Patent Document 2).
Furthermore, as a related technique for this, a method for recognizing not only text but also general objects from a photographed image is disclosed (Patent Document 3).

JP 2006-60535 A Special table 2008-510254 gazette JP 2001-307096 A

However, the related technique disclosed in Patent Document 1 is a technique for substituting an operation of performing a text search by key input, and superimposing additional information on a video object in an actually captured image. There is an inconvenience that it cannot be displayed.
In the related art disclosed in Patent Document 2, when additional information is automatically overlaid on the display screen, additional information that does not reflect the user's intention may be displayed. Inconveniences such as hindering visibility may occur.

Further, in the related art disclosed in Patent Document 3, when a plurality of different video objects are displayed on the screen, additional information for which object (video object) is displayed on the display screen. It becomes difficult to determine whether.
In addition, for example, in order to specify a video object on which additional information is displayed, an operation button or the like is provided, and when the user directly selects a specific video object on the screen, the operation becomes complicated. There is an inconvenience.
In this case, for example, the intuitive operability is greatly impaired as compared with a normal operation with a magnifying glass or a loupe, and the user can easily and intuitively operate the additional information of the video object. There is a disadvantage that it cannot be displayed.

[Object of invention]
The present invention improves the inconvenience of the related art, and obtains and displays object information, which is additional information related to a video object photographed and output by a user, by an intuitive and simple operation, The object is to provide a display method and a program.

  In order to achieve the above object, a display device according to the present invention is a display device that captures an image of an object and displays the captured image on a display screen, the image object displayed on the display screen. Video object detection means for detecting the video object based on preset object recognition information, and when the video object is detected, a recognition mark corresponding to the detected video object is displayed on the display screen. A recognition mark drawing means for drawing, and the video corresponding to the recognition mark when the recognition mark is drawn in a preset display determination area in the display screen so as to satisfy a preset condition. A configuration is provided that includes video object search means for searching for object information relating to the object, and image display means for displaying the object information acquired by the search on the display screen.

  Further, the display method according to the present invention is a display method by a display device that captures an image of an object and displays the captured image on a display screen, and the image object displayed in the display screen includes: Detecting based on preset object recognition information; and, when the video object is detected, rendering a recognition mark corresponding to each detected video object on the display screen; The object information relating to the video object corresponding to the recognition mark when the recognition mark is drawn in a predetermined display determination area in the display screen so as to satisfy a predetermined condition. And the display of the object information acquired by the search on the display screen.

  Furthermore, the program according to the present invention is a program for causing a display device that captures an image of an object and displays the captured image on a display screen to execute the image object displayed in the display screen. Detecting based on preset object recognition information; and, when the video object is detected, rendering a recognition mark corresponding to each detected video object on the display screen; The object information relating to the video object corresponding to the recognition mark when the recognition mark is drawn in a predetermined display determination area in the display screen so as to satisfy a predetermined condition. And the display of the object information acquired by the search on the display screen.

  Since the present invention is configured and functions as described above, according to this, means for drawing a recognition mark in the display screen indicating that a video object in the display screen has been detected, and the recognition mark in the display screen. By having a configuration including means for acquiring additional information related to the video object, that is, object information when the image is rendered in a predetermined display determination area while satisfying a predetermined condition, It is possible to provide a display device, a display method, and a program that can easily acquire and display object information related to a video object that has been captured and displayed.

It is a schematic block diagram which shows one Embodiment of the external appearance of the display apparatus by this invention. It is a schematic block diagram which shows one Embodiment of the internal structure of the display apparatus disclosed in FIG. 2 is a sequence chart illustrating an overall operation of the display device disclosed in FIG. 1. It is explanatory drawing which shows an example of the imaging | photography state in the target object (confectionery package) in the display apparatus disclosed in FIG. It is explanatory drawing which shows an example of the method of drawing the recognition mark in the target object (confectionery package) in the display apparatus disclosed in FIG. It is explanatory drawing which shows an example of the display method of the target object (confectionery package) information in the display apparatus disclosed in FIG. It is explanatory drawing which shows an example of the specific method of the target object (beetle) in the display apparatus disclosed in FIG. It is explanatory drawing which shows an example of the display method of the target object (beetle) information in the display apparatus disclosed in FIG.

[Embodiment]
Next, the basic configuration content of the embodiment of the present invention will be described.

  The display device 100 according to the present embodiment has a housing shape as shown in FIGS. 1A and 1B, and a photographing object (object) is provided on the rear surface of the exterior part (FIG. 1B). 1 is a display screen that includes a camera lens 101 (FIG. 1B) for shooting the image, and an image captured through the camera lens 101 is output and displayed on the exterior surface (FIG. 1A) side. 102 is provided.

Further, the display device 100 includes a zoom dial 103 as an interface for controlling enlargement photographing and reduction photographing by the camera lens 101 on the side surface of the exterior portion.
Thus, the user of the display device 100 can adjust the enlarged / reduced display of the photographed object output and displayed on the display screen 102 by operating the zoom dial 103.

Next, the internal configuration of the display device 100 will be described in detail based on the block diagram of FIG.
The display device 100 controls the zoom photographing by detecting an operation (input) by the zoom dial 103 and notifying the photographing means (camera unit) 202 of the detected operation content (enlargement / reduction operation information) of the zoom dial 103. The zoom control unit 201, the imaging unit 202 that captures an image of an object through the camera lens 101, and controls the enlargement / reduction in the camera lens 101 based on the enlargement / reduction operation information notified from the zoom control unit 201, A video object detection unit 203 is provided for detecting a video object such as text information or a specific substance, which is an object in the video imaged by 202.
The video object detection means 203 has an object recognition information holding function for storing and holding object recognition information as reference information for detecting the object.

  In addition, when the video object is detected by the video object detection unit 203, the display device 100 sets a recognition mark corresponding to each detected video object, and the drawing position and size of the recognition mark. A recognition mark drawing size control means 204 for determining the detected mark according to the detected display size of each video object, and a recognition mark drawing means 205 for drawing the determined recognition mark on the display screen 102.

  Further, the display device 100 has a recognition mark drawn by the recognition mark drawing means 205 in a predetermined display determination area in the display screen 102 that has a ratio equal to or larger than a certain value with respect to the size of the display determination area. When the overlap display permission determination unit 206 for determining whether or not the image is drawn and the recognition mark in the display determination area are drawn at a ratio of a certain value or more, the display device 100 is provided in advance. Via a communication means (network line) 208, a video object search means 207 for searching for object information as information relating to a video object corresponding to the drawn recognition mark, and the video object search means 207 The object information output control means 209 for transmitting the retrieved object information to the image display means 210 is provided.

  Note that the overlapping display availability determination unit 206, the video object search unit 207, and the object information output control unit 209 correspond to an object information acquisition unit.

The photographing unit 202 photographs a photographing object such as a text (character string) or a target object to which the camera lens 101 is directed by the user.
Note that the user of the display device 100 moves the zoom control unit 201 close to or away from the target object so that the photographing target object is displayed in an appropriate size on the display screen 102. It is assumed that zooming or the like can be performed by the operation, and an object to be photographed can be displayed in the display screen 102.

The video object detection means 203 stores and holds, as a recognizable object list, what the video object detection means 203 of the display device 100 determines as “recognizable” among the captured images displayed on the display screen 102. Recognizable object list storage means is provided.
This recognizable object list storage means is realized by a storage device such as an HDD (Hard Disk Drive) installed in the display device 100 in advance.

The video object detection means 203 is a video object that is detected based on the recognizable object list among the video objects (including text information, specific substances, organisms, etc.) displayed on the display screen 102. It has a recognition object identification function that identifies
The recognition object specifying function is set to use an image recognition technique (for example, Japanese Patent Laid-Open No. 2001-307096) for recognizing not only text but also a general object from a captured image (video). It is good.
Accordingly, in the present embodiment, a specific video object (such as an object or text) can be extracted from the captured image by image recognition technology, and marking can be performed on the video object.

  When the video object detection unit 203 identifies a video object such as a text or an object (target object) displayed on the display screen 102 as a recognizable object registered in the recognizable object list in advance, the video object detection unit 203 The recognition mark drawing size control means 204 is notified that the object has been recognized.

  Note that, as a method for specifying a video object from a captured image, for example, a setting using an image recognition method disclosed in Japanese Patent Laid-Open No. 2001-307096 may be used.

  The recognition mark drawing size control means 204 sets a star mark (corresponding to a recognition mark) corresponding to the video object specified as recognizable (recognition mark setting function). At this time, the recognition mark drawing size control means 204 determines the display position and display size of the star mark in the display screen 102.

  Further, the recognition mark drawing size control means 204 determines the star mark in the display screen 102 from the display position of the video object at a certain level in order to clarify the correspondence between the set star mark and the specified video object. It shall be drawn at a position (area) within the range.

  Specifically, as shown in FIG. 7, the recognition mark drawing size control means 204 draws so that a star mark is displayed on the upper left in the vicinity of the beetle recognized by the video object detection means 203 as a video object. Set the position.

The recognition mark drawing means 205 outputs and displays a star mark on the display screen 102 via the image display means 210 based on the information indicating the size and position of the recognition mark (star mark) set by the recognition mark drawing size control means 204. It has a recognition mark output display control function.
It is assumed that the size of the star mark output and displayed is enlarged and reduced in accordance with the zoom operation (enlargement and reduction operation) of the zoom dial 103.

When the star mark is drawn by the recognition mark drawing means 205, the overlapping display availability determination means 206 starts monitoring the display position and size (recognition mark position size monitoring function).
At this time, the overlapping display availability determination unit 206 determines whether or not a star mark is displayed in a display determination area (here, the upper left portion in the display screen 102) set in the display screen 102 (recognition). Mark display area judgment function).

Further, the overlay display availability determination unit 206 determines that “overlapping display is possible” when one of the stars is displayed so as to fall within the display determination area with a certain size or more.
On the other hand, when a star mark is not displayed in the display determination area, the overlap display permission determination unit 206 determines that the state of the display screen 102 is a state of “not overlapped display”. At this time, it is assumed that a star mark indicating that the photographing object can be specified (recognized) is still displayed on the display screen 102.

  For example, FIG. 7 illustrates a state where a star mark corresponding to a beetle that is a video object is displayed in the display determination area in the display screen 102. Here, since the display size of the displayed star mark is smaller than a preset ratio, the object information search function described later is not activated.

  The video object search means 207 corresponds to the star mark displayed in the display determination area when the overlap display possibility determination means 206 determines that “overlapping display is possible” based on the drawing position of the recognition mark (star mark). An object information search function for starting a search for additional information (object information) related to a video object is provided.

  The video object search means 207 executes this object information search function to search the web information stored on the server connected to the Internet via the communication means 208 and to obtain the shooting object (video Additional information related to the (object) is acquired as a search result.

The object information output control means 209 determines whether or not a star mark is displayed in a predetermined display determination area in the display screen 102 with a predetermined size or more.
As a result, for example, a video object unintended by the user is recognized and the object information is displayed, so that a part of the screen is covered, and as a result, it is difficult to see the captured screen. Can be reduced.

Further, the object information output control means 209 displays the search result (object information) by the image object search means 207 when the star mark is displayed in a display determination area with a certain size or larger. An output is displayed on the display screen 102 via 210 (overlapping information output display function).
Thereby, the user can control the display and non-display of the additional information of the video object that suits his / her intention by a simple operation.

  Specifically, as shown in FIG. 8, when a star mark is displayed in a size larger than a certain size in the display determination area, the image display unit 210 displays attribute information (“beetle beetle”) as a search result. A large insect, adult in summer ... ") is superimposed on the captured image (here, in the form of a jet) and displayed.

Thereby, for example, even when the user of the display device 100 does not know what the object (beetle in this case) displayed on the display screen 102 is, information on the object is obtained by photographing with the display device 100. Is displayed on the display screen 102, and whether or not to display additional information (object information) on the photographed object can be determined by an operation similar to that of a normal magnifying glass.
The image display unit 210 controls the operation in which the recognition mark drawing unit 205 outputs and displays the recognition mark (star mark) on the display screen 102, and the object information output control unit 209 displays the additional information as a search result. An output display control function for controlling an operation of outputting and displaying on the display screen 102 is provided.

[Description of Operation of Embodiment]
Next, the contents of the overall operation of this embodiment will be outlined.
First, the video object detection means 203 detects the video object displayed in the display screen 102 based on preset object recognition information (video object detection step).
Here, when a video object is detected by the video object detection means 203, the recognition mark drawing means 205 displays a recognition mark corresponding to each detected video object, and displays each detected video object. Drawing on the display screen 102 with a size corresponding to the size (recognition mark drawing step).

  Next, when the video object search unit 207 draws the recognition mark in the display determination area set in advance in the display screen 102 at a ratio of a certain value or more with respect to the size of the display determination area. Then, the object information related to the image object corresponding to the recognition mark is searched via the network line (communication line 208) (image object search step), and then the image display means 210 searches and acquires the object Object information is displayed on the display screen 102 (object information output control step).

  In addition, about the said video object detection process, a recognition mark drawing process, a video object search process, and a target object information output control process, it is good also as a structure which programs the execution content and makes a computer perform.

  Next, the operation of the display device 100 in the present embodiment will be described in detail based on the flowchart of FIG.

First, a user of the display device 100 photographs a photographing object such as a text (character string) or an object by pointing the camera lens 101 of the display device 100 (step S301: photographing an image).
Here, the user operates the zoom control unit 201 to move the display device 100 main body closer to or away from the target object so that the target object is displayed in an appropriate size on the display screen 102. Thus, zooming or the like is performed to display the object to be photographed on the display screen 102 (step S302: display an image).

At this time, for example, when a confectionery package is photographed as the photographing object, the confectionery package as the video object is displayed at the center of the display screen 102 as shown in FIG.
Here, the display area of the displayed confectionery package photographed image is smaller than the size of the preset area. Also, the confectionery package photographed image does not fall within the display determination area.
That is, since the confectionery package photographed image is not displayed in the display determination area, the video object detection unit 203 does not detect the confectionery package as a photographic object.
In the present embodiment, it is assumed that the display determination area is set in the upper left area (dotted line portion: FIG. 4) in the display screen 102.

Here, when the user of the display device 100 zooms in on the confectionery package using the zoom dial 103 or brings the camera lens 101 of the display device 100 closer to the confectionery package, for example, as shown in FIG. The confectionery package is displayed largely in the display screen 102.
At this time, the video object detection means 203 detects the text or barcode of the confectionery package displayed on the display screen 102, and based on the video object such as this text or object, a recognizable object set in advance. The list is referred to (step S303: search for a recognizable one).
Here, when the video object detection unit 203 identifies text or an object (here, text or barcode) displayed on the display screen 102 as an imaging object registered in the recognizable object list in advance, the image is detected. The recognition mark drawing size control means 204 is notified that the object has been recognized.

Next, the recognition mark drawing size control means 204 sets a star mark (recognition mark) corresponding to the video object searched as recognizable. At this time, the recognition mark drawing size control means 204 determines the display position and display size of the star mark in the display screen 102 (step S304).
Note that the recognition mark drawing size control means 204 places a star mark in a preset approximate area with respect to the display position of the video object in the display screen 102 in order to clarify the correspondence with the video object. It shall be drawn.

Specifically, the recognition mark drawing size control means 204 sets the drawing position and the drawing size of the star mark corresponding to each of the text (... Chip), the text (...), And the barcode. .
Further, as shown in FIG. 5, the recognition mark drawing unit 205 displays the set three star marks on the display screen 102 so as to overlap the object to be photographed.

  At this time, since the three star marks are not included in the display determination area set in the upper left part of the display screen 102 as shown in FIG. 5, the object information search function is not activated.

  Here, the overlapping display possibility determination unit 206 starts an operation of monitoring the display positions and sizes of the three star marks drawn. At this time, the overlap display permission / non-permission determination unit 206 determines whether or not a star mark is displayed (entered) in a display determination region (here, the upper left portion in the display screen 102) set in the display screen 102 in advance. Make a decision.

  Next, when the user moves the display device 100 while photographing the object to be photographed, as shown in FIG. 6, one of the asterisks enters the display determination area with a certain size or more. Is displayed, the overlap display availability determination unit 206 determines that “overlapping display is possible” (step S305: Y).

  On the other hand, when the star mark is not displayed in the display determination area (step S305: N), the overlap display enable / disable determining unit 206 determines the state of the display screen 102 to be the “overlay display disabled” state, thereby It is assumed that a star mark indicating that the object to be photographed can be specified (recognized) remains displayed on the display screen 102 (step S308).

When the overlap display possibility determination unit 206 determines that “overlapping display is possible” (step S305: Y), the object information related to the photographing object corresponding to the star mark displayed in the display determination area is searched for the image object. The means 207 searches (object information search starting step: step S306).
Here, the search of the object information performed by the video object search means 207 is executed by searching the web information stored on the server connected to the Internet via the communication means 208.

  Next, when the video object search means 207 obtains the search result, the object information output control means 209 outputs and displays the object information as the search result on the display screen 102 as shown in FIG. (Display object information: Step S307).

  Specifically, as shown in FIG. 6, the video object search means 207 displays the object information retrieved and acquired on the display screen 102 in the form of a jet in relation to the text information “chips”. ing. Here, on the display screen 102, as shown in FIG. 6,... Information on the calories and salt content of the chips, the URL of the homepage of the manufacturer of the chips (product), and the reputation of the chips. The URL of the home page on which is posted is output and displayed as object information.

As described above, by using the display device 100 according to the present embodiment, the user can easily display additional information on the video object on the display screen 102 by a simple operation equivalent to using a loupe or a magnifying glass. Can be displayed.
In addition, the user controls the display position and size of the recognition mark on the display screen 102 corresponding to the video object whose detailed information is desired by intuitive operation such as moving or tilting the display device 100. Additional information and detailed information of the video object can be displayed.

  Further, even when the user of the display device 100 does not know what the video object displayed on the display screen 102 is, the user can perform the video by performing a simple operation of pointing the display device 100 toward the photographing object. Additional information regarding the object can be acquired, and further, whether or not to display this additional information can be determined by an intuitive operation similar to a normal magnifying glass.

  The present invention can be applied to a system in which a photographing apparatus having a photographing and video photographing function such as a mobile phone or a digital camera is set to be communicably connected to a wide area network such as an internetwork.

100 Display Device 101 Camera Lens (Camera Unit)
102 Display Screen 201 Zoom Control Unit 202 Imaging Unit (Camera Unit)
203 Video object detection means 204 Recognition mark drawing size control means 205 Recognition mark drawing means 206 Overlapping display availability determination means 207 Video object search means 208 Communication means (network line)
209 Object information output control means 210 Image display means

Claims (8)

A display device that captures an image of an object and displays the captured image on a display screen,
Video object detection means for detecting a video object displayed on the display screen based on preset object recognition information;
A recognition mark drawing means for drawing a recognition mark corresponding to the detected video object on the display screen when the video object is detected;
When the recognition mark is drawn in a preset display determination area in the display screen so as to satisfy a preset condition, the object information related to the video object corresponding to the recognition mark is displayed. A video object search means for searching;
A display device comprising: image display means for displaying the object information acquired by the search on the display screen. The display device according to claim 1,
It further comprises a superimposed display availability determination means for monitoring the display position of the recognition mark,
The video object search means includes:
When the overlapped display permission / inhibition determining unit determines that the image is being drawn in a predetermined display determination area in the display screen based on the drawing position of the recognition mark, A display device that searches for object information relating to the video object corresponding to the recognition mark displayed in the display determination area. The display device according to claim 1,
A recognition mark drawing size control means for determining the drawing size of the recognition mark to be drawn according to the detected display size of the video object; and an overlay display availability determination means for monitoring the display position and size of the recognition mark; Further comprising
The video object search means includes:
When the overlap display possibility determination unit determines that the recognition mark is drawn in a predetermined display determination area in the display screen at a ratio of a certain value or more, a preset value is set. A display device that searches for object information relating to the video object corresponding to the recognition mark displayed in the display determination area via a network line. The display device according to any one of claims 2 and 3,
The recognition mark drawing size control means includes:
A display device, wherein when the video object is detected, a drawing position of the recognition mark is determined according to a position of the detected video object displayed on the display screen. The display device according to any one of claims 1 to 4,
The recognition mark drawing means includes
When an enlargement / reduction shooting operation is performed while the recognition mark is drawn on the display screen, the drawn recognition mark is displayed in accordance with the enlargement / reduction of the video object displayed on the display screen. A display device that displays an enlarged or reduced drawing size. The display device according to any one of claims 1 to 5,
Recognizable object list storage means for storing a recognizable object list for detecting a video object including at least one of text, substance, organism, and barcode as the object recognition information. Characteristic display device. A display method by a display device that shoots an object and displays the captured image on a display screen,
Detecting a video object displayed in the display screen based on preset object recognition information;
When the video object is detected, drawing a recognition mark corresponding to each detected video object on the display screen;
When the recognition mark is drawn in a preset display determination area in the display screen so as to satisfy a preset condition, the object information related to the video object corresponding to the recognition mark is displayed. Doing a search,
Displaying the object information acquired by the search on the display screen. A program that causes a display device to shoot an object and display the captured image on a display screen,
Detecting a video object displayed in the display screen based on preset object recognition information;
When the video object is detected, drawing a recognition mark corresponding to each detected video object on the display screen;
When the recognition mark is drawn in a preset display determination area in the display screen so as to satisfy a preset condition, the object information related to the video object corresponding to the recognition mark is displayed. Doing a search,
Displaying the object information acquired by the search on the display screen.

Images