JP5382788B2 - Object information display device, object information display method, object information display program - Google Patents

Description

  The present invention relates to a technique for acquiring object information indicating additional information related to a video object included in a captured video.

A camera that captures the object to be photographed and an output screen that displays the photographed image and video are also identified and identified from the image and video displayed on the output screen. A method for acquiring additional information related to a video object is disclosed (Patent Document 1).
In addition, as a related technique for this, a technique for displaying the additional information of the acquired video object superimposed on the same output screen as the video object is disclosed (Patent Document 2).
Furthermore, as a related technique for this, a method for recognizing not only text but also general objects from a photographed image is disclosed (Patent Document 3).

JP 2006-60535 A Special table 2008-510254 gazette JP 2001-307096 A

However, the related technique disclosed in Patent Document 1 is a technique for substituting an operation of performing a text search by key input, and superimposing additional information on a video object in an actually captured image. There is an inconvenience that it cannot be displayed.
Further, in the related art disclosed in Patent Document 2, when automatically stacked displaying the additional information to the output screen, or additional information I appear not reflect the intention of the user, thereby the output screen Inconveniences such as hindering visibility may occur.

Further, in the related technique disclosed in Patent Document 3, when a plurality of different video objects are displayed on the screen , additional information for which object (video object) is displayed on the output screen. It becomes difficult to determine whether.
In addition, for example, in order to specify a video object on which additional information is displayed, an operation button or the like is provided, and when the user directly selects a specific video object on the screen, the operation becomes complicated. There is an inconvenience.
In this case, for example, the intuitive operability is greatly impaired as compared with a normal operation with a magnifying glass or a loupe, and the user can easily and intuitively operate the additional information of the video object. There is a disadvantage that it cannot be displayed.

[Object of invention]
The present invention improves the inconvenience of the related technology, and acquires and displays additional information related to a video object photographed and output and displayed by a user by an intuitive and simple operation, and an object An object is to provide an information display method and an object information display program.

In order to achieve the above object, an object information display device according to the present invention includes a camera unit that captures an image of an object, and an output that displays an image captured by the camera unit on a preset output screen. An object information display device comprising: a display control unit; and an object information acquisition unit that acquires object information indicating information related to a video object displayed on the output screen via a preset network line. A video object detection means for detecting a video object (text information, specific substance) displayed in the output screen based on preset object recognition information; and the video object is detected. each video object recognition mark forming means for the detected identification mark corresponding to each video object which is drawn on the output screen, the drawing size of the recognition mark is the drawing which is the detected when the A recognition mark drawing size control means determining in accordance with a display size of the object information acquiring unit, fixed to the size of the display determination area on the recognition mark preset display determination area in the output screen Video object search means for searching for object information relating to the video object corresponding to the recognition mark via the network line when drawn at a size of a ratio equal to or greater than a value, and obtained by the search has taken a configuration in which an object information output control means for the object information is output to the output screens.

The object information display method according to the present invention includes a camera unit that captures an image of the object, and an output display control unit that displays an image captured by the camera unit on a preset output screen. The object information display method for acquiring the object information, which is information related to the image object displayed on the output screen, via a preset network line, and displayed on the output screen. When the detected video object is detected based on preset object recognition information, and the video object is detected, a recognition mark corresponding to the detected video object is displayed for each detected video. drawn with size corresponding to the display size of the object on the output screen, the recognition mark of a predetermined value or more with respect to the size of the display determination region to a preset display determination area in the output screen When drawn by the size of the case, to search for such object information to the image object corresponding to the recognition mark through said network line,
The object information acquired by the search is characterized by displaying to the output screen.

Furthermore, an object information display program according to the present invention includes a camera unit that shoots an object, and an output display control unit that displays an image captured by the camera unit on a preset output screen. The object information display program for acquiring the object information, which is information related to the video object displayed on the output screen, via a preset network line by the object information display device, Video object detection function for performing processing for detecting the video object displayed in the output screen based on preset object recognition information, and when the video object is detected, each detected video object the recognition mark corresponding to the object, the detected recognition mark drawing function to perform the process of drawing on the output screen in a size corresponding to the display size of each image object, the recognition mer The but when drawn in a size ratio of a certain value or more for the size of the display determination region to a preset display determination area in the output screen, corresponding to the recognition mark through the network line A video object search function for performing a process of searching for object information related to a video object, and an object information output control function for performing a process of displaying the object information acquired by the search on the output screen. It is characterized by being executed by a computer provided with the object information display device.

  Since the present invention is configured and functions as described above, according to this, means for rendering a recognition mark indicating that a video object in the output screen has been detected in accordance with the size of the video object in the output screen And a means for acquiring additional information relating to the video object via the communication means when the recognition mark is enlarged to a certain size, and displaying the image obtained by photographing the object It is possible to provide an object information display device, an object information display method, and an object information display program capable of easily acquiring and displaying information related to an object.

It is a schematic block diagram which shows one Embodiment of the external appearance of the target object information display apparatus by this invention. It is a schematic block diagram which shows one Embodiment of the internal structure of the target object information display apparatus disclosed in FIG. It is a sequence chart which shows the whole operation | movement in the target object information display apparatus disclosed in FIG. It is explanatory drawing which shows an example of the imaging | photography state in the target object (confectionery package) in the target object information display apparatus disclosed in FIG. It is explanatory drawing which shows an example of the method of drawing the recognition mark in the target object (confectionery package) in the target object information display apparatus disclosed in FIG. It is explanatory drawing which shows an example of the display method of the target object (confectionery package) information in the target object information display apparatus disclosed in FIG. It is explanatory drawing which shows an example of the specific method of the target object (beetle) in the target object information display apparatus disclosed in FIG. It is explanatory drawing which shows an example of the display method of the target object (beetle beetle) information in the target object information display apparatus disclosed in FIG.

[Embodiment]
Next, the basic configuration content of the embodiment of the present invention will be described.

The object information display device 100 according to the present embodiment has a housing shape as shown in FIGS. 1A and 1B, and the imaging object (on the rear surface of the exterior part (FIG. 1B)). A camera lens 101 (FIG. 1 (b)) for photographing an object is provided, and an image photographed via the camera lens 101 is output and displayed on the exterior surface (FIG. 1 (a)) side. The output screen 102 is provided.

Further, the object information display device 100 includes a zoom dial 103 as an interface for controlling enlargement photographing and reduction photographing by the camera lens 101 on the side surface of the exterior portion.
Thereby, the user of the object information display device 100 can adjust the enlarged / reduced display of the photographed object output and displayed on the output screen 102 by operating the zoom dial 103.

Next, the internal configuration of the object information display device 100 will be described in detail based on the block diagram of FIG.
The object information display device 100 detects an operation (input) by the zoom dial 103 and notifies the photographing means (camera unit) 202 of the detected operation content (enlargement / reduction operation information) of the zoom dial 103 to perform zoom photographing. A zoom control unit 201 that controls the image, and a shooting unit 202 that performs shooting of an object through the camera lens 101 and controls enlargement / reduction in the camera lens 101 based on enlargement / reduction operation information notified from the zoom control unit 201. The video object detection unit 203 detects a video object such as text information or a specific substance, which is an object in the video imaged by the imaging unit 202.
The video object detection means 203 has an object recognition information holding function for storing and holding object recognition information as reference information for detecting the object.

Further, the object information display device 100 sets a recognition mark corresponding to each detected image object when the image object is detected by the image object detection unit 203, and also draws the recognition mark. And a recognition mark drawing size control means 204 for determining the size in accordance with the detected display size of each video object, and a recognition mark drawing means 205 for drawing the determined recognition mark on the output screen 102.

Further, the object information display apparatus 100 is configured such that the recognition mark drawn by the recognition mark drawing unit 205 is within a predetermined display determination area in the output screen 102 and is a ratio of a certain value or more with respect to the size of the display determination area. The object information display device 100 when the overlapping display availability determination means 206 for determining whether or not the image is drawn with the size of the recognition mark and the recognition mark in the display determination area is drawn with a size of a certain ratio or more. Video object search means 207 for searching for object information as information relating to the image object corresponding to the drawn recognition mark, via a communication means (network line) 208 provided in advance, and the video a configuration that includes an object information output control section 209 to transmit the object information retrieved acquired by the object search unit 207 to the output display controller 210 That.

Note that the overlapping display availability determination unit 206, the video object search unit 207, and the object information output control unit 209 correspond to an object information acquisition unit.

The photographing unit 202 photographs a photographing object such as a text (character string) or a target object to which the camera lens 101 is directed by the user.
Incidentally, the user of the object information display device 100, so that the shooting target is displayed in appropriate size on the output screen 102, or away from the object or close an object information display device 100 main body to the object It is assumed that zooming or the like can be performed by operating the zoom control unit 201 to display the photographing target in the output screen 102.

Video object detecting means 203, in the photographed image displayed on the output screen 102, as recognizable object list those video object detecting means 203 of the object information display device 100 is determined to be "recognizable" Recognizable object list storage means for storing and holding is provided.
The recognizable object list storage means is realized by a storage device such as an HDD (hard disk drive) installed in advance in the object information display device 100.

The video object detection means 203 is a video object that is detected based on the recognizable object list among the video objects (including text information, specific substances, organisms, etc.) displayed in the output screen 102. It has a recognition object identification function that identifies
The recognition object specifying function is set to use an image recognition technique (for example, Japanese Patent Laid-Open No. 2001-307096) for recognizing not only text but also a general object from a captured image (video). It is good.
Accordingly, in the present embodiment, a specific video object (such as an object or text) can be extracted from the captured image by image recognition technology, and marking can be performed on the video object.

When the video object detection unit 203 identifies a video object such as a text or an object (target object) displayed on the output screen 102 as a recognizable object registered in the recognizable object list in advance, this video object detection unit 203 The recognition mark drawing size control means 204 is notified that the object has been recognized.

  Note that, as a method for specifying a video object from a captured image, for example, a setting using an image recognition method disclosed in Japanese Patent Laid-Open No. 2001-307096 may be used.

The recognition mark drawing size control means 204 sets a star mark (corresponding to a recognition mark) corresponding to the video object specified as recognizable (recognition mark setting function). At this time, the recognition mark drawing size control means 204 determines the display position and display size of the star mark in the output screen 102.

Further, the recognition mark drawing size control means 204 determines the star mark in the output screen 102 from the display position of the video object in order to clarify the correspondence between the set star mark and the specified video object. It shall be drawn at a position (area) within the range.

Specifically, as shown in FIG. 7, the recognition mark drawing size control means 204 draws so that a star mark is displayed on the upper left in the vicinity of the beetle recognized by the video object detection means 203 as a video object. Set the position.

The recognition mark drawing unit 205 outputs a star mark to the output screen 102 via the output display control unit 210 based on information indicating the size and position of the recognition mark (star mark) set by the recognition mark drawing size control unit 204. It has a recognition mark output display control function to display.
It is assumed that the size of the star mark output and displayed is enlarged and reduced in accordance with the zoom operation (enlargement and reduction operation) of the zoom dial 103.

When the star mark is drawn by the recognition mark drawing means 205, the overlapping display availability determination means 206 starts monitoring the display position and size ( recognition mark position size monitoring function).
In this case, superimposed display determination unit 206, a preset display determination area (here, the output screen 102 upper left) to the output screen 102 it is determined whether or not star mark is displayed (recognition Mark display area judgment function).

Further, the overlay display availability determination unit 206 determines that “overlapping display is possible” when one of the stars is displayed so as to fall within the display determination area with a certain size or more.
On the other hand, when the star mark is not displayed in the display determination area , the overlap display permission determination unit 206 determines that the state of the output screen 102 is a state of “not overlap display”. At this time, it is assumed that a star mark indicating that the photographing object can be specified (recognized) remains displayed on the output screen 102.

For example, FIG. 7 illustrates a state in which a star mark corresponding to a beetle that is a video object is displayed in a display determination area in the output screen 102. Here, since the display size of the displayed star mark is smaller than a preset ratio, the object information search function described later is not activated.

The video object search means 207 corresponds to the star mark displayed in the display determination area when the overlap display possibility determination means 206 determines that “overlapping display is possible” based on the drawing position of the recognition mark (star mark). An object information search function for starting a search for additional information ( object information ) related to a video object is provided.

The video object search means 207 executes this object information search function to search the web information stored on the server connected to the Internet via the communication means 208 and to obtain the shooting object (video Additional information related to the (object) is acquired as a search result.

The object information output control unit 209 determines whether or not a star mark is displayed in a predetermined display determination area in the output screen 102 with a predetermined size or more.
As a result, for example, a video object unintended by the user is recognized and the object information is displayed, so that a part of the screen is covered, and as a result, it is difficult to see the captured screen. Can be reduced.

Also, the object information output control section 209, when the star mark in the display determination area is displayed at a fixed size or more, the result searched by the video object search unit 207 (object information), the output display control The output is displayed on the output screen 102 via the unit 210 ( object information output display function).
Thereby, the user can control the display and non-display of the additional information of the video object that suits his / her intention by a simple operation.

Specifically, as illustrated in FIG. 8, when the star mark is displayed in a size larger than a certain size in the display determination region , the output display control unit 210 displays the attribute information (“ The beetle is a large insect and the adult is in summer ... ") superimposed on the captured image (in the form of a squirt here) for output display.

It Thereby, for example, the user of the object information display device 100, output screen 102 displayed on the the object (in this case, beetles) even if the do not know what is, shooting in the object information display device 100 Thus, information on the object is displayed on the output screen 102 , and whether or not additional information ( object information ) is displayed over the photographed object can be determined by an operation similar to that of a normal magnifying glass. .
The output display control unit 210 controls the operation in which the recognition mark drawing unit 205 outputs and displays the recognition mark (star mark) on the output screen 102, and the object information output control unit 209 provides additional information as a search result. Is provided with an output display control function for controlling the operation of outputting and displaying the image on the output screen 102.

[Description of Operation of Embodiment]
Next, the contents of the overall operation of this embodiment will be outlined.
First, the video object detection unit 203 detects the video object displayed in the output screen 102 based on preset object recognition information (video object detection step).
Here, when a video object is detected by the video object detection means 203, the recognition mark drawing means 205 displays a recognition mark corresponding to each detected video object, and displays each detected video object. Drawing on the output screen 102 with a size corresponding to the size (recognition mark drawing step).

Next, when the video object search unit 207 draws the recognition mark in the display determination area set in advance in the output screen 102 at a ratio of a certain value or more with respect to the size of the display determination area. The object information related to the video object corresponding to the recognition mark is searched via the network line (communication line 208) (video object search process), and then the output display control unit 210 is retrieved and acquired. Object information is displayed on the output screen (object information output control step).

  In addition, about the said video object detection process, a recognition mark drawing process, a video object search process, and a target object information output control process, it is good also as a structure which programs the execution content and makes a computer perform.

Next, the operation of the object information display apparatus 100 in the present embodiment will be described in detail based on the flowchart of FIG.

First, the user of the object information display device 100, by directing the camera lens 101 of the object information display apparatus 100 photographs the photographing object such as a text (character string) and the object (step S301: image Shooting).
Here, the user moves the zoom control unit 201 closer to or away from the target object so that the shooting target object is displayed in an appropriate size on the output screen 102. The zooming or the like is performed by the operation, and the photographing object is displayed in the output screen 102 (step S302: display an image).

At this time, for example, when a confectionery package is photographed as the photographing object, the confectionery package as the video object is displayed at the center of the output screen 102 as shown in FIG.
Here, the display area of the displayed confectionery package photographed image is smaller than the size of the preset area . Also, the confectionery package photographed image does not fall within the display determination area .
That is, since the confectionery package photographed image is not displayed in the display determination area , the video object detection unit 203 does not detect the confectionery package as a photographic object.
In the present embodiment, it is assumed that the display determination area is set in the upper left area (dotted line portion: FIG. 4) in the output screen 102.

Here, when the user of the object information display device 100 zooms in on the confectionery package using the zoom dial 103 or brings the camera lens 101 of the object information display device 100 close to the confectionery package, for example, As shown in FIG. 5, the confectionery package is largely displayed in the output screen 102.
At this time, the video object detection means 203 detects the text or barcode of the confectionery package displayed on the output screen 102 and, based on the video object such as the text or the object, a recognizable object set in advance. The list is referred to (step S303: search for recognizable items).
Here, the video object detection unit 203 takes a picture when the text or object (here, text, barcode) displayed on the output screen 102 is specified as a photographing object registered in the recognizable object list in advance. The recognition mark drawing size control means 204 is notified that the object has been recognized.

Next, the recognition mark drawing size control means 204 sets a star mark ( recognition mark) corresponding to the video object searched as recognizable . At this time, the recognition mark drawing size control means 204 determines the display position and display size of the star mark in the output screen 102 (step S304).
Note that the recognition mark drawing size control means 204 places a star mark in a preset approximate area with respect to the display position of the video object in the output screen 102 in order to clarify the correspondence with the video object. It shall be drawn.

Specifically, the recognition mark drawing size control means 204 sets the drawing position and the drawing size of the star mark corresponding to each of the text (... Chip), the text (...), And the barcode. .
Further, as shown in FIG. 5, the recognition mark drawing unit 205 displays the set three star marks on the output screen 102 so as to overlap the photographing object.

At this time, since the three star marks are not within the display determination area set in the upper left portion of the output screen 102 as shown in FIG. 5, the object information search function is not activated.

Here, the overlapping display possibility determination unit 206 starts an operation of monitoring the display positions and sizes of the three star marks drawn. At this time, the overlapping display permission / non-permission determining unit 206 determines whether or not a star mark is displayed (entered) in a display determination area (here, the upper left portion in the output screen 102) set in the output screen 102 in advance. Make a decision.

Next, when the user moves the object information display device 100 while photographing the object to be photographed, as shown in FIG. 6, one of the stars is larger than a certain size in the display determination area . When it is displayed so as to enter, the overlapping display availability determination unit 206 determines that “overlapping display is possible” (step S305: Y).

On the other hand, when the star mark is not displayed in the display determination area (step S305: N), the overlap display enable / disable determining unit 206 determines the state of the output screen 102 to be the “overlay display disabled” state, thereby In the output screen 102, it is assumed that the star mark indicating that the object to be photographed can be specified (recognized) remains displayed (step S308).

When the overlap display possibility determination unit 206 determines that “overlapping display is possible” (step S305: Y), the object information related to the photographing object corresponding to the star mark displayed in the display determination area is searched for the image object. The means 207 searches ( object information search starting step: step S306).
Here, the search of the object information performed by the video object search means 207 is executed by searching the web information stored on the server connected to the Internet via the communication means 208.

Next, when the video object search means 207 obtains the search result, the object information output control means 209 displays the object information as the search result on the output screen 102 as shown in FIG. (Display object information : Step S307).

Specifically, as shown in FIG. 6, the video object search means 207 displays the object information retrieved and acquired on the output screen 102 in the form of a jet in relation to text information called chips. ing. Here, on the output screen 102, as shown in FIG. 6, the information on the calories and salt content of the chips, the URL of the homepage of the manufacturer of the chips (product), and the reputation of the chips The URL of the home page on which is posted is output and displayed as object information .

As described above, by using the object information display device 100 according to the present embodiment, the user can easily output additional information related to the image object by a simple operation equivalent to using a loupe or a magnifying glass. It can be displayed on the screen 102 .
Further, the user controls the display position and size of the recognition mark on the output screen 102 corresponding to the video object whose detailed information is desired by intuitive operation such as moving or tilting the object information display device 100. By doing so, it is possible to display additional information and detailed information of the video object.

Furthermore, the user of the object information display device 100, even if the video object unit which is displayed on the output screen 102 does not know what, simple that directs the object information display device 100 in the shooting target operation By performing the above, it is possible to acquire additional information related to the video object, and furthermore, whether to display this additional information can be determined by an intuitive operation similar to a normal magnifying glass.

  The present invention can be applied to a system in which a photographing apparatus having a photographing and video photographing function such as a mobile phone or a digital camera is set to be communicably connected to a wide area network such as an internetwork.

100 object information display device 101 camera lens (camera unit)
102 output screen 201 zoom control means 202 photographing means (camera part)
203 Video object detection means 204 Recognition mark drawing size control means 205 Recognition mark drawing means 206 Overlapping display availability determination means ( object information acquisition unit )
207 video object search means (object information acquisition unit )
208 Communication means (network line)
209 object information output control means (object information acquisition unit )
210 Output display controller

Claims (7)

A camera unit that captures an image of an object, an output display control unit that displays an image captured by the camera unit on a preset output screen, and information related to the image object displayed on the output screen. A target object information display device comprising a target object information acquisition unit for acquiring target object information to be acquired via a preset network line,
Video object detection means for detecting the video object displayed in the output screen based on preset object recognition information;
A recognition mark drawing means for drawing a recognition mark corresponding to each detected video object on the output screen when the video object is detected;
A recognition mark drawing size control means for determining a drawing size of the recognition mark to be drawn according to a display size of each detected video object ;
When the recognition mark is drawn in a predetermined display determination area in the output screen at a ratio of a certain value or more with respect to the size of the display determination area, the recognition mark is displayed on the recognition mark via the network line. Video object search means for searching for object information related to the corresponding video object ;
Object information display device characterized by comprising an object information output control means for outputting the object information obtained by the search to the output screen. The object information display device according to claim 1,
The recognition mark forming means, when said scaling display captured by the camera unit is performed when the recognition mark is drawn, scaling video object displayed on the output screen on the output screen An object information display device comprising a recognition mark enlargement / reduction control function for controlling enlargement / reduction of a drawn drawing size of the drawn recognition mark according to a ratio. The object information display device according to claim 1 or 2,
The recognition mark drawing means includes
A recognition mark drawing position determination function for determining a drawing position of the recognition mark according to a position where the detected video object is displayed in the output screen when the video object is detected; An object information display device characterized by the above. The object information display device according to claim 1 or 2,
The recognition mark drawing means includes
A recognition mark drawing position determining function for determining a drawing position of the recognition mark according to a position where the detected video object is displayed in the output screen when the video object is detected;
An object information display device comprising: a correspondence mark drawing function for drawing a correspondence mark indicating a correspondence relation with each detected video object as each recognition mark. The object information display device according to claim 1,
The object information display characterized in that the image object detection means is provided with a recognition object list storage means for storing a recognition object list for detecting an object including specific text information as the object recognition information. apparatus. A video object displayed on the output screen by a configuration including a camera unit that captures an object and an output display control unit that displays an image captured by the camera unit on a preset output screen. An object information display method for acquiring object information, which is information related to an object, through a preset network line,
Detecting a video object displayed in the output screen based on preset object recognition information,
When the video object is detected, a recognition mark corresponding to each detected video object is drawn on the output screen with a size corresponding to the display size of each detected video object,
When the recognition mark is drawn in a predetermined display determination area in the output screen at a ratio of a certain value or more with respect to the size of the display determination area, the recognition mark is displayed on the recognition mark via the network line. Search for object information related to the corresponding video object,
The object information display method characterized by displaying the object information acquired by the search on the output screen. Displayed on the output screen by an object information display device comprising a camera unit that captures an object and an output display control unit that displays an image captured by the camera unit on a preset output screen. An object information display program for acquiring object information, which is information related to a video object that has been set, via a preset network line,
A video object detection function for performing processing for detecting a video object displayed in the output screen based on preset object recognition information;
When the video object is detected, a process of drawing a recognition mark corresponding to each detected video object on the output screen with a size corresponding to the display size of each detected video object Recognition mark drawing function,
When the recognition mark is drawn in a predetermined display determination area in the output screen at a ratio of a certain value or more with respect to the size of the display determination area, the recognition mark is displayed on the recognition mark via the network line. A video object search function for performing processing for searching for object information related to the corresponding video object;
An object information output control function for performing processing for displaying the object information acquired by the search on the output screen;
Is executed by a computer provided with the object information display device.

Images