JP4077608B2 - Feature region extraction method and apparatus, and information providing method and apparatus - Google Patents

Description

【００７７】
の小さな画像データが類似広告物として移動機２０の表示部２８に提示される。または、
【００７８】
【数２】

【００７９】
のように、全ての特徴空間において、近隣の特徴点を有する画像を類似広告物として扱うことも考えられる。
【００８０】
図８は、本発明の一実施形態にかかる情報提供システムの利用方法を説明するための図である。ユーザが街角の情景を撮影することによって得られる画像データと、広告物データベース３６に格納された情報とを照合して、その情景に含まれる広告物を検出し、検出した広告に関する情報を広告物データベース３６から移動機２０に送信する。移動機２０の表示部２８に表示されている画像データには、検出した広告に関する情報が重畳して表示され、ユーザは、容易に店舗や場所の情報を確認することができる。
【００８１】
【発明の効果】
以上説明したように、本発明によれば、移動機から送られた画像情報に関連する情報を画像特徴量から選択して提供することにより、ユーザが必要とする目標の情報を容易に得ることが可能となる。
【図面の簡単な説明】
【図１】本発明の一実施形態にかかる情報提供システムの構成を示したブロック図である。
【図２】本発明の一実施形態にかかる情報提供方法を示したフローチャートである。
【図３】本発明の一実施形態にかかる対象判別方法におけるズームによる広告物の指定方法を示した図である。
【図４】本発明の一実施形態にかかる対象判別方法における形状指定による広告物の指定方法を示した図である。
【図５】ＨＭＤを使用した移動機における広告物の指定方法を示した図である。
【図６】撮影位置ごとにプロットされた高彩色領域比と色相平均値の分布の例を示した図である。
【図７】撮影位置ごとにプロットされた領域アスペクト比の分布の例を示した図である。
【図８】本発明の一実施形態にかかる情報提供システムの利用方法を説明するための図である。
【符号の説明】
１０ 無線通信網
１１ データ中継装置
２０，３０２ 移動機
２１，３１ データ送受信部
２２ 音声送信部
２３，３３ 画像変換部
２４ 撮像部
２５ 測位部
２８ 表示部
２９ 無線通信部
３０ 処理センタ
３４ 広告物データ検索部
３５ 広告物データ照合部
３６ 広告物データベース
３７ 広告物データ登録部
３０１ ユーザ
３０３ 広告物
３０４ ファインダ
４０１ カーソル
５０１ ＨＭＤ（Head Mounted Display）
５０２ レーザポインタ
５０３ 仮想的なカッター
５０４ 仮想的なブラシ[0001]
BACKGROUND OF THE INVENTION
The present invention relates to a feature region extraction method and apparatus, an information providing method and apparatus, and more particularly, a feature region extraction method and apparatus for obtaining a feature region for calculating an image feature amount of a specific image, and The present invention relates to an information providing method and apparatus for selecting and providing information related to image information sent from a mobile device from image feature amounts.
[0002]
[Prior art]
In recent years, a mobile terminal is used to connect to the Internet, and various information can be obtained from an information providing system such as WWW (World Wide Web) on the go. On the other hand, devices that can acquire position information by using GPS (Global Positioning System) are widely used. Accordingly, by notifying the information providing system from the mobile terminal to the information providing system, it is possible to quickly obtain information associated with the position information.
[0003]
Also, with the spread of CCD (Charge Coupled Device) cameras, it has become possible to capture and store image information even in small devices such as mobile terminals. Furthermore, the captured image can be easily sent as image data via a communication network by digital image processing.
[0004]
[Problems to be solved by the invention]
However, even if the information associated with the location information of the mobile terminal can be obtained, limited information such as target information required by the user, for example, store information visible from the location of the mobile terminal is obtained. In addition, there is a problem that operations such as selection of information or setting of targets are necessary.
[0005]
In particular, in setting a target, when a specific store name is unknown and is specified by an abstract facility name, such as an advertisement, it is difficult to associate information. There is also a problem that it is difficult to select information only by extracting a specific image from image information sent from a mobile device.
[0006]
The present invention has been made in view of such a problem, and an object of the present invention is to provide a feature region extraction method and a device for obtaining a feature region for calculating an image feature amount of a specific image, and a movement thereof. It is an object to provide an information providing method and apparatus for selecting and providing information related to image information sent from a machine from image feature amounts.
[0007]
[Means for Solving the Problems]
In order to achieve such an object, the invention according to claim 1 extracts a specific image from image information photographed by an imaging unit of a mobile terminal and calculates an image feature amount. A feature region extracting method for obtaining a feature region for obtaining a normalized image in which the region of the specific image is smoothed to remove noise and the density of each pixel value is normalized; and Mapping to a space and performing clustering to obtain a region-divided image, and selecting a class to which the vicinity of the center of the region of the specific image belongs from the entire region-divided image To select an area belonging to the class Steps, After selecting an area belonging to the class, Labeling each region of the region-divided image; deleting each region labeled with a region area ratio of the specific image; and deleting a peripheral region of the region of the specific image; , Said Step to delete After, Left Effectiveness Obtaining the feature region by obtaining a minimum circumscribed convex polygon including the region.
[0008]
The invention according to claim 2 is the invention according to claim 1, Said The feature region is obtained from a specific image cut out using a zoom function of a mobile terminal.
[0009]
The invention according to claim 3 is the invention according to claim 1, Said The feature region is obtained from a specific image cut out by designating a shape on a display device of a mobile terminal.
[0010]
The invention according to claim 4 is the invention according to claim 1, The mobile terminal is body-mounted, and the mobile terminal In the display device, the feature region is obtained from a specific image cut out in a virtual field of view.
[0011]
The invention according to claim 5 is an information providing method in which an advertisement is extracted from image information received from a mobile terminal, and an information providing server retrieves provided information related to the advertisement from a database. In the area of the advertisement object cut out from the image information, a feature area calculation step for obtaining an advertisement object feature area for calculating an image feature amount, and the advertisement object obtained in the feature area calculation step A feature amount calculating step for calculating the image feature amount for searching for provided information related to the advertisement from the feature region, and the advertisement based on the image feature amount calculated in the feature amount calculating step. A search step for searching for provision information related to an object from the database, and provision information related to the advertisement thing searched in the search step is transmitted to the mobile terminal. A step of obtaining a normalized image obtained by smoothing the area of the advertising object to remove noise and normalizing the density of each pixel value; and Mapping to the HVC space and performing clustering to obtain a region-divided image, and selecting a class to which the vicinity of the center of the area of the advertisement belongs belongs from the entire region-divided image To select an area belonging to the class Steps, After selecting an area belonging to the class, Labeling each region of the region-divided image, deleting a labeled region based on a region area ratio of the advertisement, and deleting a peripheral region of the region of the advertisement , Said Step to delete After, Left Effectiveness Obtaining the advertisement feature area by obtaining a minimum circumscribed convex polygon including the area.
[0012]
The invention described in claim 6 is characterized in that the area of the advertisement object described in claim 5 is cut out using a zoom function of the mobile terminal.
[0013]
The invention according to claim 7 is characterized in that the area of the advertisement object according to claim 5 is cut out by designating a shape on the display device of the mobile terminal.
[0014]
The invention described in claim 8 is characterized in that the area of the advertisement according to claim 5 is cut out in a virtual field of view in the display device of the body-mounted mobile terminal.
[0016]
Claim 9 The invention described in claims 5 to 5 8 The search step described in any one of the above is characterized in that, based on the image feature amount, the provided information of a small evaluation function is searched for in each feature space of the database.
[0017]
Claim 10 The invention described in claim 5 Or 9 The image feature amount described in any one of the above is one of a high chromatic area ratio and a hue average value relating to a color, and an area aspect ratio relating to an outer shape.
[0018]
According to an eleventh aspect of the present invention, in the feature region extraction apparatus for extracting a specific image from image information captured by an imaging unit of a mobile terminal and obtaining a feature region for calculating an image feature amount, the specific region Means for smoothing an image area to remove noise and obtaining a normalized image in which the density of each pixel value is normalized, and means for mapping the normalized image to an HVC space and performing clustering to obtain an area divided image And a class to which the vicinity of the center of the area of the specific image belongs is selected from the entire area-divided image. To select an area belonging to the class Means, After the region belonging to the class is selected, Means for labeling each region of the region-divided image; and means for deleting each of the labeled regions according to a region area ratio of the specific image and deleting a peripheral region of the region of the specific image , After the peripheral area is deleted, Left Effectiveness Means for obtaining the feature region by obtaining a minimum circumscribed convex polygon including the region.
[0019]
Claim 12 The invention described in claim 11 In Said The feature region is obtained from a specific image cut out using a zoom function of a mobile terminal.
[0020]
Claim 13 The invention described in claim 11 The feature region is obtained from a specific image cut out by designating a shape on the display device of the mobile terminal.
[0021]
Claim 14 The invention described in claim 11 In the display device of the body-mounted mobile terminal, the feature region is obtained from a specific image cut out in a virtual field of view.
[0022]
The invention according to claim 15 is an information providing apparatus that extracts an advertisement from image information received from a mobile terminal, and retrieves and provides provided information related to the advertisement from a database. In the area of the advertisement object cut out from the information, the feature area calculation means for obtaining the advertisement object feature area for calculating the image feature amount, and the advertisement object feature area obtained by the feature area calculation means, A feature amount calculation unit for calculating the image feature amount for searching for provision information related to the advertisement item, and a feature amount calculation unit that relates to the advertisement item based on the image feature amount calculated by the feature amount calculation unit Search means for searching provided information from the database; and transmission means for transmitting provided information related to the advertisement searched for by the searching means to the mobile terminal, the feature region calculation The stage smoothes the area of the advertising material to remove noise and obtains a normalized image in which the density of each pixel value is normalized; and maps the normalized image to the HVC space to perform clustering A means for obtaining a divided image, and a class to which the vicinity of the center of the area of the advertisement item belongs is selected from the entire area divided image. To select an area belonging to the class Means, After the region belonging to the class is selected, Means for labeling each region of the region-divided image; and means for deleting each of the labeled regions based on a region area ratio of the advertisement and deleting a peripheral region of the region of the advertisement , After the peripheral area is deleted, Left Effectiveness Means for obtaining the advertisement feature area by obtaining a minimum circumscribed convex polygon including the area.
[0023]
Claim 16 The invention described in claim 15 The area of the advertisement described in the above item is cut out using a zoom function of the mobile terminal.
[0024]
Claim 17 The invention described in claim 15 The area of the advertisement described in the above item is cut out by designating a shape on the display device of the mobile terminal.
[0025]
Claim 18 The invention described in claim 15 The area of the advertising material described in the above is cut out in a virtual field of view in the display device of the body-mounted mobile terminal.
[0027]
Claim 19 The invention described in claim 15 Or 18 The search means according to any one of the above is characterized in that provided information having a predetermined evaluation function smaller than a predetermined value is searched for in each feature space of the database based on the image feature amount.
[0028]
Claim 20 The invention described in claim 15 Or 18 The transmission means according to any one of the above, rearranges the provision information related to the advertisement in order from the provision information having position information close to the mobile terminal based on the position information received from the mobile terminal. And transmitting.
[0029]
Claim 21 The invention described in claim 15 Or 20 In any one of the above, the image information received from the mobile terminal is stored together with the image feature amount in the database.
[0030]
Claim 22 The invention described in claim 15 Or 21 Either Leave The image feature amount is one of a high chromatic area ratio and a hue average value relating to color, and an area aspect ratio relating to an outer shape.
[0031]
According to a twenty-third aspect of the present invention, there is provided a program for extracting a specific image from image information photographed by an imaging unit of a mobile terminal and controlling a feature region extracting apparatus for obtaining a feature region for calculating an image feature amount. A recording medium on which the specific image is smoothed to remove noise and obtain a normalized image in which the density of each pixel value is normalized, and mapping the normalized image to the HVC space Clustering to obtain a region divided image, and selecting a class to which the vicinity of the center of the region of the specific image belongs from the entire region divided image To select an area belonging to the class Steps, After the region belonging to the class is selected, Labeling each region of the region-divided image; deleting each region labeled with a region area ratio of the specific image; and deleting a peripheral region of the region of the specific image; , Said Step to delete After, Left Effectiveness A computer-readable recording medium storing a program for causing a computer to execute the step of obtaining the feature region by obtaining a minimum circumscribed convex polygon including the region.
[0032]
According to a twenty-fourth aspect of the present invention, there is provided a program for controlling an information providing apparatus that extracts an advertisement material from image information received from a mobile terminal, and retrieves and provides provided information related to the advertisement material from a database. A feature area calculating step of obtaining an advertisement feature area for calculating an image feature amount in the advertisement article area cut out from the image information, the advertisement article Smoothing the region to remove noise and obtaining a normalized image in which the density of each pixel value is normalized; and mapping the normalized image to an HVC space to perform clustering to obtain a region-divided image; , Select a class to which the vicinity of the center of the area of the advertisement belongs from the entire area-divided image To select an area belonging to the class Steps, After selecting an area belonging to the class, Labeling each region of the region-divided image, deleting a labeled region based on a region area ratio of the advertisement, and deleting a peripheral region of the region of the advertisement , Said Step to delete After, Left Effectiveness Obtaining the advertisement feature area by obtaining a minimum circumscribed convex polygon including the area, and calculating the advertisement from the feature area calculated in the feature area calculation step. A feature amount calculation step for calculating the image feature amount for searching for provision information related to the object, and the provision information related to the advertisement based on the image feature amount calculated in the feature amount calculation step A computer-readable recording medium storing a program for causing a computer to execute a search step of searching the database for a search step, and a transmission step of transmitting provided information related to the advertisement searched for in the search step to the mobile terminal It is a characteristic recording medium.
[0033]
According to a twenty-fifth aspect of the present invention, there is provided a program for controlling a feature region extracting apparatus that obtains a feature region for calculating an image feature amount by extracting a specific image from image information captured by an imaging unit of a mobile terminal. A step of smoothing a region of the specific image to remove noise and obtaining a normalized image in which the density of each pixel value is normalized, and performing clustering by mapping the normalized image to an HVC space Obtaining a region-divided image, and selecting a class to which the vicinity of the center of the region of the specific image belongs from the entire region-divided image To select an area belonging to the class Steps, After selecting an area belonging to the class, Labeling each region of the region-divided image; deleting each region labeled with a region area ratio of the specific image; and deleting a peripheral region of the region of the specific image; , Said Step to delete After, Left Effectiveness It is a program for causing a computer to execute the step of obtaining the feature region by obtaining the minimum circumscribed convex polygon including the region.
[0034]
The invention according to claim 26 is an information providing apparatus program that extracts an advertisement from image information received from a mobile terminal, and retrieves and provides provided information related to the advertisement from a database. A feature area calculating step for obtaining an advertisement feature area for calculating an image feature amount in the advertisement article area cut out from the image information, wherein the advertisement article area is smoothed to reduce noise. A step of obtaining a normalized image obtained by removing and normalizing the density of each pixel value; a step of mapping the normalized image to an HVC space and performing clustering to obtain a region-divided image; Select the class to which the vicinity of the center of the advertisement object belongs from To select an area belonging to the class Steps, After selecting an area belonging to the class, Labeling each region of the region-divided image, deleting a labeled region based on a region area ratio of the advertisement, and deleting a peripheral region of the region of the advertisement , The step of deleting After, Left Effectiveness Obtaining the advertisement feature area by obtaining a minimum circumscribed convex polygon including the area, and calculating the advertisement from the feature area calculated in the feature area calculation step. A feature amount calculation step for calculating the image feature amount for searching for provision information related to the object, and the provision information related to the advertisement based on the image feature amount calculated in the feature amount calculation step A program for causing a computer to execute a search step of searching the database for a search step and a transmission step of transmitting provision information related to the advertisement searched for in the search step to the mobile terminal. To do.
[0035]
DETAILED DESCRIPTION OF THE INVENTION
Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.
[0036]
FIG. 1 is a block diagram showing a configuration of an information providing system according to an embodiment of the present invention. The information providing system includes a wireless communication network 10 that connects the mobile device 20 and the processing center 30, and a data relay device 11 that connects the processing center 30 and the wireless communication network 10. The wireless communication network 10 is a communication network known by a name such as a PDC (Personal Digital Cellular) network, a wireless packet switching network, or IMT-2000 (International Mobile Telecommunication-2000).
[0037]
The mobile device 20 is connected to the wireless communication network 10, and a wireless communication unit 29 that establishes a wireless link and a data transmission / reception unit 21 that transmits and receives data such as images are connected to the wireless communication unit 29. . The data transmission / reception unit 21 is connected to an imaging unit 24 and an image conversion unit 23 for transmitting an image, and a display unit 28 for displaying the received image. Further, a positioning unit 25 that acquires position information is connected to the data transmitting / receiving unit 21 and the wireless communication unit 29.
[0038]
In the present embodiment, the mobile unit 20 includes the imaging unit 24. However, the mobile unit 20 and the imaging unit 24 are separate casings, and the imaging unit 24 can communicate with the image conversion unit 23 of the mobile unit 20. It is good also as a structure using.
[0039]
With such a configuration, the image captured by the imaging unit 24 is converted into a format suitable for transmission by the image conversion unit 23. The converted image data is sent to the processing center 30 via the data transmitting / receiving unit 21, the wireless communication unit 29, and the data relay device 11.
[0040]
On the other hand, the processing center 30 is connected to the wireless communication network 10 via the data relay device 11 and has a data transmitting / receiving unit 31 for transmitting and receiving data such as images. Also, an image conversion unit 33 that converts the received image data, an advertisement data collation unit 35 that collates advertisement data from the converted image, and an advertisement that searches advertisement data from the received data. And a data search unit 34.
[0041]
In addition, the processing center 30 includes an advertisement product database 36 in which advertisement product data is stored, and an advertisement product data registration unit 37 that registers the advertisement product in the advertisement product database 36.
[0042]
With such a configuration, the image data acquired by the mobile device 20 is received by the data transmission / reception unit 31 via the wireless communication network 10 and the data relay device 11, and converted to the original format by the image conversion unit 33. The The converted image data is collated with the advertisement data retrieved by the advertisement data retrieval unit 34 in the advertisement data collation unit 35.
[0043]
The positioning data acquired by the mobile device 20 is received by the data transmitting / receiving unit 31. The advertisement data search unit 34 searches for advertisement data in the vicinity of the position of the mobile device 20 from the advertisement database 36 based on the positioning data. When target data is input from the mobile device 20, the advertisement database 36 is searched based on the target data.
[0044]
Further, the advertisement product data registration unit 37 can also store the image taken by the mobile device 20 in the advertisement product database 36 via the wireless communication network 10. In this way, it is possible to register image data provided by the user, and further use character and voice information input from the user as additional information.
[0045]
The advertisement database 36 stores position information and additional information in addition to image data. The position information is input to the advertisement product database 36 as data or an address obtained by positioning the position where the image data is acquired with GPS or the like. Additional information is information provided by advertisers as well as users.
[0046]
Here, the advertising matter is one that is continuously or for a certain period and displayed to the public. The advertisement in the present invention can be considered including signs and name displays posted for purposes other than advertising.
[0047]
FIG. 2 is a flowchart illustrating an information providing method according to an embodiment of the present invention. A case where the user obtains information from the position of the mobile device and the photographed advertisement will be described. First, the mobile device 20 performs a positioning process by the positioning unit 25 to obtain position information (S21), and transmits it to the processing center 30 (S26). The positioning process uses, for example, a method using GPS, a method using a time code transmitted from a base station in IMT-2000, or a method using a location information service in PHS (Personal Handy-phone System). can do.
[0048]
In addition, the mobile device 20 converts the image captured by the imaging unit 24 (S22) into a format suitable for transmission by the image conversion unit 23 (S23), and transmits it to the processing center 30 (S26). The image may be a moving image or a still image, and is converted using a digital image compression technique such as JPEG (Joint Photographic Experts Group) or MPEG (Moving Picture Experts Group). Note that the mobile device 20 executes the positioning and the image capturing order.
[0049]
When the processing center 30 receives image data from the mobile device 20 (S27), the image conversion unit 33 converts the image data into the original format (S28). When positioning data is received from the mobile device 20 (S27), it is used for searching for advertisements in the advertisement data search unit 34.
[0050]
The processing center 30 uses the positioning data to search the advertisement database 36 for advertisements in the vicinity of the mobile device 20 that meet the user's desired information and goals (S29). The processing center 30 collates the searched advertisement with image data taken by the user (S33).
[0051]
First, a description will be given of a method of cutting out an advertisement object area from an image photographed by the imaging unit 24 of the mobile device 20.
[0052]
FIG. 3 is a diagram illustrating a method for specifying an advertisement object by zooming in the object determination method according to the embodiment of the present invention. FIG. 3A shows a state in which the user 301 captures the advertisement object 303 to be searched in the finder of the mobile device 302. FIG. 3B shows a state in which the zoom ratio is changed while capturing the center of the image of the advertising object 303 on the finder 304. As the zoom, for example, any one of an optical zoom by changing a focal length by driving a lens of a camera and a digital zoom by a scaling process of image processing may be used. In addition, the user 301 may move in the perspective direction with respect to the advertisement object 303 to be searched.
[0053]
The user 301 cuts out an area including the advertising material so that the target advertising material is contained in the frame of the finder 304 of the mobile device 302. The finder 304 of the mobile device may be realized optically or may be a display screen.
[0054]
FIG. 4 is a diagram illustrating a method for designating an advertisement by shape designation in the target discrimination method according to an embodiment of the present invention. By indicating several areas including the advertisement object to be searched, an approximate area including the advertisement object is cut out.
[0055]
FIG. 4A shows a method of cutting out with a quadrilateral shape. At this time, the frame is moved while the cursor 401 is positioned at the center of the finder 304, but the cursor 401 may be moved. In this embodiment, the area of the advertisement is cut out by instructing the upper left and lower right of the frame.
[0056]
FIG. 4B shows a method of cutting out in a circular shape. At this time, the cursor 401 is moved while the frame is positioned at the center of the finder 304, but the frame may be moved. In the present embodiment, the area of the advertisement is cut out by designating the center and radius of the frame.
[0057]
FIG. 5 is a diagram illustrating a method for specifying an advertisement in a mobile device using an HMD. A method of designating an advertisement in a virtual field of view using a mobile device that is worn on the body of the user 301 such as an HMD (Head Mounted Display) will be described. FIG. 5A shows a state in which the user 301 captures the advertisement object 303 to be searched by the mobile device using the HMD 501. FIG. 5B shows a method in which the user 301 operates the virtual cutter 503 to cut out the advertisement object 303 in the virtual field of view in the HMD 501. Further, it is possible to apply a method in which a user 301 operates a small laser pointer 502 in real space and draws a trajectory that surrounds an outline of an advertisement.
[0058]
FIG. 5C shows a method in which the user 301 cuts out several points in the area of the advertisement 303 so as to perform coloring by indicating with the virtual brush 504. At this time, the contour image that is the basis of the coloring book can be created by image segmentation processing or edge extraction processing.
[0059]
In this way, the area of the advertisement object to be cut out may be not only one but a plurality of areas, or a specific image other than the advertisement object. Moreover, although the example which a user performs in the field was shown, you may carry out on the computer of a remote place.
[0060]
Next, a method for obtaining an advertisement feature area for calculating an image feature amount, which will be described later, from the advertisement article area cut out by the above-described method will be sequentially described. It goes without saying that this method can extract a specific image from arbitrary image information and obtain a feature region for calculating an image feature amount.
[0061]
(1) Noise removal and density normalization
The image is smoothed using a Gaussian filter with appropriate parameters, and the density of each pixel value in the image is normalized.
[0062]
(2) Clustering in uniform perceptual space
The RGB signals obtained from the imaging unit are mapped to a modified Munsell color space (hereinafter referred to as an HVC space) in which human color perception is uniform. Thereafter, the HVC space is clustered to obtain a region-divided image. As the clustering method, a known fuzzy c-means clustering method or k-means clustering method can be used.
[0063]
(3) Selection of the class to which the vicinity of the image center belongs
With appropriate parameters Cx, Cy (an integer greater than 1), W is the horizontal size of the image and H is the vertical size of the image.
W (1-1 / Cx) / 2 ≤ x ≤ W (1 + 1 / Cx) / 2
H (1-1 / Cy) / 2 ≤ y ≤ H (1 + 1 / Cy) / 2
The (x, y) image region that satisfies the condition corresponds to the region of the central portion having a size obtained by equally dividing the vertical and horizontal directions of the image by Cx and Cy, and a class included in this is selected from the entire image. In the image center region and other regions, regions belonging to the same class are selected.
[0064]
(4) Spatial labeling process
Label each area of the entire image.
[0065]
(5) Deletion of image periphery and minute area by area area ratio
In each labeled area, a very minute area of the image is removed from the total area of the image. With an appropriate parameter bx, by (an integer greater than 1), W is the horizontal size of the image, and H is the vertical size of the image.
0 ≤ x <W / bx or W (1-1 / bx) <x <W
0 ≤ y <H / by or H (1-1 / by) <x <H
An image area satisfying (x, y) corresponds to the area of the peripheral portion obtained by equally dividing the vertical and horizontal directions of the image by bx and by, but the area including this is deleted.
[0066]
(6) Single region extraction by circumscribed polygon calculation
Find the minimum circumscribed convex polygon that includes all remaining regions. This can be solved by a two-dimensional convex hull problem.
[0067]
Next, a process for calculating the image feature amount from the advertisement feature area automatically cut out will be described. The advertisement database 36 of the processing center 30 stores the obtained image data together with the result of obtaining the image feature amount described above.
[0068]
(1) Image feature amount related to color
The image feature amount relating to color includes a high chromatic area ratio and a hue average value. In the HVC space, the amount of color is defined as the product of brightness and saturation, and the region where the amount of color exceeds an appropriate threshold is defined as a high color region. That is, it indicates an area where the lightness or saturation is high and the hue value is sufficiently meaningful. The ratio of the high color area to all the single areas is defined as the high color area ratio. Further, the hue average value is an average value of hues in all the high color areas in a single area.
[0069]
FIG. 6 is a diagram showing an example of the distribution of high chromatic area ratio and hue average value plotted for each photographing position. There are five shooting positions, L1 to L5, and represents the distribution of image feature values related to the color of the image at each shooting position. Plotting is performed for each image shown in lowercase letters with the r-axis as the high chromatic area ratio and the θ-axis as the hue average value. All indicates the result of superimposing the image feature amounts at five locations.
[0070]
The numerical value at the lower left of each pie chart represents the average distance to the closest point at each sample point in the feature space. It can be seen that the feature amount distribution at each photographing position is wider than the whole, and an improvement in the discrimination rate is expected.
[0071]
(2) Image feature values related to external shape
The image feature amount related to the outer shape has a region aspect ratio. Find the smallest circumscribed quadrilateral that circumscribes a circumscribed convex polygon. As a simple method, the circumscribed quadrilateral can be obtained by obtaining the maximum value and the minimum value from the coordinate value (x, y) at the contour of the minimum circumscribed convex polygon. In the obtained circumscribed quadrilateral, the area aspect ratio (AR) is
AR = (width) ² /(area)
It is represented by
[0072]
FIG. 7 is a diagram showing an example of the distribution of the area aspect ratio plotted for each photographing position. There are five shooting positions from L1 to L5, which represent the distribution of the area aspect ratio at each shooting position. The horizontal axis is the area aspect ratio, and each image shown in lowercase letters is plotted. All shows the result of superimposing the five area aspect ratios.
[0073]
In the feature space, the distance between nearest points having the smallest distance between arbitrary images is called the nearest point distance. When the average of the closest contact distance is obtained using the feature amount shown in FIG. 6, even if the image is equal to or less than the average value, the feature amount shown in FIG. When the average is obtained, it may be equal to or greater than the average value, and even if it is difficult to determine only by color information, it may be determined by outline information.
[0074]
Next, the collation (S33) between the acquired image and the advertisement data in the flowchart shown in FIG. 2 will be described.
[0075]
The processing center 30 also obtains the image feature amount for the acquired image by the method described above. In each feature space of the advertisement product database 36, the distance from the image feature amount of the acquired image to the feature point of the image data registered in the advertisement product database 36 is defined as Dfi. Here, 1 ≦ f ≦ N, N is the number of feature spaces, 1 ≦ i ≦ Fn, and Fn is the number of registered images in the feature space f. At this time, the evaluation function
[0076]
[Expression 1]

[0077]
Are displayed on the display unit 28 of the mobile device 20 as similar advertisements. Or
[0078]
[Expression 2]

[0079]
As described above, in all feature spaces, it is also possible to treat images having neighboring feature points as similar advertisements.
[0080]
FIG. 8 is a diagram for explaining a method of using the information providing system according to the embodiment of the present invention. The user collates image data obtained by photographing a scene of a street corner with information stored in the advertisement database 36, detects an advertisement included in the scene, and detects information about the detected advertisement as advertisement. The data is transmitted from the database 36 to the mobile device 20. Information relating to the detected advertisement is displayed superimposed on the image data displayed on the display unit 28 of the mobile device 20, and the user can easily check the store and location information.
[0081]
【The invention's effect】
As described above, according to the present invention, target information required by the user can be easily obtained by selecting and providing information related to the image information sent from the mobile device from the image feature amount. Is possible.
[Brief description of the drawings]
FIG. 1 is a block diagram showing a configuration of an information providing system according to an embodiment of the present invention.
FIG. 2 is a flowchart illustrating an information providing method according to an embodiment of the present invention.
FIG. 3 is a diagram showing a method for specifying an advertisement object by zooming in the object discrimination method according to the embodiment of the present invention.
FIG. 4 is a diagram showing a method for designating an advertisement by shape designation in an object discrimination method according to an embodiment of the present invention.
FIG. 5 is a diagram showing a method for specifying an advertisement in a mobile device using an HMD.
FIG. 6 is a diagram illustrating an example of a distribution of high chromatic area ratios and hue average values plotted for each photographing position.
FIG. 7 is a diagram showing an example of a distribution of area aspect ratios plotted for each photographing position.
FIG. 8 is a diagram for explaining a method of using the information providing system according to the embodiment of the present invention.
[Explanation of symbols]
10 Wireless communication network
11 Data relay device
20,302 Mobile
21, 31 Data transceiver
22 Voice transmitter
23, 33 Image converter
24 Imaging unit
25 Positioning Department
28 Display section
29 Wireless communication unit
30 processing center
34 Advertisement data search section
35 Advertisement data verification part
36 advertisement database
37 Advertisement Data Registration Department
301 users
303 Advertisement
304 Finder
401 cursor
501 HMD (Head Mounted Display)
502 Laser pointer
503 Virtual cutter
504 Virtual brush

Claims (26)

In a feature region extraction method for extracting a specific image from image information captured by an imaging unit of a mobile terminal and obtaining a feature region for calculating an image feature amount,
Smoothing a region of the specific image to remove noise and obtaining a normalized image in which the density of each pixel value is normalized;
Mapping the normalized image to an HVC space to perform clustering to obtain a region-divided image;
By selecting the center near belongs class area in the predetermined image from among the area dividing the whole image, selecting a region belonging to the class,
Labeling each region of the region divided image after selecting a region belonging to the class ; and
For each of the labeled regions, deleting a minute region according to a region area ratio of the specific image, and deleting a peripheral region of the region of the specific image;
After the step of the deletion, by obtaining the minimum circumscribed convex polygon containing the active space left, the feature region extraction method characterized by comprising the steps of obtaining the feature region. From said specific image extracted using the zoom function of the mobile terminal, the feature region extraction method according to claim 1, characterized in that determining the feature region. From the specific image extracted by specifying a shape in the display device of the mobile terminal, the feature region extraction method according to claim 1, characterized in that determining the feature region. The feature region according to claim 1, wherein the mobile terminal is a body-mounted type, and the feature region is obtained from a specific image cut out in a virtual field of view in the display device of the mobile terminal. Extraction method. In an information providing method in which an advertisement is extracted from image information received from a mobile terminal, and an information providing server retrieves and provides provided information related to the advertisement from a database.
A feature region calculation step of obtaining an advertisement feature region for calculating an image feature amount in the region of the advertisement product cut out from the image information;
A feature amount calculating step for calculating the image feature amount for searching for provision information related to the advertisement object from the advertisement object feature region obtained in the feature area calculating step;
A search step of searching provided information related to the advertising matter from the database based on the image feature amount calculated in the feature amount calculating step;
Transmitting the provision information related to the advertisement searched in the search step to the mobile terminal, and the feature region calculating step includes:
Smoothing the area of the advertising material to remove noise and obtaining a normalized image in which the density of each pixel value is normalized;
Mapping the normalized image to an HVC space to perform clustering to obtain a region-divided image;
By selecting the center near belongs class area of the advertising material from the area dividing the whole image, selecting a region belonging to the class,
Labeling each region of the region divided image after selecting a region belonging to the class ; and
For each of the labeled regions, deleting a minute region according to a region area ratio of the advertisement, and deleting a peripheral region of the region of the advertisement,
After the step of the deletion, by obtaining the minimum circumscribed convex polygon containing the active space left, the information providing method characterized by including the step of determining the advertising material characteristic region.   The information providing method according to claim 5, wherein the area of the advertisement is cut out using a zoom function of the mobile terminal.   6. The information providing method according to claim 5, wherein the area of the advertisement is cut out by designating a shape on a display device of the mobile terminal.   6. The information providing method according to claim 5, wherein the area of the advertisement is cut out in a virtual field of view on the display device of the mobile terminal mounted on the body. The information according to any one of claims 5 to 8 , wherein the search step searches for small provided information of a predetermined evaluation function in each feature space of the database based on the image feature amount. How to provide. Wherein the image characteristic amount, a high color region ratio and hue average values for color, the information providing method according to any one of claims 5 to 9, characterized in that any area aspect ratio for profile. In a feature region extraction apparatus for extracting a specific image from image information captured by an imaging unit of a mobile terminal and obtaining a feature region for calculating an image feature amount,
Means for smoothing a region of the specific image to remove noise and obtaining a normalized image in which the density of each pixel value is normalized;
Means for mapping the normalized image to an HVC space and performing clustering to obtain a region-divided image;
Means for selecting a region belonging to the class by selecting a class to which the vicinity of the center of the region of the specific image belongs from the entire region-divided image;
Means for labeling each region of the region-divided image after a region belonging to the class is selected ;
For each of the labeled areas, means for deleting a micro area according to the area area ratio of the specific image, and deleting a peripheral area of the area of the specific image;
A feature region extraction apparatus comprising: means for obtaining the feature region by obtaining a minimum circumscribed convex polygon including a remaining effective region after the peripheral region is deleted . From said specific image extracted using the zoom function of the mobile terminal, the characteristic region extracting device according to claim 11, characterized in that determining the feature region. From the specific image extracted by specifying a shape in the display device of the mobile terminal, the characteristic region extracting device according to claim 11, characterized in that determining the feature region. The feature region according to claim 11 , wherein the mobile terminal is a body-mounted type, and the feature region is obtained from a specific image cut out in a virtual field of view in the display device of the mobile terminal. Extraction device. In an information providing apparatus that extracts an advertisement from image information received from a mobile terminal and retrieves and provides provided information related to the advertisement from a database.
In the area of the advertisement object cut out from the image information, a feature area calculation means for obtaining an advertisement object feature area for calculating an image feature amount;
Feature amount calculating means for calculating the image feature amount for searching for provision information related to the advertisement object from the advertisement object feature area obtained by the feature area calculation means;
Search means for searching provided information related to the advertising matter from the database based on the image feature quantity calculated by the feature quantity calculating means;
Transmitting means for transmitting provided information related to the advertisement searched for by the search means to the mobile terminal, and the feature area calculating means includes:
Means for smoothing the area of the advertising matter to remove noise and obtaining a normalized image in which the density of each pixel value is normalized;
Means for mapping the normalized image to an HVC space and performing clustering to obtain a region-divided image;
Means for selecting a region belonging to the class by selecting a class to which the vicinity of the center of the region of the advertisement object belongs from the entire region-divided image;
Means for labeling each region of the region-divided image after a region belonging to the class is selected ;
For each of the labeled areas, means for deleting a micro area according to the area area ratio of the advertisement object, and deleting a peripheral area of the area of the advertisement object;
An information providing apparatus comprising: means for obtaining the advertisement feature region by obtaining a minimum circumscribed convex polygon including a remaining effective region after the peripheral region is deleted . The information providing apparatus according to claim 15 , wherein the area of the advertisement is cut out using a zoom function of the mobile terminal. The information providing apparatus according to claim 15 , wherein the area of the advertising matter is cut out by designating a shape on a display device of the mobile terminal. 16. The information providing apparatus according to claim 15 , wherein the area of the advertisement is cut out in a virtual field of view in the display device of the body-mounted mobile terminal. The search unit according to any one of claims 15 to 18 , characterized in that, based on the image feature amount, search information provided in a feature space of the database having a predetermined evaluation function smaller than a predetermined value. The information providing device according to 1. The transmission means rearranges and transmits provided information related to the advertisement based on position information received from the mobile terminal, in order from the provided information having position information close to the mobile terminal. An information providing apparatus according to any one of claims 15 to 18 . Wherein the image information received from the mobile terminal, together with the image feature amount, information providing apparatus according to any one of claims 15 to 20, characterized in that it comprises a storage means for storing in the database. The information providing apparatus according to any one of claims 15 to 21 , wherein the image feature amount is one of a high chromatic area ratio and a hue average value relating to a color, and an area aspect ratio relating to an outer shape. A recording medium that records a program for controlling a feature region extraction device that extracts a specific image from image information captured by an imaging unit of a mobile terminal and calculates a feature region for calculating an image feature amount,
Smoothing a region of the specific image to remove noise and obtaining a normalized image in which the density of each pixel value is normalized;
Mapping the normalized image to an HVC space to perform clustering to obtain a region-divided image;
By selecting the center near belongs class area in the predetermined image from among the area dividing the whole image, selecting a region belonging to the class,
Labeling each region of the region-divided image after the region belonging to the class is selected ;
For each of the labeled regions, deleting a minute region according to a region area ratio of the specific image, and deleting a peripheral region of the region of the specific image;
After the step of the deletion, by obtaining the minimum circumscribed convex polygon containing the active space left, computer-readable recording medium storing a program for executing the steps of obtaining the feature region on the computer. A recording medium that records a program for controlling an information providing apparatus that extracts an advertisement from image information received from a mobile terminal and retrieves and provides provided information related to the advertisement from a database,
A feature region calculating step for obtaining an advertisement feature region for calculating an image feature amount in the region of the advertisement product cut out from the image information,
Smoothing the area of the advertising material to remove noise and obtaining a normalized image in which the density of each pixel value is normalized;
Mapping the normalized image to an HVC space to perform clustering to obtain a region-divided image;
By selecting the center near belongs class area of the advertising material from the area dividing the whole image, selecting a region belonging to the class,
Labeling each region of the region divided image after selecting a region belonging to the class ; and
For each of the labeled regions, deleting a minute region according to a region area ratio of the advertisement, and deleting a peripheral region of the region of the advertisement,
After the step of the deletion, by obtaining the minimum circumscribed convex polygon containing the active region left, and determining the advertising material characteristic region, and the feature region calculating step,
A feature amount calculating step for calculating the image feature amount for searching for provision information related to the advertisement object from the advertisement object feature region obtained in the feature area calculating step;
A search step of searching provided information related to the advertising matter from the database based on the image feature amount calculated in the feature amount calculating step;
A computer-readable recording medium having recorded thereon a program for causing a computer to execute a transmission step of transmitting provision information related to the advertisement searched for in the search step to the mobile terminal. A program for extracting a specific image from image information captured by an imaging unit of a mobile terminal and controlling a feature region extraction device for obtaining a feature region for calculating an image feature amount,
Smoothing a region of the specific image to remove noise and obtaining a normalized image in which the density of each pixel value is normalized;
Mapping the normalized image to an HVC space to perform clustering to obtain a region-divided image;
By selecting the center near belongs class area in the predetermined image from among the area dividing the whole image, selecting a region belonging to the class,
Labeling each region of the region divided image after selecting a region belonging to the class ; and
For each of the labeled regions, deleting a minute region according to a region area ratio of the specific image, and deleting a peripheral region of the region of the specific image;
After the step of the deletion, by obtaining the minimum circumscribed convex polygon containing the active space left, the program for executing the steps of obtaining the feature region on the computer. An information providing apparatus program that extracts an advertisement from image information received from a mobile terminal, and retrieves and provides provision information related to the advertisement from a database,
A feature region calculating step for obtaining an advertisement feature region for calculating an image feature amount in the region of the advertisement product cut out from the image information,
Smoothing the area of the advertising material to remove noise and obtaining a normalized image in which the density of each pixel value is normalized;
Mapping the normalized image to an HVC space to perform clustering to obtain a region-divided image;
By selecting the center near belongs class area of the advertising material from the area dividing the whole image, selecting a region belonging to the class,
Labeling each region of the region divided image after selecting a region belonging to the class ; and
For each of the labeled regions, deleting a minute region according to a region area ratio of the advertisement, and deleting a peripheral region of the region of the advertisement,
After the step of deleting, the feature area calculating step includes the step of determining the advertisement feature area by determining a minimum circumscribed convex polygon including a remaining effective area;
A feature amount calculating step for calculating the image feature amount for searching for provision information related to the advertisement object from the advertisement object feature region obtained in the feature area calculating step;
A search step of searching provided information related to the advertising matter from the database based on the image feature amount calculated in the feature amount calculating step;
A program for causing a computer to execute a transmission step of transmitting provision information related to the advertisement searched for in the search step to the mobile terminal.

Images