JP6210554B2 - Recognition device, recognition program, and recognition method - Google Patents

Description

  The present invention relates to a recognition apparatus, a data server, a recognition program, and a recognition method that can recognize a surrounding object or the like by keeping reference data in object recognition at an appropriate size.

  In image recognition, a technique for recognizing an object in a query image by comparing a feature amount of an image input as a query image with a feature amount of a reference image registered in a database is known. However, when performing object recognition using this technology, it is necessary to store a feature image database of reference images in memory, but it is impossible to store reference image data of any object in the world. It is. By sending input features and feature quantities calculated from the input images to the server and performing object recognition on the server, it is possible to use a huge memory on the server, but in that case the amount of communication with the server increases. Resulting in.

  With respect to this problem, in Patent Document 1, focusing on the fact that an object existing in real space is biased depending on the location, reference image data is selected based on the current position information and azimuth information, and stored in the memory. The size of the stored reference image DB (database) is being reduced.

JP 2011-242861 A (Object recognition apparatus, object recognition system, and object recognition method)

  As described above, in Patent Document 1, the reference image DB size is reduced based on current position information and orientation information. However, in reality, there are many personal items that are carried around by individuals and objects around them that are not associated with location information. It is difficult to make a choice.

  For example, since it is assumed that the personal items carried by each individual above appear at any position where the individual moves, as is clear from the carrying, the reference image data about the personal item based on the current position information and direction information It is difficult to choose.

  The present invention provides a recognition system, a data server, a recognition program, and a recognition method capable of reducing the data size by selecting reference data even if it is a private item in view of the above-described problems of the prior art. .

  In order to achieve the above object, the present invention provides a recognition device, a photographing unit that captures an image, an article information acquisition unit that acquires article information as the name of an article existing around the user, and the acquired article. A management unit that manages a feature amount corresponding to information; and a recognition unit that recognizes an article included in an image captured by the imaging unit based on the managed feature amount. The management unit obtains the feature quantity corresponding to the acquired article information by transmitting the obtained article information and receiving the corresponding feature quantity to the data server that holds and holds Management is the first feature.

  Further, the present invention is a data server that associates and holds the name and feature amount of each article, and receives the article information acquired and transmitted by the recognition device, and corresponds to the article information. A second feature is that the amount is returned to the recognition device.

  A third feature of the present invention is a recognition program that causes a computer to function as the recognition device.

Further, the present invention is characterized in that, in the recognition device, the article information acquisition unit acquires article information as a product name by at least one of the following methods (1) to (3).
(1) A method for extracting a product name by analyzing an image of a receipt photographed by the photographing unit. (2) By analyzing a code in a product image provided with a code photographed by the photographing unit, Method of obtaining the corresponding product name (3) Method of extracting the purchased product name from the user's electronic commerce history

Furthermore, the present invention provides a data server, a step of holding in association with the names and characteristics of each article, the recognition apparatus shadow facilities Taking for capturing an image, as the name of the article around the user An article information acquisition step for acquiring article information, a management step for managing a feature quantity corresponding to the acquired article information, and an article included in an image captured by the imaging unit based on the managed feature quantity A recognition step, wherein the management step corresponds to the acquired article information by transmitting the acquired article information to the data server and receiving a corresponding feature quantity. The fifth feature is to acquire and manage the feature amount.

  According to the first, second, third, or fifth feature, since only the feature quantity corresponding to the article information is acquired from the data server as the name of the article existing around the user, the reference is performed. It is possible to keep the data size of the feature quantity as data at an appropriate size.

  According to the fourth feature, the article information can be easily acquired as the product name by at least one of the methods (1) to (3).

It is a functional block diagram of the recognition system concerning one embodiment. It is a flowchart of operation | movement of the recognition system which concerns on one Embodiment. It is a detailed flowchart of the 1st step (reference data acquisition) of FIG. It is a detailed flowchart of the 2nd step (recognition process etc.) of FIG. It is a figure which shows the example of content presentation. It is a functional block diagram for explaining an update process and a data sharing process including the detail of a management part.

  FIG. 1 is a functional block diagram of a recognition system according to an embodiment. The recognition system 100 includes a recognition device 10, 10B that can be realized as a head-mounted display (HMD) type terminal for each user, a smartphone or other portable information terminal, and a data server 15.

  In FIG. 1, the recognition devices 10 and 10B are shown for only two users, but the recognition system 100 can have any number of recognition devices for any number of users. In addition, although each user's recognition apparatus can perform a common process by providing a common functional unit, in the description of the present invention, the assumed “self” user recognition apparatus is assumed to be the recognition apparatus 10. A recognition device of an “other” user (an arbitrary “other” user as viewed from the “self” user) will be described as a recognition device 10B.

  The recognition device 10 includes a photographing unit 21, a purchase information acquisition unit 22, an article information acquisition unit 25, a transmission unit 26, a reception unit 35, a management unit 40, a proximity user identification unit 45, a recognition unit 50, a presentation unit 60, and a reception unit. With 65. The data server 15 includes a server side reception unit 31, a data acquisition unit 32, a reference DB (database) 33, and a server side transmission unit.

  In addition, in order to show the exchange between each recognition apparatus 10 and 10B and the data server 15, in FIG. 1, in addition to the own user's recognition apparatus 10 and the data server 15, the recognition apparatus 10B of another user is shown. Therefore, as described above, the other user's recognition device 10B has the same configuration as that of the own user's recognition device 10, but the detailed illustration of the same functional block is omitted in FIG. doing. The function outline of each part is as follows.

  The imaging unit 21 captures an image and passes it to the article information acquisition unit 25 and the recognition unit 50. The purchase information acquisition unit 22 acquires purchase information and passes it to the article information acquisition unit 25. The article information acquisition unit 25 acquires the article information from the information passed from the photographing unit 21 and / or the purchase information acquisition unit 22 and passes it to the transmission unit 26 so that the article information is transmitted to the server side reception unit 31. Sent.

  The data acquisition unit 32 searches the reference DB 33 based on the article information transmitted and received by the server-side reception unit 31, acquires the feature amount, and passes the feature amount to the server-side transmission unit 34 for transmission. Thus, the feature amount or the like is received by the receiving unit 35.

  The management unit 40 manages the received feature amount and the like by the reception unit 35 and provides the reference for the recognition unit 50 and the like. The recognition unit 50 recognizes an article in the image photographed by the photographing unit 21 by using the managed feature amount as a reference. The presentation unit 60 presents content to the user based on the recognized result. Note that content managed by the management unit 40 is referred to. The accepting unit 65 receives a response to the presented content from the user.

  In addition to managing the feature quantities as described above, the management section 40 manages the content to be presented by the presentation section 60 as described above, and also manages a series of histories. That is, by managing the recognition result by the recognition unit 50 and the received response by the reception unit 65, the management unit 40 can display a series of recognized article histories and a series of presented content histories. to manage. Based on the history, the management unit 40 deletes and manages the managed feature quantities that are determined to be less necessary to be stored.

  The proximity user specifying unit 45 specifies another user that exists in the vicinity of the user and has the recognition device 10B. Data sharing processing can be performed by mutually exchanging the same type of feature quantity acquired from the data server 15 with the identified proximity user recognition device 10B.

  FIG. 2 is a flowchart of the operation of the recognition system 100 according to an embodiment. In step S1, the recognition device 10 transmits the article information to the data server 15 and receives the feature quantities and the like of each article from the data server 15, whereby the management section 40 acquires the feature quantities and the like as reference data. . In step S2, recognition processing or the like is performed when the recognition device 10 functions the recognition unit 50 or the like by referring to the feature amount as reference data acquired by the management unit 40.

  In order to execute step S2, it is necessary to execute step S1 at least once and prepare reference data in the recognition apparatus 10, but in parallel with the execution of step S2, step S1 It is also possible to continue step S2 while replenishing the reference data in the recognition apparatus 10 by appropriately executing.

  Hereinafter, the details of the steps S1 and S2 and the functional units in FIG. 1 will be described.

  FIG. 3 is a detailed flowchart of step S1 in FIG. In step S11, after the article information acquisition unit 25 acquires the article information and the accumulation reaches a certain amount, the article information is transmitted from the transmission unit 26 to the data server 15. The article information is information on the name of an article that is assumed to be around the user of the recognition device 10, particularly, for example, the name of a commodity or other commodity. The acquisition of the article information can be any one of the following first to fourth embodiments or any combination thereof. In addition, when using several embodiment together, according to user operation etc., it is possible to apply an individual embodiment for every time which acquires article information. For example, the article information “A” is acquired in the first embodiment for the first time, the article information “B” is acquired in the second embodiment for the second time, and the article information “C” is acquired in the third embodiment for the third time. In this case, the article information “A, B, C” listing them is acquired from the series of acquisition histories.

  In the first embodiment, in the photographing unit 21 that can be realized as a camera provided in the user terminal (recognition device 10), a receipt of a product received by a user through purchase behavior is photographed under a user operation, and an image of the receipt By performing a well-known OCR (optical character recognition) process on the article information acquisition unit 25, the article information can be acquired as the product name purchased by the user. The receipt may be a purchase made by the user or the like.

  In the second embodiment, a code such as a barcode in a product purchased by the user (or the relative) is photographed by the photographing unit 21 by a user operation, and an image of the code is analyzed by the article information acquiring unit 25. The article information as the purchased product name may be acquired. Here, for the code analysis, a known method corresponding to the type of the code can be used. In addition, when photographing an image of a code such as a barcode on a product in this way, the user does not necessarily have to purchase the product, and the image is acquired when the user takes it with interest. It is also possible.

  In the third embodiment, the article information as the product name purchased by the user as in the first and second embodiments is directly from the purchase information acquisition unit 22 instead of the image acquired by the imaging unit 21. The acquisition unit 25 may acquire it. Specifically, the purchase information acquisition unit 22 acquires the item information by referring to the user's electronic commerce history such as the user's electronic money use history and the net shopping history, and passes the item information to the item information acquisition unit 25.

  In the fourth embodiment, the article information as the article name (including the case of the article name) is entered by manually inputting the input means (not shown in FIG. 1) such as a touch panel or a keypad by the user. Acquired by the information acquisition unit 25.

  In addition, when the article is a commodity, a commodity name as in each of the above embodiments (a commodity name as a target to be registered in advance by a manager or the like in the reference DB 33 described later) can be easily prepared. The present invention can be implemented even for general articles that are not limited to commodities, as long as the article names are prepared and registered in advance in the reference DB 33 in association with other information. For example, road signs that are not generally purchased items as daily necessities can also be used as articles in the present invention, and the article information acquisition unit 25 acquires the article name by some method including manual input from the user (fourth embodiment). do it. In addition, when using goods, since articles belonging to the same type can be distinguished by the name of the goods, it is possible to carry out recognition processing and content presentation according to the present invention by distinguishing various daily necessities. . Also, the article is preferably mass-produced.

  In step S12, the data server 15 receives the article information acquired and transmitted in the above step S11 at the server-side receiving unit 31, and the data acquisition unit 32 searches the reference DB 33 using the article information as a key for each article. A corresponding feature amount or the like is obtained, and the feature amount or the like is returned from the server-side transmission unit 34 to the reception unit 35 of the recognition apparatus 10.

In order to enable the search, the reference DB 33 associates each article name N with each article name N, and various kinds of spatial angles A _i (i = 1, 2,..., _{M N} ; m _N represents each article. Stored are feature values F _{i, N} acquired from images G _{i, N} taken in a predetermined number for each name N). In this way, the data acquisition unit 32 acquires the feature amounts F _{i, N} corresponding to the article name N by searching the reference DB 33 using each article name N included in the article information transmitted from the recognition device 10 as a query. can do.

Here, the feature values F _{i, N} stored in the reference DB 33 are the same kind of feature values extracted from the image by the recognition unit 50 described later, and for example, a well-known SIFT feature value is adopted. can do. Further, the feature amounts Fi _{, N} may be stored in the reference DB 33 in the form of textured 3D model data. At this time, SIFT and other feature quantities are extracted from the texture. In this format as well, as in the case of photographing at various angles, the recognition unit 50 can recognize an article at an arbitrary viewpoint (various spatial angles A _i ).

In the reference DB 33, in order to realize a well-known AR (augmented reality) display in the presentation unit 60 described later, the feature quantity F _{i, N} is further linked to the three-dimensional model data of the article. It may be left. If the feature amounts _{Fi, N} are given in the form of textured 3D model data as described above, it is not necessary to further associate 3D model data for AR display in this way. Further, in the reference DB 33, in order to realize content presentation associated with an article in the presentation unit 60 described later, the content to be presented may be associated with each article name and held. In this way, when used by the presentation unit 60, the 3D model data and content are returned to the reception unit 35 of the recognition apparatus 10 together with the feature amounts _{Fi, N.}

  In addition, generally, the reference DB 33 stores in advance data such as feature quantities for a large number of article names by the administrator's work, etc., and among these, the recognition apparatus 10 corresponds to the article information acquired by itself. Only a partial feature amount or the like is received. Therefore, in the present invention, the recognition apparatus 10 can recognize the articles around us while keeping the reference data size at an appropriate size.

  In step S13, the feature quantity or the like as a reference acquired in the data server 15 in the above step S12 is received by the receiving unit 35 and stored in the management unit 40. In addition to the feature amount, when 3D model data or content is acquired as described above, these are also stored in the management unit 40 together.

  In the reference DB 33, a plurality of contents may be associated with each article name and stored. When the user acquires content as being associated with the article name in step S13, the user selects and acquires one or more of the associated content and stores it in the management unit 40 Alternatively, the content to be acquired may be automatically selected according to the group, community, or other user attribute to which the user belongs.

  As described above, the flow of FIG. 3 has been described as being performed collectively after a predetermined amount of product information is obtained in step S11, that is, after a predetermined number of product names are accumulated, the product information acquisition unit Each time the article information is acquired at 25, the flow of FIG. In this case, each time the article information is acquired, the corresponding feature amount or the like is acquired and stored in the management unit 40.

  FIG. 4 is a detailed flowchart of step S2 in FIG.

  In step S21, it is confirmed whether or not an image has been acquired in the photographing unit 21, and if acquired, the process proceeds to step S22, and if not acquired, the process waits in step S21 until it is acquired. Here, if the photographing unit 21 is realized as a camera provided in the HMD, it is possible to reach step S22 for each image of the first person video obtained in real time by the user wearing the HMD. Although it is possible, in the case of other cameras, the same can be done by user operation. Alternatively, only the image explicitly designated by the user may reach step S22.

  In step S22, the recognition unit 50 performs a recognition process on the acquired image, and identifies an article included in the image. The identification is performed by extracting a feature amount from the image and identifying which feature amount is most similar to the feature amount managed as reference data by the management unit 40. A product identification result can be obtained as the product name associated with the feature quantity. Here, well-known methods can be used for extracting feature amounts, calculating similarity of feature amounts, and the like.

Further, as described above in the description of the reference DB 33, the feature amount is obtained when the article N is viewed from a plurality of spatial angles A _i ((i = 1, 2,..., _{M N} ) for each article N. is managed in the management unit 40 as the feature amount F _{i, N} the. Therefore, the recognition unit 50, recognizes the article N in the image, which the article N saw in any angle a _i in the image That is, the recognition unit 50 can recognize the article N at each angle A _i .

  The recognition result by the recognition unit 50 is stored in the management unit 40, and the management unit 40 manages a series of past recognition results as a history.

  In step S22, only the identification of the article may be performed as described above, but in one embodiment, the presentation unit 60 may further present the content based on the identification result. Here, the content may be presented by acquiring predetermined content corresponding to the identified article from the management unit 40. At this time, the content can be presented in a form that is superimposed on the image captured by the image capturing unit 21 using a known AR technique. Also, by preparing in the reference DB 33 in advance, it is possible to present arbitrary content including formats such as audio other than images and video.

  FIG. 5 is a diagram illustrating an example of content presentation. The image G1 obtained by the photographing unit 21 includes a product “OX Milk” as an article as shown in (1), and the recognition unit 50 recognizes the product. (2) and (3) show examples in which English-language learning content by AR display is presented as content of predetermined content linked in advance to the article “OX milk” based on the recognition result.

  In (2) of FIG. 5, the user is asked what the word related to the bottle of “OX milk” is based on the superimposed display A2 on the image G1. In (3), the answer is also shown to the user as a superimposed display A3 on the image G1.

  When the content is such as to receive a response such as accepting an answer from the user, as in the above example, the response can be received by the receiving unit 65, and the presentation unit 60 can receive the response based on the response content. Can continue to present content. For example, in the case of the above-mentioned English word learning content, the user can input an answer and present the correct / incorrect answer distinction.

  When the presentation unit 60 presents content, the management unit 40 manages the presented content as a history, and when there is a response, the management unit 40 also stores and manages the response.

  When step S22 as described above is completed, the process proceeds to step S23. If the similarity in the most similar feature amount is equal to or less than the threshold value in step S22, a result that the image managed by the management unit 40 is not included in the image is obtained. The process proceeds to step S23.

  In step S23, the management unit 40 determines whether or not the data managed by the management unit 40 at that time can be updated. If the management unit 40 determines that the data can be updated, the process proceeds to step S24 to update the data. If not, the process returns to step S21. The update process in steps S23 and S24 is as follows, and is realized as a portable terminal or the like by deleting and updating data that is judged to be outdated or has been used by the management unit 40. The recognition device 10 can efficiently use the memory area.

  FIG. 6 is a functional block diagram including a functional unit responsible for the update processing in the management unit 40. The management unit 40 includes a storage unit 41, a period monitoring unit 42, a frequency monitoring unit 43, a deletion unit 44, and a disclosure setting unit 49.

  In FIG. 6, the transmission / reception unit 46 and the authentication unit 47 included in the recognition device 10 are also illustrated as a configuration that is not illustrated in FIG. 1, but these (and the disclosure setting unit 49) are also provided by other users. This is referred to when performing data sharing processing described later with the recognition device 10B. That is, FIG. 6 is a functional block diagram for explaining the update process and the data sharing process. The update process is performed by the deletion unit 44 as follows.

  The storage unit 41 is responsible for the storage function in the management unit 40, and stores the feature amount, content, 3D data model, and the like associated with each article name to be stored.

  The period monitoring unit 42 monitors the duration of the feature quantity for each article name stored in the storage unit 41 and notifies the deletion unit 44 of the article name whose duration has reached a certain value. The monitored continuation period is a continuation period from the time when it is stored in the storage unit 41 (that is, an elapsed period after being acquired from the data server 15 or obtained by sharing processing described later), or used. This is a duration of the time that the item has not been made (that is, that the corresponding article has not been recognized by the recognition unit 50).

  The frequency monitoring unit 43 monitors the frequency of use of the feature quantity for each article name stored in the storage unit 41, that is, the frequency at which the corresponding article is recognized by the recognition unit 50, and the usage frequency is The article name that has become a certain value or less is notified to the deletion unit 44.

  The deletion unit 44 deletes, from the storage unit 41, data related to the article name notified from the period monitoring unit 42 and / or the frequency monitoring unit 43 as described above (feature amount, content, 3D data model, etc.). Thus, in step S23 of FIG. 4, whether or not a new notification has been received from the period monitoring unit 42 and / or the frequency monitoring unit 43 until the time (from the time when the previous step S23 was reached to the time of the current step S23). Is determined, and when there is a new notification, the deletion unit 44 performs a corresponding deletion process in step S24.

  Note that the threshold for notification determination in each of the period monitoring unit 42 and the frequency monitoring unit 43 can be set as a predetermined value for each article. For example, if it is the period threshold in the period monitoring unit 42, the item corresponding to food and drink is 3 days, the item corresponding to stationery is 1 month, etc. It can be set accordingly.

  In the deletion unit 44, as another embodiment different from the determination based on the period and frequency relating to the use as described above, when the use of the content linked to the product name in the storage unit 41 ends, Data related to the name may be deleted from the storage unit 41. For example, if learning content is linked to a certain article name, a series of data linked to the article name may be deleted when learning by the learning content is completed by the user.

  In the above, each step of FIG. 4 was demonstrated. Next, sharing of reference data between users according to the present invention, which can be performed as needed when requested by a user, will be described. This sharing makes it possible to omit or reinforce the acquisition of the reference data from the data server 15 after the article information is accumulated in the individual user, which is described as step S1 in FIG.

  For this reason, first, the proximity user specifying unit 45 specifies a proximity user as a candidate for data sharing. The specified proximity user is typically a user who frequently contacts in real life or a user whose life style is similar, and can be specified by the following embodiments.

  In one embodiment, the recognizing devices 10 (terminals) detect each other by well-known wireless communication (especially short-range wireless communication), and terminals that have been physically close to each other by passing each other. By detecting, it is possible to identify the detected terminal as that of a nearby user. At this time, only the terminals that have reached the predetermined frequency may be specified.

  In one embodiment, the proximity user may be specified by performing face recognition. For this reason, the user continuously photographs in daily life or the like from the camera attached to the HMD, and performs face recognition on the acquired image. By storing the recognized face feature amount in a temporary area, a person who frequently meets the face within a certain period is detected. By querying the face image DB server (not shown) based on the feature amount of the person who frequently meets the face and specifying the person, the person can be specified as a proximity user. Furthermore, an inquiry can be made to a person / terminal DB server (not shown) to identify the terminal of the adjacent user.

  The above is an embodiment in which the proximity user can be automatically specified, but the user may manually specify the proximity user and inform the proximity user specifying unit 45 of the proximity user and its terminal. . For example, you may make it select a near user from the person connected on SNS (social network service).

  When the proximity user specifying unit 45 specifies the proximity user according to each of the above embodiments, data sharing is performed with the terminal of the proximity user. The data sharing process will be described with reference to FIG. Here, it is assumed that the recognition device 10B illustrated in FIG. 6 is a terminal of a user specified as a proximity user.

  That is, in the data sharing process, data (features, content, 3D model) corresponding to each article name stored in the storage units 41 and 41B between the recognition devices 10 and 10B via the transmission and reception units 46 and 46B. Data). As a result, data that does not exist in the storage unit 41 but exists in the storage unit 41B is added to the storage unit 41, and vice versa. The trigger for starting the data sharing process is reception of a sharing request from one or both users.

  It should be noted that at least only the article name is shared, and each recognition device 10 and 10B executes step S1 in FIG. 2 as article information obtained by newly obtaining the shared article name. It may be ordered from the data server 15. In particular, when content is set for each user, it is preferable to obtain the content from the data server 15. Moreover, it is preferable to manage history-related information monitored by the period monitoring unit 42 and the frequency monitoring unit 43 for each user without sharing.

  Here, before actually performing the data sharing process as described above, after receiving the sharing request, the authentication units 47 and 47B first authenticate each other and agree to allow data sharing with each other and succeed in authentication. In such a case, data sharing processing may be performed.

  Or you may authenticate by recognizing the specific goods memorize | stored in at least any one of the memory | storage parts 41 and 41B simultaneously (in a short period) between the users who share. In this case, the user actually prepares the product, the two users photograph the product with the respective photographing units 21, analyze the image with the recognition unit 50, obtain the product name, and the product name is at least one of the product names If the data are stored in the storage units 41 and 41B and match each other, the authentication is successful. At this time, since the recognition process by the recognition unit 50 is impossible in the terminal of the user who does not possess the feature quantity necessary for the recognition of the product, the second embodiment in the above-described article information acquisition unit 25 is used instead. Similarly, the product name is obtained by photographing the barcode of the product and analyzing the image.

  That is, more generally, when both users acquire images during a certain period and it is determined that the common products are included in both images, it is determined that the authentication is successful. May be. The common product may be an arbitrary product, or may be a product on which the above restrictions are imposed. That is, the product may be a product with a restriction that the product name is stored in at least one of the storage units 41 and 41B and can be recognized by at least one of the recognition units 50.

  Further, when actually sharing data, it is possible not to share all data as much as possible as described above, but to limit the range to be shared. In this case, it is manually set whether or not to make it a sharing target for the article names stored in the mutual storage units 41 and 41B in the mutual disclosure setting units 49 and 49B, respectively. Then, the sharing process as described above is performed on the data set as the sharing target.

  Alternatively, in the case where authentication is performed by simultaneously recognizing the barcode of the specific product, the disclosure range may be set according to the product used for authentication. For this reason, category classification may be provided for each product in advance, and only the category to which the product used for authentication belongs may be disclosed. Categories can be used as default product categories such as beverages, stationery, etc., registered in the reference DB 33 as additional information linked to each product name, or users can be individually selected according to their interest level, importance level, etc. You may use the category set in.

  Alternatively, a category may be provided for each article name as described above, and the article name data corresponding to the category existing in common in the storage units 41 and 41B of both users may be shared.

  As described above, according to the present invention, for example, by using a receipt or a barcode attached to a product, a product or the like that is around the user is identified in advance, and the feature amount of the item around you is obtained by inquiring the data server. Is stored in the terminal, and an article around us is recognized with the stored feature amount. As a result, it is possible to recognize objects around us while reducing the size of the reference image DB on the memory in the terminal while recognizing the object in the terminal without having to inquire the server every time of recognition. Become. In addition, by sharing feature quantities between users based on a social network or the like, the range in which object recognition can be performed can be expanded.

  Note that the present invention can also be provided as a program (which may be stored in a storage medium) that causes a computer to function as the recognition device 10. The hardware configuration of the computer may be a known configuration including a CPU (Central Processing Unit) that executes instructions, a memory, and various input / output interfaces. The present invention can also be provided as an operation method of the recognition apparatus 10 or the recognition system 100.

  Hereinafter, the operation of the English word learning system will be described as an example using the recognition system 100 of the present invention. In the description, related functional blocks are indicated by parentheses as appropriate.

  A feature DB server (data server 15) is prepared in which the product name, the feature used for product object recognition, and the English word content related to the product are linked and stored. The feature DB server receives a product name as a request from the learner's terminal (recognition device 10), and returns a feature value associated with the product name and English word learning content as a response.

  A learner who learns English words using this system wears an HMD as a learner terminal. When a learner looks at a barcode or receipt of a product while living with an HMD, the learner reads or reads the contents of the barcode or receipt with a camera attached to the HMD, or purchases it. The product name of the product is specified (article information acquisition unit 25). The specified product name is sent to the feature DB server, English word learning data related to the feature necessary for object recognition of the product is acquired, and stored in the memory in the HMD terminal (management unit 40).

  Here, it is assumed that the English language learning content includes a plurality of problems in which the meaning of English words is answered with four choices.

  Thereafter, the image acquired from the HMD camera (imaging unit 21) recognizes whether or not the previously acquired product is included based on the feature amount stored in the memory (recognizing unit 50). When the product is recognized, the English word learning content linked to the product is read from the memory, and one question is randomly displayed on the HMD display from the English word learning content (presentation unit 60) . The learner answers the displayed English word problem (reception unit 65). Here, when all the English word problems associated with the product are answered and correctly answered, the feature amount data and the content data of the product are deleted from the memory (management unit 40). Further, after acquiring the object recognition feature value of the product, the feature value data that has passed the storage period set for each product is deleted from the memory (management unit 40).

  The learner's HMD uses wireless communication to perform passing communication with a nearby learner (proximity learner) HMD and accumulates the number of passing in the memory (proximity user identification unit 45). When there is a proximity learner whose pass count exceeds a certain number, a feature amount acquisition request is transmitted to the proximity learner (transmission / reception unit 46).

  When a proximity learner receives a feature amount acquisition request, a notification is made on the proximity learner's HMD. The proximity learner who has received the notification selects whether or not to provide the feature data (disclosure setting unit 49B). When permitting the provision of data, the proximity learner (1) provides all the feature data, (2) provides the feature data of the category held in common, (3) the recognized object Select a method of providing three patterns of providing categories.

When (1) is selected, all feature amount data possessed by the proximity learner is provided.
When (2) is selected, the feature amount data of the product category that is commonly held between the learner and the proximity learner is provided.
When (3) is selected, the proximity learner takes out the product holding the feature amount data, makes the learner recognize the barcode, and at the same time recognizes the product itself. When the barcode recognition and the product recognition are performed at the same time, the feature amount data of the category to which the product belongs is shared.

  DESCRIPTION OF SYMBOLS 100 ... Recognition system, 10 ... Recognition apparatus, 15 ... Data server, 25 ... Article information acquisition part, 40 ... Management part, 50 ... Recognition part, 60 ... Presentation part, 45 ... Proximity user identification part

Claims (13)

A shooting section for shooting images;
An article information acquisition unit for acquiring article information as the name of an article existing around the user;
A management unit for managing a feature amount corresponding to the acquired article information;
A recognition unit that recognizes an article included in an image captured by the imaging unit based on the managed feature amount,
By transmitting the acquired article information and receiving the corresponding feature quantity to the data server that holds the name and the feature quantity associated with each article, the management unit receives the acquired article information. The recognition apparatus characterized by acquiring and managing the feature-value corresponding to. The recognition apparatus according to claim 1, wherein the article information acquisition unit acquires article information as a product name by at least one of the following methods (1) to (3).
(1) A method for extracting a product name by analyzing an image of a receipt photographed by the photographing unit. (2) By analyzing a code in a product image provided with a code photographed by the photographing unit, Method of obtaining the corresponding product name (3) Method of extracting the purchased product name from the user's electronic commerce history The data server holds the name of each article and the feature amount extracted from the image of the article viewed from each angle,
The recognition apparatus according to claim 1, wherein the management unit manages feature quantities at each angle so that the recognition unit recognizes an article at each angle. The recognition apparatus according to claim 1, wherein the management unit deletes a feature amount corresponding to any of the following (1) to (3).
(1) A feature quantity whose elapsed period after being acquired from the data server has reached a certain period (2) A feature quantity whose continuation period that the corresponding article is not recognized by the recognition unit has reached a certain period (3 ) The feature amount at which the frequency at which the recognition unit recognizes the corresponding article is less than a certain value A proximity user specifying unit for specifying a proximity user for the user of the recognition device;
The management unit includes a feature amount managed by the management unit and a feature amount corresponding to article information acquired by the proximity user, which is managed by a second management unit included in the second recognition device of the proximity user. The recognition apparatus according to claim 1, wherein the recognition apparatus is shared and managed.   The recognition apparatus according to claim 5, wherein the management unit manages the shared information when authentication with the proximity user is successful.   In the authentication, when both users acquire images during a certain period and it is determined that a common product is included in both images, it is determined that the authentication is successful. The recognition device according to claim 6. Categories are assigned to goods,
The management unit includes: a feature amount managed by the management unit; and a feature amount corresponding to article information acquired by the proximity user, which is managed by a second management unit included in the second recognition device of the proximity user. 8. The recognition apparatus according to claim 5, wherein feature quantities corresponding to article information in articles having a common category are shared and managed.   The recognition apparatus according to claim 5, wherein the management unit shares and manages article information for which sharing permission is given by a user. In the data server, the name, feature amount, and content of each article are linked and held,
By transmitting the acquired item information and receiving the corresponding feature amount and content, the management unit acquires and manages the feature amount and content corresponding to the acquired item information,
The recognition apparatus according to claim 1, further comprising a presentation unit that presents the managed content corresponding to the recognized article.   The recognition apparatus according to claim 10, wherein when the use of the presented content by a user is completed, the management unit deletes a feature amount corresponding to the content.   A recognition program for causing a computer to function as the recognition device according to claim 1. In the data server, the step of associating and holding the name and feature amount of each article;
In a recognition device equipped with a photographing unit for photographing an image,
An article information acquisition step of acquiring article information as the name of an article existing around the user;
A management step of managing a feature amount corresponding to the acquired article information;
A recognition step of recognizing an article included in an image captured by the imaging unit based on the managed feature amount,
By transmitting the acquired article information to the data server and receiving the corresponding feature quantity, the management step acquires and manages the feature quantity corresponding to the acquired article information. How to recognize.