JP2019070887A - Object recognition device for recognizing place and object by two steps - Google Patents

Abstract

To solve problems in a case of recognizing objects appearing in an image: that when the number of objects to be recognized increases, the number of very similar objects may increase to degrade identification accuracy; and that a name to be used for the object in the image cannot be determined when different names are used in different places, for objects similar to the object in the image.SOLUTION: A device for recognizing objects appearing in an image improves identification accuracy, by adding information about a place where the image is captured, to reduce erroneous recognition between the objects and very similar objects in other places, and to enable adding the information peculiar to that place to recognized results.SELECTED DRAWING: Figure 1

Description

  The present invention is an apparatus for recognizing an object captured in an image, which adds information on a photographed place to improve recognition accuracy and reduce false recognition with similar objects in other places, as well as the place It is an object recognition device that makes it possible to add unique information to the recognition result.

  It became possible to perform machine learning of a large number of digital images with a computer and use the learning results to display what an image taken with a smartphone etc. is by a device that recognizes an object shown in the image. .

  However, as the number of objects to be recognized increases, similar objects increase, and the identification accuracy decreases. Further, even if similar images are used, it is not possible to determine which designation should be used if different designations are used depending on places.

  On the other hand, the number of users using social networks that publish photos is increasing, and there is an increasing opportunity to request information on both places and objects, such as, for example, which restaurant the food is served.

JP, 2013-114596, A

  The machine learning of a large number of digital images by a computer and using the learning results, it is different depending on the place when displaying what is the image taken with a smart phone etc. Even when a name is used, displaying it using an appropriate name can provide more detailed information on the object.

  However, it is difficult to know where the similar object was shot from the image information alone.

  By using the technology of Patent Document 1, it is possible to recognize a plurality of objects from a photographed image, and if there is an object relating to a place, it is possible to specify the photographed place, Information on location can be reflected in the recognition process.

  However, an object relating to a place is not necessarily photographed. Moreover, in the process of recognition, it is necessary to recognize from all target objects, and the efficiency is low.

  An object of the present invention is to make it possible to display an image taken with a smartphone or the like by using an appropriate name even when a name different from one place to another is used. .

To this end, the invention classifies the objects of the objects to be recognized into primary objects and secondary objects in order to limit the places taken, the primary objects being used to limit the places, It is characterized in that the information on the place is used to increase the recognition rate of the secondary object, and further, it has means for displaying a name unique to the place.

Further, according to the present invention, when the secondary object is recognized, a candidate related to a place recognized by the primary object is selected with respect to a plurality of candidates having a high degree of coincidence, whereby the object shown in the image is displayed. It has means for increasing the recognition rate.

Furthermore, the present invention has a means for increasing the recognition rate of an object shown in an image by recognizing a secondary object using the result of machine learning of an object within the range of the location recognized by the primary object. .

Alternatively, the present invention is characterized in that the location is specified by reading a QR code (a QR code is an image of a specific number, which is a registered trademark of Denso Wave Inc.) installed at a specific location. .

Alternatively, the present invention is characterized by specifying a place by receiving a beacon ID transmitted by a beacon signal transmitter installed at a specific place.

Alternatively, the present invention is characterized in that a GPS satellite signal is received, and the location is recognized by the information of its latitude and longitude.

Alternatively, according to the present invention, when receiving a GPS satellite signal and using information on its latitude and longitude to recognize a place, the learning result of machine learning associated with a plurality of positions near the latitude and longitude is obtained. One or more recognition results are displayed after weighting the recognition results of the object in accordance with the distance to the position obtained by receiving the signal of the previous GPS satellite.

  The present invention classifies the objects of the object to be recognized into primary objects and secondary objects in order to limit the locations taken as described above, the primary objects being used to define the location, which locations Because the recognition rate of secondary objects is increased using information from the above, when displaying what is the image taken with a smart phone etc., even when different names are used depending on the location, use appropriate names Can be displayed.

  In addition, when the secondary object is recognized, the recognition rate of the object shown in the image is increased by selecting candidates related to the place recognized by the primary object with respect to a plurality of candidates with high matching degrees. You can also

  Also, instead of capturing the primary object and specifying the location, read the QR code (QR code is an image of a specific number, a registered trademark of Denso Wave Inc.) installed at a specific location. To receive a beacon ID transmitted from a beacon signal transmitter installed at a specific location, to receive GPS satellite signals, and to recognize the location by using information such as latitude and longitude it can.

  Alternatively, instead of capturing the primary object and specifying the location, a GPS satellite signal is received, and when the location is recognized using the latitude and longitude information, a plurality of positions near the latitude and longitude With respect to the learning result of machine learning linked to, it is possible to display the recognition results of objects related to a plurality of places in the vicinity of the position obtained by receiving the signal of the previous GPS satellite.

  Alternatively, instead of capturing the primary object and specifying the location, a GPS satellite signal is received, and when the location is recognized using the latitude and longitude information, a plurality of positions near the latitude and longitude After weighting the recognition result of the object according to the distance to the position obtained by receiving the signal of the previous GPS satellite with respect to the learning result of machine learning linked to the one or more recognition results Can be displayed.

When displaying what is an image taken with a smartphone or the like by using the system of the present invention, it is possible to display using an appropriate name even when a different name is used depending on the place. Has the advantage of being able to

Also, even if different names are used depending on the location, an image can be associated with a specific product of a specific store, because an appropriate name can be known, whereby the image has information on a specific product of a specific store. There is an advantage that it can be linked.

FIG. 1 is a schematic view of primary object imaging according to an embodiment of the present invention. FIG. 2 is a schematic diagram of primary object recognition according to an embodiment of the present invention. FIG. 3 is a schematic view of secondary object imaging according to an embodiment of the present invention. FIG. 4 is a schematic view of secondary object recognition according to an embodiment of the present invention FIG. 5 is a diagram showing the flow of the operation of the embodiment of the present invention. FIG. 6 is a diagram showing the configuration of the server and the flow of data according to the embodiment of this invention.

  When displaying what an image taken with a smart phone or the like is, the object of the object to be recognized is classified into a primary object and a secondary object, and the primary object is used to limit the location. The information was used to increase the recognition rate of secondary objects, and even when different names were used depending on the location, it was possible to display using appropriate names.

  Although an example is described supposing the user who uses a store, the same effect is acquired also in facilities other than a store. Moreover, it is not necessary for the product to be a specific object, but may be an act such as a service provided at a place, or a sentence such as an explanation of a place.

  FIG. 1 is a schematic view of primary object imaging according to an embodiment of the present invention, wherein 101 is a user, 102 is a store, and 103 is an information terminal with a camera.

  The user 101 recognizes the store 102 registered in advance as a primary object by photographing the store 102 with the camera-equipped information terminal 103.

  FIG. 2 is a schematic view of primary object recognition according to an embodiment of the present invention, wherein 201 is a camera-equipped information terminal, 202 is a store reflected in the camera-equipped information terminal, 203 is a primary object in the image of the store 202 From the first candidate 204 and the second candidate 205, the first recognition candidate for the primary object, the second recognition candidate for the primary object, and the second recognition candidate for the first object. It is a finger for the user to select.

  When there is image information for recognizing a primary object in the image of the store 202 shown on the camera-equipped information terminal 201, a mark 203 indicating the image information for recognizing the primary object is displayed At the same time, the recognized primary object candidates are displayed. If a plurality of candidates can not be narrowed down, a first candidate 204 and a second candidate 205 are displayed on the lower side of the screen, and the user determines which of them is determined.

  When the mark 203 indicating the image information is not displayed, it indicates that there is no primary object for recognizing the store in the image, so the user will shoot another part, but in general Since it is known in advance that what is the primary object for recognizing the store, the user will not be bothered.

  Also in the case of using a geometric figure such as a QR code as image information for recognizing a primary object, the same effect can be obtained by photographing similarly.

  In the case of recognizing a store by receiving a beacon signal, it is not necessary to take a picture with the camera-equipped information terminal 201, and according to the code of the beacon signal received by the camera-equipped information terminal 201, An image indicating a store such as an image and a candidate for a primary object are displayed and determined automatically or determined by the user or determined. A plurality of signals are received, and if there are a plurality of candidates, a plurality of candidates are displayed and determined by the user.

  The same applies to the case of recognizing a store by receiving a GPS signal, and it is not necessary to take a picture with the camera-equipped information terminal 201, and registered in advance according to the position calculated from the GPS signal received by the camera-equipped information terminal 201. An image showing the store, such as an image of the store in question, and a candidate for the primary object are displayed, and are automatically determined or determined by the user or determined. If there are multiple candidates near the position calculated from the GPS signal, those candidates are displayed and determined by the user.

  FIG. 3 is a schematic diagram of secondary object imaging according to an embodiment of the present invention, wherein 301 is a user, 310 is a table, 311 is a commodity which is a secondary object, 320 is a camera information terminal, 330 is a server, 340 Is image data including the product 311 captured by the camera-equipped information terminal 320, 350 is a computer of the store, and 351 is information on the product in the store.

  The user 301 photographs the product 311 with the camera-equipped information terminal 320, and transmits the image data 340 to the server 330 through the wireless circuit provided in the camera-equipped information terminal 320. The server 330 recognizes what the product 311, which is a secondary object, is by using the parameter for object recognition according to the photographed location, and the product 350 from the information 351 on the product in the store from the computer 350 of the store in advance Information related to 311 is linked to image data 340 and stored.

  FIG. 4 is a schematic view of secondary object recognition according to an embodiment of the present invention, in which 401 is a camera-equipped information terminal, 402 is a product to be recognized that is displayed on the camera-equipped information terminal, and 403 is displayed on the camera-equipped information terminal Another item which is not the object, 404 is a mark indicating image information for recognizing a secondary object in the image, 410 is a first recognition candidate of the secondary object, 411 is a second recognition candidate of the secondary object, Reference numeral 420 denotes a finger for the user to select from the first candidate 410 and the second candidate 411.

  If the image shown on the camera-equipped information terminal 401 includes image information for recognizing a secondary object, a mark 404 indicating the image information for recognizing the secondary object is displayed and recognized. The candidates for the next object are displayed. If the candidates can not be narrowed down and there are a plurality of candidates, the first candidate 410 and the second candidate 411 are displayed on the lower side of the screen, and the user determines and decides which one. Moreover, the mark 404 which shows the image information for recognizing a secondary object is not displayed on another goods 403 which are not recognition object.

  FIG. 5 is a diagram showing the flow of the operation of the embodiment of the present invention, wherein 501, 504, 505, 506, 507, 510, 511, 513 are server-side operations, 502, 503, 508, 509, 512. Is the operation of the user terminal.

  At the start of operation, at operation 501, the server sets parameters for recognizing an object of an image to those for a store. At operation 502, the user captures an object that is a symbol of a predetermined store on the user terminal. In operation 503, an image of an object to be a symbol of the store photographed from the user terminal is transmitted to the server. At act 504, the server receives an image of an object that is a symbol of the store. At act 505, the server object recognizes the image received at act 504 and identifies the store from the object that is a symbol of the store. At act 506, the server sets the object recognition parameters to those targeted for the item. At operation 507, the server notifies the user terminal that the parameter has been set for the product. At act 508, the user shoots the item in the store. At act 509, the user sends the product image to the server. At act 510, the server receives an image of the item. In operation 511, the server performs object recognition on the image received in operation 5010 and identifies a product. If there are multiple candidates, the candidates are notified. At operation 512, the user confirms and notifies that the recognized product matches the photographed object. Here, if there is no product among the candidates, it may be requested to notify another candidate. In operation 513, the server associates and stores the image, information on the image registered in advance, and information on the photographer. Operations 508 to 513 may be performed repeatedly. Upon notifying that the user has left the store at operation 514, the server returns the parameters for recognizing the object to those intended for the store at operation 501.

  The above is the flow of operations when object recognition is performed by the server, but when object recognition is performed by the user terminal, operations 501, 505, 506, and 511 are performed by the user terminal. At this time, operations 503, 504, 507, 509, 510, and 512 are unnecessary.

  In addition, when specifying a store with a beacon signal or a GPS signal, the server receives the position calculated from the beacon signal code or the received GPS signal from the user terminal instead of the operations 501, 502, 503, 504, and 505. And identify the store based on the value.

  FIG. 6 is a diagram showing the configuration of the server and the flow of data according to an embodiment of the present invention, wherein 601 is an image receiving device, 602 is an object recognition device, 603 is an object determination device, 604 is a weight coefficient replacement device, 605 Is a weight coefficient storage device for object recognition, 607 is a product information linking device, 608 is a product information storage device, 609 is an image information storage device, 611 is an image input unit, 612 is an image recognition unit, 613 is a result output unit, 621 Is a candidate display device, 622 is a result determination device, 623 is a branch as to whether the store has been determined, 631 is a weight coefficient storage unit for a product recognition database for store A, 632 is a weight for a product recognition database for store B Coefficient storage unit, 633 is a weighting coefficient storage unit for a product recognition database for store C, 634 is a weighting coefficient storage unit for a product recognition database for store Z, 651 is Product information selection unit, 661 is a product information storage unit for store A, 662 is a product information storage unit for store B, 663 is a product information storage unit for store C, 664 is a product information storage unit for store Z, 671 Is an image information AA storage unit, and 672 is an image information ZZ storage unit.

  The image reception device 601 receives a photographed image from the user terminal and transmits it to the object recognition device 602. The object recognition device 602 transmits the recognition result to the object determination device 603. In the state where the store is not determined, the weight coefficient for object recognition of the object recognition device 602 is a weight coefficient for the store recognition database, and the image received from the image reception device 601 is used to recognize the store. The object determination device 603 identifies a store from the recognition result of the object recognition device 602, and notifies the weighting factor replacement device 604 of the identified store. The weighting factor replacing device 604 receives the weighting factor for the product recognition database corresponding to the specified store from the weighting factor storage device 605 for object recognition, and the weighting factor for the object recognition device image recognition device 602 for the store recognition database The coefficients are replaced with weighting coefficients for the product recognition database. In this state, the image reception device 601 receives the next captured image from the user terminal and transmits it to the object recognition device 602. The object recognition device 602 transmits the recognition result to the object determination device 603, but since the store is determined, the result is transmitted to the commodity information linking device 607. The product information linking device 607 receives information on the specified product from the product information storage device 608, and further stores the information in the image information storage device 609 in association with the image received from the image reception device 601. At this time, information related to the user can be similarly associated.

  After receiving an image by the image input unit 611, the object recognition device 602 detects and recognizes the presence or absence of an object already registered in the image and what the object is from the image. At this time, a large number of image data are mechanically learned in advance using a technique such as a neural network, and a weighting factor such as a neural network learned for each set of image data is used. The recognized objects are output from the result output unit 613 with the order of the degree of coincidence.

  The candidate display device 621 of the object determination device 603 receives the recognition result of the object given the order of the coincidence from the result output unit 613 of the object recognition device 602, and instructs the user terminal to display the object as the object candidate in descending order of coincidence. put out. The result determination device 622 receives from the user terminal what object the user has selected from the object candidates, and determines the result as an object. In the state where the store is not determined, the store can be determined because this object is an object that symbolizes the store. In this case, the identified store is transmitted to the weight coefficient replacing device 604. In the state where the store is determined, the specified object is a product, and the specified product is transmitted to the product information linking device 607.

  The object recognition weight coefficient storage device 605 comprises a weight coefficient storage unit for a store recognition database used to recognize a store and a weight coefficient storage unit for each store oriented product recognition database for recognizing a product for each store. . For example, the weighting coefficient storage unit for item recognition database for store A is a parameter for recognizing what the object shown in the photographed image is on the premise that it is in the store A, for example, Even if there is a similar product, it can be determined that the product is at the store A. A weighting factor is stored for recognizing, for each store, the products of all stores registered in the same manner.

  The product information linking device 607 links the product confirmed by the object fixing device 603 with the information of the product stored in advance in the product information storage device 608. Also, at this time, user information can be linked. The image information storage device 609 stores the image, the store name, the product name, the product information, and the user information in a linked state.

  When a store is specified using a QR code, after a store is determined by a general QR code recognition device, a weighting factor for a product recognition database may be used according to the store.

  Also, in the case where a store is specified by a beacon signal or a GPS signal, the store may be specified based on the value obtained by receiving the code of the beacon signal from the user terminal or the position calculated from the received GPS signal.

  The object for determining the store is preferably as unique as possible to the store, and if it is a signboard with the store name described, a characteristic exterior or interior, the degree of coincidence is high and it is easy to limit the candidates. However, in the case of a similar signboard or exterior like a chain store, the degree of coincidence can be enhanced by combining beacon signals and GPS signals.

  Since the present embodiment is configured as described above, for example, even if the product of the store A and the product of the store B are very similar in appearance, it can be recognized that the stores are different. Can be associated and recorded.

  Here, the information that varies depending on the store includes, for example, the price of the product, the name of the product, the name of the producer, the name of the seller, the weather at the time of purchase, the name of the service provider, the origin of the product, the advertisement of the product, the store Advertising, etc.

  Although the present embodiment has been described using a store, the same operation is possible at other facilities. Shrines and temples can associate information such as the origin of a place with photographs of similar objects like trees and stones taken. In the case of an amusement park, it is possible to recognize an amusement park by primary object recognition, recognize an attraction by secondary object recognition, and recognize landscapes and props by tertiary object recognition.

  As described in the present embodiment, the information finally stored in the image information storage device 609 includes the image, the store name, the product name, the product information, and the user information. If it is searched, it is possible to refer to other information as related information, and further new information can be obtained by newly searching from the related information.

  In the device that recognizes an object captured in an image, information on the location where the image was taken is added to improve the recognition accuracy, reduce misrecognition with similar objects in other locations, and reduce information specific to that location. It was possible to add to the recognition result. Thus, when the photographed image is browsed, it is possible to know information related to the image that the photographed person did not know. Also, by publishing it on a social network etc., a third party can view the image and similarly get new information. By promoting the photographing using the curiosity of the person thus photographed, it is possible to expect an effect that many people come to the store. Further, the information associated with the image, the store name, the product name, the product information, and the user information can be used to improve the sales plan of the store, and the information itself has great value.

  The present invention is an apparatus for recognizing an object captured in an image, which adds information on a photographed place to improve recognition accuracy and reduce false recognition with similar objects in other places, as well as the place It is possible to add unique information to the recognition result.

101 users 102 stores 103 information terminals with cameras

201 camera-equipped information terminal 202 store 203 shown on camera-equipped information terminal mark 204 indicating image information 204 first recognition candidate for primary object 205 first recognition candidate for primary object 210 finger for user selection

301 user 310 table 311 secondary object product 320 information terminal with camera 330 server 340 image data including product 311 computer of store 351 information about product in store

401 A camera-equipped information terminal 402 A product 403 to be recognized Another product 404 not to be recognized A mark 410 indicating image information 410 A first recognition candidate for a secondary object 411 A second recognition candidate for a secondary object 420 For the user to select finger

501 server side operation 502 user terminal operation 503 user terminal operation 504 server side operation 505 server side operation 506 server side operation 507 server side operation 508 user terminal operation 509 user terminal operation 510 server side operation 511 Server side operation 512 User terminal operation 513 Server side operation

601 image reception device 602 object recognition device 603 object determination device 604 weight coefficient replacing device 605 weight coefficient storage device for object recognition 607 product information linking device 608 product information storage device 609 image information storage device 611 image input unit 612 image recognition unit 613 Result output unit 621 Candidate display device 622 Result confirmation device 623 Branch whether or not the store has been decided 631 Weight coefficient storage unit for product recognition database for store A 632 Weight coefficient storage unit for product recognition database for store B Weight coefficient storage unit 634 for product recognition database for C Weight coefficient storage unit 651 for product recognition database for store Z Product information selection unit 661 Product information storage unit 662 for store A Product information storage unit 663 for store B Product information storage unit 664 for store C Product information for store Z Information storage unit 671 Image information AA storage unit 672 Image information ZZ storage unit

Claims (6)

  Means for recognizing an object in an image using parameters learned in advance, and a first set of parameters for recognizing a first object for identifying a place where an image was taken from a location registered in advance are recorded Means, means for recording a second set of parameters for recognizing a second object in the range of each registered location, and recognition of an object with said first set of parameters at least once or more After the location is identified by the result, a parameter set for recognizing a second object belonging to the identified location is selected from the second parameter set, and the first parameter set is selected. Object recognition device comprising means to replace.   A means for recognizing an object in an image using parameters learned in advance, a means for recording a code of a beacon signal received at the time of image taking, and a place where an image was taken from a place registered in advance by the recorded code. After the location is identified by the means for identifying the object, the means for recording the parameter set for recognizing the object within the range of the registered place, and the means for identifying the place by the beacon signal, An object recognition apparatus comprising means for selecting a parameter set for recognizing an object belonging to a specified place from among a parameter set.   A means for recognizing an object in an image using parameters learned in advance, a means for recording a code of a positioning signal received at the time of image taking, and a place where an image was taken from a place registered in advance by the recorded code. After the location is identified by the means for identifying the object, the means for recording the parameter set for recognizing the object within the range of the registered place, and the means for identifying the place by the beacon signal, An object recognition apparatus comprising means for selecting a parameter set for recognizing an object belonging to a specified place from among a parameter set.   A means for recognizing an object in an image using parameters learned in advance, a means for recording a code of a positioning signal received at the time of image taking, and a place where an image was taken from a place registered in advance by the recorded code. A means for identifying the first set of parameters for identifying the first object for identifying the place where the image was taken from the place registered in advance; A means for recording a second set of parameters for recognizing a second object in range, and an object at said first set of parameters associated with a pre-registered location in a range identified from said positioning signal To identify the second object belonging to the specified place from the second parameter set after the recognition of the Select La meter set consists of means for replacing said first parameter set, the object recognition device.   A method according to claim 1 or claim 2 or claim 3 or claim 4 having means for storing information associated with an object recognized in a recorded image and different information for each photographed place in the image. Object recognition device. 6. The object recognition apparatus according to claim 5, further comprising means for adding information that varies with time to information that is associated with the recognized object in the recorded image and that is different for each photographed location.

Images