JP2023068487A - Information processing method, program, and information processor - Google Patents

Abstract

To provide an information processing method and the like capable of improving identification accuracy of an object.SOLUTION: An information processing method includes: acquiring image information on an image of an object captured by an information processing terminal; acquiring sensor information detected by a sensor of the information processing terminal when capturing the object; and referring to a storage section that stores the image information and the sensor information corresponding to a plurality of object candidates to execute processing of identifying the captured object from the plurality of object candidates on the basis of the acquired image information and sensor information.SELECTED DRAWING: Figure 1

Description

The present invention relates to an information processing method, a program, and an information processing apparatus.

In recent years, there has been an increasing need to identify objects based on images. In Patent Document 1, a classifier corresponding to an imaging condition is selected from a plurality of classifiers generated for each of a plurality of different imaging conditions, and an object in image data is identified using the selected classifier. An identification method for identifying is disclosed.

JP 2018-081404 A

However, the invention according to Japanese Patent Laid-Open No. 2003-100002 can identify an object when the photographed objects have the same shape, when there is no change in the background of the object, or when a plurality of objects with the same shape exist. There is a problem that cannot be (specified).

An object of one aspect is to provide an information processing method and the like capable of improving the accuracy of identifying a target object.

An information processing method according to one aspect acquires image information related to an image of an object photographed by an information processing terminal, acquires sensor information detected by a sensor of the information processing terminal when the object is photographed, obtains a plurality of referring to a storage unit storing image information and sensor information corresponding to the target object candidates, and based on the acquired image information and sensor information, identifying the photographed object from among the plurality of object candidates; It is characterized by executing

In one aspect, it is possible to improve the accuracy of object identification.

1 is an explanatory diagram showing an outline of an object identification system; FIG. It is a block diagram which shows the structural example of a server. FIG. 4 is an explanatory diagram showing an example of a record layout of an object DB; 2 is a block diagram showing a configuration example of a terminal; FIG. FIG. 4 is an explanatory diagram showing a processing operation for identifying an object photographed by a terminal; FIG. 10 is an explanatory diagram illustrating processing for calculating similarity based on weighting; FIG. 4 is an explanatory diagram showing an example of a target object identification screen; 4 is a flow chart showing a processing procedure when specifying a photographed object; FIG. 10 is a flowchart showing the procedure of a subroutine of processing for calculating similarity; FIG. FIG. 11 is a block diagram showing a configuration example of a server according to Embodiment 2; FIG. 4 is an explanatory diagram showing an example of a record layout of a training data DB; It is an explanatory view explaining a similarity output model. FIG. 10 is a flowchart showing a procedure of a subroutine for similarity calculation processing using a similarity output model; FIG.

Hereinafter, the present invention will be described in detail based on the drawings showing its embodiments.

(Embodiment 1)
Embodiment 1 relates to a mode for identifying a photographed object from among a plurality of object candidates based on image information of the object photographed by an information processing terminal and sensor information detected by a sensor of the information processing terminal. . The object is, for example, a switchboard or power receiving and transforming equipment such as a distribution board, an environment measuring device, any device such as a personal computer, a vehicle, an object, an animal, a person, or the like.

FIG. 1 is an explanatory diagram showing an outline of an object identification system. The system of this embodiment includes an information processing device 1 and an information processing terminal 2, and each device transmits and receives information via a network N such as the Internet.

The information processing device 1 is an information processing device that processes, stores, and transmits/receives various information. The information processing device 1 is, for example, a server device, a personal computer, a general-purpose tablet PC (personal computer), or the like. In the present embodiment, the information processing device 1 is assumed to be a server device, which is replaced with the server 1 for the sake of brevity.

The information processing terminal 2 is a terminal device that acquires image information about an image of a target object, detects sensor information using a sensor of the information processing terminal 2, and transmits image information and sensor information. The information processing terminal 2 is, for example, an information processing device such as a smart phone, a mobile phone, a wearable device such as Apple Watch (registered trademark), a tablet, wearable glasses, or a personal computer terminal. For the sake of brevity, the information processing terminal 2 is replaced with the terminal 2 below.

A server 1 according to the present embodiment acquires image information regarding an image of an object captured by a terminal 2 having a plurality of sensors. The server 1 acquires a plurality of sensor information detected by each sensor of the terminal 2 when the object is photographed. The server 1 refers to a storage unit that stores image information and sensor information corresponding to a plurality of object candidates, and based on the acquired image information and the plurality of sensor information, the object and the relevant object for each object candidate. Calculate the degree of similarity with the candidate. The server 1 identifies the object candidate with the highest degree of similarity as the photographed object. Image information and sensor information will be described later.

FIG. 2 is a block diagram showing a configuration example of the server 1. As shown in FIG. The server 1 includes a control section 11 , a storage section 12 , a communication section 13 , a reading section 14 and a mass storage section 15 . Each configuration is connected by a bus B.

The control unit 11 includes arithmetic processing units such as a CPU (Central Processing Unit), an MPU (Micro-Processing Unit), and a GPU (Graphics Processing Unit), and reads out the control program 1P (program product) stored in the storage unit 12. Various information processing, control processing, etc. related to the server 1 are performed by execution. It should be noted that the control program 1P is deployed to run on a single computer or on multiple computers located at one site or distributed across multiple sites and interconnected by a communication network. can do. Note that although FIG. 2 illustrates the controller 11 as a single processor, it may be a multiprocessor.

The storage unit 12 includes memory elements such as RAM (Random Access Memory) and ROM (Read Only Memory), and stores the control program 1P or data necessary for the control unit 11 to execute processing. The storage unit 12 also temporarily stores data and the like necessary for the control unit 11 to perform arithmetic processing. The communication unit 13 is a communication module for performing processing related to communication, and transmits/receives information to/from the terminal 2 or the like via the network N.

The reader 14 reads a portable storage medium 1a including CD (Compact Disc)-ROM or DVD (Digital Versatile Disc)-ROM. The control unit 11 may read the control program 1P from the portable storage medium 1a via the reading unit 14 and store it in the large-capacity storage unit 15 . Alternatively, the control unit 11 may download the control program 1P from another computer via the network N or the like and store it in the large-capacity storage unit 15 . Furthermore, the control unit 11 may read the control program 1P from the semiconductor memory 1b.

The large-capacity storage unit 15 includes a recording medium such as an HDD (Hard disk drive) or an SSD (Solid State Drive). The mass storage unit 15 includes an object DB (database) 151 . The target object DB 151 stores information about the target object, image information of the target object, sensor information detected by the sensor of the terminal 2 when the target object is photographed, and the like.

In addition, in this embodiment, the storage unit 12 and the large-capacity storage unit 15 may be configured as an integrated storage device. Also, the large-capacity storage unit 15 may be composed of a plurality of storage devices. Furthermore, the large-capacity storage unit 15 may be an external storage device connected to the server 1 .

The server 1 may execute various information processing, control processing, and the like as a single computer, or may be distributed among a plurality of computers. Moreover, the server 1 may be realized by a plurality of virtual machines provided in one server, or may be realized by using a cloud server. Note that the processing of the server 1 may be executed by the terminal 2 .

FIG. 3 is an explanatory diagram showing an example of the record layout of the object DB 151. As shown in FIG. The object DB 151 includes an object ID (Identifier) column, a type column, an installation information column, an image information column, and a sensor information column. The object ID column stores the ID of a uniquely specified object in order to identify each object. The type column stores the types of objects (distribution board, distribution board, etc.). The installation information column stores owner information such as the name of the person who owns the object, or the installation position.

The image information string includes a character string, a feature point string and a feature amount string. The character string stores characters included in the image of the object, the coordinates of the characters, and the like. The feature points store the feature points included in the image of the object, the coordinates of the feature points, and the like. Characteristic points are, for example, stamped shapes or drilled holes, edge shapes, paints, seals, forced dent holes, laser markings, three-dimensional characters, or the like.

The feature quantity string stores feature quantity data of the image of the object. Feature data includes RGB (Red, Blue, Green) gradation values, YMCK (Yellow, Cyan, Magenta, Key plate) gradation values, brightness, luminance, hue, density, saturation or contrast feature values in an image, Alternatively, LBP (Local Binary Pattern) feature quantity or the like is included. The LBP feature amount is composed of relative values obtained by comparing each pixel with its surrounding neighboring pixels, with the local expression of the image as the feature amount.

The sensor information column includes a wireless LAN column, a GPS (Global Positioning System, Global Positioning Satellite) column, an atmospheric pressure column, an altitude column, a gyro sensor column, and a LiDAR (Light Detection and Ranging) sensor column. The wireless LAN column stores wireless LAN information of the terminal 2 . Note that the wireless LAN information will be described later. The GPS column stores GPS data (latitude and longitude, etc.) read via the GPS function of the terminal 2 .

The atmospheric pressure column stores atmospheric pressure data acquired by a pressure sensor (barometer) built in the terminal 2 . The altitude column stores altitude data acquired by an altitude sensor (altimeter) built in the terminal 2 . The gyro sensor array stores gyro sensor data acquired by a gyro sensor (angular velocity sensor) built in the terminal 2 . A gyro sensor is a type of inertial sensor that enables the measurement of rotational angular velocity (the speed at which an object is rotating).

The lidar sensor column stores lidar sensor data acquired by the lidar sensor built into the terminal 2 . A lidar sensor detects the distance, position, shape, etc. to a distant object by irradiating the object with scanning laser light and observing the scattered or reflected light.

Note that the storage form of each DB described above is an example, and other storage forms may be used as long as the relationship between data is maintained.

FIG. 4 is a block diagram showing a configuration example of the terminal 2. As shown in FIG. The terminal 2 includes a control section 21 , a storage section 22 , a communication section 23 , an input section 24 , a display section 25 , an imaging section 26 , a GPS module 27 , an atmospheric pressure module 28 , an altitude module 29 , a gyro module 30 and a lidar module 31 . Each configuration is connected by a bus B.

The control unit 21 includes an arithmetic processing unit such as a CPU and an MPU, and reads and executes a control program 2P (program product) stored in the storage unit 22 to perform various information processing, control processing, etc. related to the terminal 2. conduct. Note that although FIG. 4 describes the controller 21 as a single processor, it may be a multiprocessor. The storage unit 22 includes memory elements such as RAM and ROM, and stores the control program 2P or data necessary for the control unit 21 to execute processing. The storage unit 22 also temporarily stores data and the like necessary for the control unit 21 to perform arithmetic processing.

The communication unit 23 is a communication module for performing processing related to communication, and transmits/receives information to/from the server 1 or the like via the network N. The input unit 24 may be a keyboard, a mouse, or a touch panel integrated with the display unit 25 . The display unit 25 is a liquid crystal display, an organic EL (electroluminescence) display, or the like, and displays various information according to instructions from the control unit 21 .

The imaging unit 26 is an imaging device such as a CCD (Charge Coupled Device) camera, a CMOS (Complementary Metal Oxide Semiconductor) camera, or the like. Note that the photographing unit 26 may be directly connected to the terminal 2 externally, instead of being built in the terminal 2, so as to be capable of photographing.

The GPS module 27 is a module for acquiring position information (latitude and longitude, etc.) using GPS satellites. The atmospheric pressure module 28 is a sensor module for acquiring atmospheric pressure data. The altitude module 29 is a sensor module for acquiring altitude data. The gyro module 30 is a sensor module for acquiring gyro sensor data. The lidar module 31 is a sensor module for acquiring lidar sensor data. Note that the types of sensor modules are not limited to those described above, and for example, an acceleration module for acquiring acceleration sensor data may be provided.

The atmospheric pressure module 28, the altitude module 29, the gyro module 30, or the lidar module 31 may not be built in the terminal 2, but may be directly connected to the terminal 2 outside to acquire sensor data.

It should be noted that a plurality of types of the atmospheric pressure module 28, the altitude module 29, the gyro module 30, and the rider module 31 may be used together to form an integrated structure.

FIG. 5 is an explanatory diagram showing a processing operation for specifying an object photographed by the terminal 2. As shown in FIG. The terminal 2 captures an image of the object via the imaging unit 26 . The terminal 2 acquires image information related to the captured image of the object. The image information includes characters or feature points included in the image of the object, or feature amounts of the image.

For the image feature amount, the terminal 2 can, for example, use A-KAZE (Accelerated KAZE), SIFT (Scale Invariant Feature Transform), SURF (Speeded-Up Robust Features), ORB (Oriented FAST and Rotated BRIEF), or HOG (Histograms of Oriented Gradients ) is used to extract the feature amount of the captured image of the object. As for characters or feature points, for example, the terminal 2 extracts the feature amount of the image of the object, and recognizes the characters or feature points in the captured image based on the extracted feature amount. Note that characters or feature points may be automatically recognized using a learning model constructed by machine learning, or may be recognized using an image processing library, OpenCV (Open Source Computer Vision Library).

The terminal 2 acquires sensor information detected by the sensor of the terminal 2 when the object is photographed. Sensor information includes wireless LAN information, satellite positioning system data (eg, GPS data), barometric pressure data, altitude data, gyro sensor data, or lidar sensor data. Not limited to these sensor data, for example, sensor information includes AR (Augmented Reality) data, MR (Mixed Reality) data, geomagnetic (compass) sensor data, acceleration sensor data, luminance sensor data, , proximity sensor data or fingerprint sensor data. That is, sensor data detected by various types of sensors may be used according to the location, environment, situation, state, or the like at the time of photographing the object.

Terminal 2 acquires wireless LAN information using a wireless communication method. The wireless LAN information may include at least one of the MAC address (Media Access Control address) of the access point, the SSID (Service Set Identifier) that is the identifier of the wireless LAN function, the radio wave intensity of the SSID, and the frequency channel. . Wireless communication methods include BLE (Bluetooth Low Energy) communication method, communication method conforming to the Bluetooth (registered trademark) standard other than the BLE standard, communication method conforming to the short-range wireless communication standard other than the Bluetooth standard, Wi-Fi (registered Trademark) Direct communication system, terminal wireless communication system such as LTE (Long Term Evolution), 4G (Generation) or 5G.

The terminal 2 acquires GPS data (latitude and longitude, etc.) by receiving transmission radio waves from GPS positioning satellites via the GPS function of the terminal 2 . The terminal 2 acquires atmospheric pressure data, altitude data, gyro sensor data, and lidar sensor data from the pressure sensor, altitude sensor, gyro sensor, and lidar sensor built into the terminal 2 . Note that the GPS data may include altitude data. It should be noted that the present invention is not limited to these types of sensors, and can be applied to other sensors as well.

The terminal 2 transmits the acquired image information and sensor information of the object to the server 1 . In this embodiment, an example in which the terminal 2 transmits the image information of the target object to the server 1 has been described, but the present invention is not limited to this. For example, the terminal 2 may transmit the image of the target object to the server 1 . In this case, the server 1 acquires image information based on the image of the object transmitted from the terminal 2 .

The server 1 refers to the image information and sensor information corresponding to the plurality of target object candidates stored in the target object DB 151 based on the image information and sensor information of the target object transmitted from the terminal 2, and selects the plurality of target object candidates. Identify the photographed object from among the candidates.

Specifically, the server 1 stores image information of an object, a plurality of sensor information detected by each sensor built in the terminal 2, image information of each object candidate stored in the object DB 151, and Based on information from a plurality of sensors, the degree of similarity between the target object and the target object candidate is calculated for each target object candidate. For example, the server 1 calculates the degree of similarity based on each evaluation item related to the image information of the object and the information from the plurality of sensors, and the weighting corresponding to each evaluation item.

FIG. 6 is an explanatory diagram for explaining the process of calculating the degree of similarity based on weighting. The server 1 acquires each evaluation item related to the image information of the object and the information from the plurality of sensors, and the weighting corresponding to each evaluation item. Evaluation items are provided based on image information and sensor information. As illustrated, the server 1 acquires evaluation items including "feature point", "character string", "wireless LAN information", "altitude" and "gyro".

The server 1 acquires the weighting corresponding to each evaluation item. Weighting is a coefficient representing the correlation with the importance of the evaluation item. For example, five levels of weighting are provided for each evaluation item according to the degree of importance, and the weighting points in each level are 9 points, 7 points, 5 points, 3 points, and 1 point, respectively. The weighting corresponding to each evaluation item may be stored in advance in the storage unit 12 or the large-capacity storage unit 15, or the weighting setting may be accepted.

The weighting setting acceptance process may be performed on the server 1 side or the terminal 2 side. For example, the terminal 2 receives an input of weighting corresponding to each evaluation item and transmits the received weighting to the server 1 . The server 1 receives the weighting transmitted from the terminal 2 and stores it in the storage unit 12 or the large-capacity storage unit 15 .

In addition, it is possible to receive settings for weighting corresponding to each evaluation item for each object. For example, for an object that is a switchboard, the five-level weighting in each evaluation item is "9 points, 7 points, 5 points, 3 points and 1 point". Five-level weighting in each evaluation item is "10 points, 8 points, 6 points, 4 points and 2 points" for the target object which is the environment measuring device.

The server 1 calculates the score of each similarity evaluation item using a predetermined calculation formula based on the acquired evaluation items and the weights corresponding to each evaluation item. The calculation formula is not particularly limited as long as it can calculate the score of each evaluation item.

An example of the score calculation formula for the “feature point” evaluation item is represented by the following formula (1).
"Feature point" score = coefficient x matching rate x weighting (1)
A coefficient is an arbitrary numerical value (for example, 255) provided for convenience of calculation. The match rate is a comparison result of feature points obtained using, for example, OpenCV (Open Source Computer Vision Library).

An example of the score calculation formula for the “character string” evaluation item is represented by the following formula (2).
"Character string" score = coefficient x number of identical character strings x weighting (2)
The coefficients are the same as those in equation (1). The number of identical character strings is the number of identical character strings among the character strings included in the image information.

An example of the score calculation formula for the “wireless LAN information” evaluation item is represented by the following formula (3).
"Wireless LAN information" score = coefficient x number of access points x weighting (3)
The coefficients are the same as those in equation (1). The number of access points is the number of matching access points among a plurality of access points.

An example of the score calculation formula for the "advanced" evaluation item is represented by the following formula (4).
"Altitude" score = coefficient x altitude score x weighting (4)
The coefficients are the same as those in equation (1). The altitude score may be provided based on the difference in altitude. The altitude difference is the difference between the detected altitude value of the object and the detected altitude value of the candidate object. For example, if the altitude difference is less than 5 m, it is determined that the floors are on the same floor in the building, and the altitude score is 1 point. If the height difference is 5 m or more, it is determined as a different floor in the building, and the altitude score is 0 points.

An example of the score calculation formula for the “gyro” evaluation item is represented by the following formula (5).
"Gyro" score = coefficient x angular velocity coefficient x weighting (5)
The coefficients are the same as those in equation (1). Angular velocity is the rotation angle per unit time. The angular velocity coefficient is an index indicating changes in angular velocity. The angular velocity coefficient may be calculated by angular velocity coefficient=1−(XYZ)/360 using the rotation angles in the X-axis, Y-axis and Z-axis directions. For example, when the terminal 2 does not rotate in any of the three axial directions, the angular velocity coefficient calculated by angular velocity coefficient=1-(0-0-0)/360 is 1. Alternatively, when the terminal 2 rotates 10° in the X-axis direction, rotates 40° in the Y-axis direction, and does not rotate in the Z-axis direction, the angular velocity coefficient = 1-(10-40-0)/ The angular velocity coefficient calculated by 360 is 0.92.

When the server 1 acquires the weighting, image information, and sensor information corresponding to the "feature point", "character string", "wireless LAN information", "altitude", and "gyro" evaluation items, the above calculation formula ( Using 1) to (5), the score for each evaluation item is calculated. Note that the server 1 may calculate the score of at least one evaluation item among the "feature point", "character string", "wireless LAN information", "altitude", and "gyro" evaluation items. The server 1 calculates the total score of each calculated evaluation item. The server 1 calculates the maximum score of each evaluation item. The maximum score is calculated by multiplying the weight by a factor (eg, 255). For example, when the weighting corresponding to the “feature point” evaluation item is 9 points, the server 1 calculates that the maximum value of the score of the evaluation item is 2295 (255×9).

Based on the calculated total score of each evaluation item and the total maximum score of each evaluation item, the server 1 calculates the degree of similarity between the target object and the target object candidate (total score/maximum score x 100%) is calculated. As shown, the total score is 6024.8 and the maximum total score is 9435. The server 1 calculates that the degree of similarity between the target object and the target object candidate is 63.86% (6024.8/9435×100%) based on the total score and the total maximum score value.

In this way, the server 1 calculates the degree of similarity between the target object and the target object candidate for each target object candidate. The server 1 identifies the object candidate with the highest degree of similarity as the photographed object. Alternatively, a target object may be specified using a predetermined threshold value of similarity. For example, when a predetermined similarity threshold (for example, 70%) is set in advance, the server 1 determines whether or not the similarity between the target object and the target object candidate is greater than or equal to the predetermined threshold. When the server 1 determines that the degree of similarity is equal to or greater than the predetermined threshold value, the server 1 identifies the object candidate as the photographed object. When the server 1 determines that the degree of similarity is less than the predetermined threshold, it excludes the object candidate.

Note that the processing is not limited to simply specifying an object candidate with a high degree of similarity as a photographing object. For example, the server 1 acquires target object candidates up to a predetermined high order (for example, third place) in similarity from a plurality of target object candidates. In this way, it is possible to specify a plurality of object candidates by ranking them according to similarity.

Note that the similarity calculation process is not limited to the process based on the weighting described above. For example, cosine similarity, Pearson's correlation coefficient, or deviation pattern similarity may be used as the similarity. Also, the degree of similarity can be calculated using a machine learning model. A similarity calculation process using a machine learning model will be described in a second embodiment.

The server 1 acquires information about the target object from the target object DB 151 based on the target object ID of the specified target object. The information about the object includes object ID, object type or position information, and the like. The information about the object may include image information of the object and sensor information detected by the sensor of the terminal 2 when the object is photographed. The server 1 transmits the acquired information about the object to the terminal 2 . The terminal 2 receives the information about the object transmitted from the server 1 and displays the received information about the object on the screen.

In addition, when the server 1 acquires target object candidates up to a predetermined high ranking in similarity from a plurality of target object candidates, the server 1 transmits information regarding all the acquired target object candidates to the terminal 2 . The terminal 2 receives the information about all the object candidates transmitted from the server 1, and displays the received information about all the object candidates on the screen. Note that the information about the target object candidate is the same as the information about the target object, so a description thereof will be omitted.

FIG. 7 is an explanatory diagram showing an example of a target object identification screen. The specific screen includes an image display field 11a, a shooting button 11b, an image selection button 11c, a specific button 11d, a specific result display field 11e, and a candidate display field 11f.

The image display column 11a is a display column for displaying the image of the object. The shooting button 11b is a button for shooting an object. The image selection button 11c is a button for selecting an image of an object. The specifying button 11d is a button for specifying a photographed object from a plurality of object candidates. The identification result display column 11e is a display column for displaying the identification result of the object. The candidate display column 11f is a display column for displaying the identified object candidates by ranking them according to the degree of similarity.

When receiving the touch operation of the shooting button 11b, the terminal 2 shoots the object via the shooting unit 26 and acquires the image of the shot object. The terminal 2 acquires the image of the object stored in the storage unit 12 or the large-capacity storage unit 15 when receiving the touch operation of the image selection button 11c.

When the terminal 2 receives the touch operation of the specific button 11d, based on the image of the object acquired by the shooting button 11b or the image selection button 11c, image information (characters, feature points, feature amounts, etc.) related to the image is obtained. to get The terminal 2 acquires a plurality of sensor information (wireless LAN information, GPS data, atmospheric pressure data, altitude data, gyro sensor data, lidar sensor data, etc.) detected by the sensor of the terminal 2 when the object is photographed. .

The terminal 2 transmits the acquired image information and multiple sensor information to the server 1 . The server 1 receives image information and a plurality of sensor information transmitted from the terminal 2 . Based on the received image information corresponding to the target object and the plurality of sensor information, and the image information and the plurality of sensor information corresponding to the plurality of target object candidates stored in the target object DB 151, the server 1 performs Then, the similarity between the target object and the target object candidate is calculated.

Based on the calculated degree of similarity, the server 1 acquires object candidates up to a predetermined high rank (for example, third rank) in similarity from the plurality of object candidates. The server 1 acquires information about each target object candidate from the target object DB 151 based on the acquired target object ID of each target object candidate. Information about object candidates includes object ID, object type, owner information (eg owner's name) who owns the object, or location information (eg location name or latitude and longitude). . The server 1 transmits information about each acquired object candidate to the terminal 2 .

The terminal 2 receives the information about each target object candidate transmitted from the server 1 and displays it on the screen. Based on the degree of similarity between the object and the object candidate for each object candidate, the terminal 2 displays information about the object candidate with the highest degree of similarity in the identification result display field 11e, and displays information corresponding to the next degree of similarity. Information about the object candidate is displayed in the candidate display field 11f.

As shown, the object ID, object type, owner information (for example, Mr. A), and position information (position name, latitude and longitude, etc.) of the highest object candidate are displayed in the identification result display column 11e. . The object ID, object type, owner information, and position information of each object candidate corresponding to the next degree of similarity are displayed in the candidate display field 11f. Note that the degree of similarity of the target object candidates may be displayed in the identification result display field 11e and the candidate display field 11f.

Although FIG. 7 illustrates an example in which three target object candidates are acquired, the number of target object candidates is not particularly limited. When there are many object candidates, for example, when the plurality of object candidates arranged in the horizontal direction cannot fit in the candidate display field 11f, the plurality of object candidates are horizontally scrollable. Also good.

FIG. 8 is a flow chart showing a processing procedure for identifying a photographed object. The control unit 21 of the terminal 2 takes an image of the object via the imaging unit 26 (step S201). Note that if the captured image of the object is stored in advance in the storage unit 12 or the large-capacity storage unit 15, the control unit 21 may obtain the image of the object from the storage unit 12 or the large-capacity storage unit 15. good. The control unit 21 acquires image information (characters, feature points, feature amounts, etc.) related to the captured image of the object (step S202).

The control unit 21 acquires a plurality of sensor information (wireless LAN information, GPS data, atmospheric pressure data, altitude data, gyro sensor data, lidar sensor data, etc.) detected by the sensor of the terminal 2 when the object is photographed. (step S203).

Specifically, the control unit 21 acquires wireless LAN information (MAC address of access point, SSID, radio wave intensity or frequency, etc.) using a wireless communication method (eg, Wi-Fi Direct). The control unit 21 receives radio waves transmitted from GPS positioning satellites via the GPS module 27 and acquires GPS data (latitude and longitude, etc.). The control unit 21 acquires atmospheric pressure data via the atmospheric pressure module 28 . The control unit 21 acquires altitude data via the altitude module 29 . The control unit 21 acquires gyro sensor data via the gyro module 30 . The control unit 21 acquires lidar sensor data via the lidar module 31 . It should be noted that the present invention is not limited to these types of sensors, and can be applied to other sensors as well.

The control unit 21 transmits the obtained image information of the object and the information of the plurality of sensors to the server 1 through the communication unit 23 (step S204). The control unit 11 of the server 1 receives the image information of the object and the plural sensor information transmitted from the terminal 2 through the communication unit 13 (step S101). The control unit 11 executes a subroutine of processing for calculating the degree of similarity between the target object and the target object candidate based on the received image information and multiple sensor information (step S102). A subroutine for similarity calculation processing will be described later.

Based on the calculated degrees of similarity, the control unit 11 identifies the object candidate with the highest degree of similarity as the photographed object (step S103). It should be noted that the target object may be specified using a predetermined similarity threshold. For example, when the control unit 11 determines that the degree of similarity between the target object and the target object candidate is equal to or greater than a predetermined threshold value (for example, 70%), the control unit 11 identifies the target object candidate as the photographed target object.

Based on the object ID of the specified object, the control unit 11 acquires information about the object (object ID, type, position information, etc.) from the object DB 151 of the large-capacity storage unit 15 (step S104). The control unit 11 transmits the acquired information about the object to the terminal 2 through the communication unit 13 (step S105). The control unit 21 of the terminal 2 receives the information about the object transmitted from the server 1 through the communication unit 23 (step S205). The control unit 21 displays the received information about the object on the display unit 25 (step S206), and ends the process.

FIG. 9 is a flow chart showing the procedure of a subroutine of processing for calculating the degree of similarity. The control unit 11 of the server 1 acquires each evaluation item related to the image information of the target object and the information of a plurality of sensors from the storage unit 12 or the large-capacity storage unit 15 (step S01). Evaluation items include, for example, "feature point", "character string", "wireless LAN information", "altitude" and "gyro".

The control unit 11 receives settings for weighting corresponding to each evaluation item (step S02). Specifically, the control unit 21 of the terminal 2 receives the setting of weighting corresponding to each evaluation item through the input unit 24 and transmits the received weighting to the server 1 through the communication unit 23 . The control unit 11 of the server 1 receives the weighting corresponding to each evaluation item transmitted from the terminal 2 through the communication unit 13 . Note that the control unit 11 of the server 1 may directly receive settings for weighting corresponding to each evaluation item.

When the weighting corresponding to each evaluation item is stored in advance in the storage unit 12 or the large-capacity storage unit 15, the control unit 11 stores the weighting corresponding to each evaluation item in the storage unit 12 or the large-capacity storage unit 15. Get from In addition, it is possible to receive settings for weighting corresponding to each evaluation item for each object.

The control unit 11 acquires image information and sensor information corresponding to a plurality of object candidates from the object DB 151 of the large-capacity storage unit 15 (step S03). The control unit 11 acquires image information and sensor information corresponding to one target object candidate from the multiple acquired target object candidates (step S04). The control unit 11 calculates the score of each evaluation item using the above-described calculation formulas (1) to (5) based on the weighting corresponding to each evaluation item, the image information, and the information from the plurality of sensors (step S05 ).

The control unit 11 calculates the total score of each calculated evaluation item (step S06). The control unit 11 calculates the maximum score of each evaluation item by multiplying the coefficient (for example, 255) by the weight (step S07). The control unit 11 calculates the degree of similarity between the target object and the target object candidate (total score/maximum score×100%) based on the calculated total score and maximum score (step S08).

The control unit 11 determines whether or not the target object candidate is the last target object candidate among the plurality of target object candidates (step S09). When the control unit 11 determines that the target object candidate is not the last target object candidate (NO in step S09), the process returns to step S04. When determining that the target object candidate is the last target object candidate (YES in step S09), the control unit 11 acquires the similarity calculated for each target object candidate (step S10). The control unit 11 ends the subroutine of the similarity calculation process and returns.

According to this embodiment, it is possible to specify a photographed object from a plurality of object candidates based on image information and sensor information.

According to this embodiment, it is possible to output information about the specified object.

According to this embodiment, it is possible to obtain target object candidates up to a predetermined high rank in similarity from a plurality of target object candidates, and output information about all the obtained target object candidates.

(Embodiment 2)
In the second embodiment, based on image information and sensor information corresponding to the target object and image information and sensor information corresponding to the target object candidate, AI (Artificial Intelligence) identifies the target object and the target object candidate. It relates to a form of outputting the similarity of . In addition, description is abbreviate|omitted about the content which overlaps with Embodiment 1. FIG.

FIG. 10 is a block diagram showing a configuration example of the server 1 according to the second embodiment. In addition, the same code|symbol is attached|subjected about the content which overlaps with FIG. 2, and description is abbreviate|omitted. The large-capacity storage unit 15 of the server 1 contains a similarity output model 152 and a training data DB 153 .

The similarity output model 152 outputs (estimates) the similarity between the target object and the target object candidate based on the image information and sensor information corresponding to the target object and the image information and sensor information corresponding to the target object candidate. ) and is a trained model generated by machine learning. The training data DB 153 stores training data for constructing (creating) the similarity output model 152 .

FIG. 11 is an explanatory diagram showing an example of the record layout of the training data DB 153. As shown in FIG. The training data DB 153 includes input data strings and output data strings. The input data string includes an object string and an object candidate string. The target object column stores image information and sensor information corresponding to the target object. The target object candidate string stores image information and sensor information corresponding to the target object candidate. The output data string stores information on whether or not the target object matches the target object candidate (for example, match or mismatch).

FIG. 12 is an explanatory diagram for explaining the similarity output model 152. As shown in FIG. Similarity output model 152 is utilized as a program module that is part of artificial intelligence software. The similarity output model 152 receives image information and sensor information corresponding to an object and image information and sensor information corresponding to the object candidate, and outputs the similarity between the object and the object candidate. It is an output device with a built-in neural network.

The neural network is, for example, a CNN (Convolutional Neural Network). and an intermediate layer that has been learned by back propagation. Each layer has one or more neurons (nodes), and each neuron has a value. Neurons between one layer and the next layer are connected by edges, and each edge has variables (or parameters) such as weights and biases.

In the CNN, the values of neurons in each layer are obtained by executing a predetermined operation based on the values of neurons in the preceding layer and edge weights. Then, when the input data is input to the neurons of the input layer, the values of the neurons of the next layer are obtained by a predetermined operation. It is obtained by a predetermined operation of layers. Then, the value of the neuron in the output layer, which is the final layer, becomes the output data for the input data.

For example, the server 1 uses training data accumulated in the training data DB 153 to generate the similarity output model 152 . Each record of the training data DB 153 is training data. The value of the output data string is the correct data (information as to whether or not the target object matches the target object candidate) to be output from the output layer. The input data are image information and sensor information corresponding to the target object in the input data string and image information and sensor information corresponding to the target object candidate.

The training data includes image information and sensor information corresponding to the object, image information and sensor information corresponding to the object candidate, and information indicating whether or not the object matches the object candidate. It is the data of the combination attached. Training data is generated based on large amounts of empirical data collected from objects. Note that the training data may be data separately created manually.

The server 1 performs learning using the acquired training data. Specifically, the server 1 inputs image information and sensor information corresponding to an object, which is training data, and image information and sensor information corresponding to the object candidate into the input layer, and performs calculations in the intermediate layer. After processing, the similarity between the target object and the target object candidate is output from the output layer. The output layer includes, for example, a sigmoid function or a softmax function, and outputs estimated similarity based on the feature quantity output from the intermediate layer.

The server 1 compares the similarity (estimation result) output from the output layer with the correct value (value of the output data string) in the training data, and adjusts the intermediate layer so that the output value from the output layer approaches the correct value. Optimize the parameters used in the calculation process. The parameters are, for example, weights (coupling coefficients) between neurons. Although the parameter optimization method is not particularly limited, for example, the server 1 optimizes various parameters using the error backpropagation method.

The server 1 performs the above processing for each record stored in the training data DB 153 and learns the similarity output model 152 . This makes it possible to build a model capable of outputting the degree of similarity between the target object and the target object candidate. Note that the similarity output model 152 may be generated by performing the above-described learning process using another computer (not shown). In this case, the server 1 obtains and installs the similarity output model 152 generated by another computer. The similarity may be output by using a WEB API (Application Programming Interface) using a machine learning model without building the similarity output model 152 .

When the server 1 acquires the image information and the sensor information corresponding to the target object and the image information and the sensor information corresponding to the target object candidate, the server 1 acquires the image information and the sensor information corresponding to the acquired target object and the target object. Image information and sensor information corresponding to object candidates are input to the similarity output model 152 . The server 1 extracts features of the input information by changing the number of dimensions of the input information input from the input layer in the intermediate layer of the similarity output model 152 . The server 1 inputs the extracted features to the output layer of the similarity output model 152 and outputs an estimation result of estimating the degree of similarity between the target object and the target object candidate.

As shown in the figure, the similarity probability values estimated by the similarity output model 152 for the image information and sensor information corresponding to the target object and the image information and sensor information corresponding to the target object candidate include , the highest probability is output. For example, the estimated similarity is 75.4%. The output result is not limited to the above-described output result, and the estimated probability value of each degree of similarity may be directly output.

In addition, the similarity output model 152 is not limited to CNN, RCNN (Regions with Convolutional Neural Network), Fast RCNN, Faster RCNN, SSD (Single Shot Multibook Detector), YOLO (You Only Look Once), SVM (Support Vector Machine ), Bayesian Networks, Transformer Networks, Regression Trees or Random Forests.

FIG. 13 is a flowchart showing the procedure of a subroutine for similarity calculation processing using the similarity output model 152 . The control unit 11 of the server 1 acquires image information and sensor information corresponding to multiple object candidates from the object DB 151 of the large-capacity storage unit 15 (step S21). The control unit 11 acquires image information and sensor information corresponding to one target object candidate from the multiple acquired target object candidates (step S22).

The control unit 11 inputs the image information and sensor information corresponding to the object and the image information and sensor information corresponding to the acquired object candidate to the similarity output model 152 (step S23). The control unit 11 outputs the degree of similarity between the target object and the target object candidate from the similarity output model 152 (step S24).

The control unit 11 determines whether or not the target object candidate is the last target object candidate among the plurality of target object candidates (step S25). When the control unit 11 determines that the target object candidate is not the last target object candidate (NO in step S25), the process returns to step S22. When determining that the target object candidate is the last target object candidate (YES in step S25), the control unit 11 acquires the similarity calculated for each target object candidate (step S26). The control unit 11 ends the subroutine of the similarity calculation process and returns.

According to this embodiment, based on the image information and sensor information corresponding to the target object and the image information and sensor information corresponding to the target object candidate, the similarity output model 152 is used to extract the target object and the target object candidate. It is possible to output the similarity with

According to this embodiment, it is possible to obtain a highly accurate similarity by outputting the similarity using the similarity output model 152 .

The embodiments disclosed this time are illustrative in all respects and should be considered not restrictive. The scope of the present invention is indicated by the scope of the claims rather than the above-described meaning, and is intended to include all modifications within the scope and meaning equivalent to the scope of the claims.

1 Information processing device (server)
REFERENCE SIGNS LIST 11 control unit 12 storage unit 13 communication unit 14 reading unit 15 large-capacity storage unit 151 object DB
152 similarity output model 153 training data DB
1a portable storage medium 1b semiconductor memory 1P control program 2 information processing terminal (terminal)
21 control unit 22 storage unit 23 communication unit 24 input unit 25 display unit 26 photographing unit 27 GPS module 28 atmospheric pressure module 29 altitude module 30 gyro module 31 rider module 2P control program

Claims (11)

Acquiring image information related to an image of an object photographed by an information processing terminal,
Acquiring sensor information detected by a sensor of the information processing terminal when the object is photographed,
referring to a storage unit storing image information and sensor information corresponding to a plurality of object candidates, and specifying a photographed object from among the plurality of object candidates based on the obtained image information and sensor information; Processing method. The information processing terminal includes a plurality of sensors,
Based on the acquired image information, a plurality of sensor information detected by each sensor, and the image information and the plurality of sensor information corresponding to each target object candidate stored in the storage unit, the target object for each target object candidate and calculating the similarity between the object candidate and
The information processing method according to claim 1, wherein an object candidate with the highest degree of similarity is specified as the photographed object. Acquiring each evaluation item related to the image information of the object and the plurality of sensor information and the weighting corresponding to each evaluation item,
The information processing method according to claim 2, wherein the degree of similarity is calculated for each of the object candidates based on each acquired evaluation item and weighting corresponding to each evaluation item. 4. The information processing method according to claim 3, wherein setting of weighting corresponding to each evaluation item is received for each object. acquiring image information and sensor information corresponding to a plurality of object candidates from the storage unit;
When image information and sensor information corresponding to the object and image information and sensor information corresponding to the object candidate are input, learning is performed to output the degree of similarity between the object and the object candidate. inputting image information and sensor information corresponding to the target object and image information and sensor information corresponding to the obtained target object candidate to the learning model, and outputting a similarity for each target object candidate;
The information processing method according to claim 1, wherein an object candidate with the highest degree of similarity is specified as the photographed object. acquiring target object candidates up to a predetermined high rank in similarity from the plurality of target object candidates;
6. The information processing method according to any one of claims 2 to 5, further comprising: outputting information about all acquired object candidates. 7. The information processing method according to any one of claims 1 to 6, further comprising outputting information about the specified object. The information processing method according to any one of claims 1 to 7, wherein the image information includes characters or feature points included in the image of the object, or feature amounts of the image. The information processing method according to any one of claims 1 to 8, wherein the sensor information includes wireless LAN information, satellite positioning system data, atmospheric pressure data, altitude data, gyro sensor data, or lidar sensor data. Acquiring image information related to an image of an object photographed by an information processing terminal,
Acquiring sensor information detected by a sensor of the information processing terminal when the object is photographed,
referring to a storage unit storing image information and sensor information corresponding to a plurality of object candidates, and identifying the photographed object from among the plurality of object candidates based on the acquired image information and sensor information; A program that makes a computer run a first acquisition unit that acquires image information related to an image of an object captured by an information processing terminal;
a second acquisition unit that acquires sensor information detected by a sensor of the information processing terminal when the object is photographed;
Identifying by referring to a storage unit storing image information and sensor information corresponding to a plurality of object candidates and identifying a photographed object from among the plurality of object candidates based on the acquired image information and sensor information An information processing device comprising:

Images