JP2020038071A - Information processing device, information processing method, and program - Google Patents

Abstract

To provide an information processing device, an information processing method, and a program for improving convenience at a time of meeting.SOLUTION: The information processing device includes: a first acquisition unit configured to acquire information about a meeting point at which a second merging entity and a first merging entity join; a second acquisition unit configured to acquire an image relating to the meeting point at which the second merging entity and the second merging entity join; and a position estimation unit configured to estimate a position of the meeting point on the basis of the information related to the meeting point acquired by the first acquisition unit and the image acquired by the second acquisition unit.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an information processing device, an information processing method, and a program.

  Conventionally, a position for estimating the position of a moving object with high accuracy based on map data, position information of the moving object equipped with the device, and a detection result of an object such as a landmark existing around the moving object. An estimation device is known (see Patent Document 1).

  Also, in recent years, a generation method has been disclosed in which a stereoscopic image diagram is generated from a still image from two or more viewpoints, which predicts and reproduces the positional relationship of a still object reflected in the still image (see Non-Patent Document 1).

JP 2015-230213 A

Ali Eslami and 22 others, "Neural scene representation and rendering", [online], [searched August 1, 2018], Internet <https://deepmind.com/blog/neural-scene-representation-and-rendering / ＞

  However, according to the conventional technology, a meeting place cannot be naturally estimated based on a flow of a conversation (such as a chat) by voice or text. For this reason, convenience was not sufficient.

  SUMMARY An advantage of some aspects of the invention is to provide an information processing device, an information processing method, and a program that can improve convenience at the time of a meeting.

  One aspect of the present invention is a first acquisition unit that acquires information about a merging point where a first merging body and a second merging body merge, a second acquiring unit that acquires an image about the merging point, and the first acquisition body. An information processing apparatus comprising: a position estimating unit configured to estimate a position of the merging point based on the information acquired by the unit and the image acquired by the second acquiring unit.

  According to one embodiment of the present invention, convenience at the time of meeting can be improved.

FIG. 2 is a diagram illustrating a use environment of the information processing apparatus 100. FIG. 2 is a block diagram of the information processing apparatus 100. FIG. 5 is a diagram illustrating a method of interpreting a first area by a semantic interpreting unit 142. It is a figure which shows typically the estimation method of the scenery from a specific position by GQN. FIG. 4 is a diagram illustrating a method of estimating a waiting position by a position estimating unit 140. 9 is a flowchart illustrating an example of a flow of a process of the information processing apparatus 100.

  Hereinafter, embodiments of an information processing apparatus, an information processing method, and a program according to the present invention will be described with reference to the drawings.

  The information processing device is realized by one or more processors. The information processing device is a device that assists the first merging entity and the second merging entity to merge at a merging point. The merge is, for example, a meeting for transfer of luggage or pick-up and drop-off. For example, one of the first merging body and the second merging body is a user who specifies a merging point, and the other is a moving body (an autonomous vehicle or a drone) having an imaging unit.

  The information processing device is, for example, a case where a user cannot accurately explain a merging point when the user requests transfer and luggage transfer to and from a moving body in an unfamiliar place such as a travel destination and waits. Is a device that assists a user and a moving body to meet by estimating a merging point. Further, the information processing device supports a user and a mobile body even when a driver (or a pilot) of the mobile body exists and the person is not familiar with the geography around the merging point. .

  The information processing device generates, for example, peripheral information of the merging point based on one or more images captured after the moving object arrives around the merging point, and estimates the merging point based on the generated geographic information. . The peripheral information is geographical information around the merging point (for example, a result of positioning by a GNSS (Global Navigation Satellite System) receiver) and a three-dimensional space of a target such as a road, a sign, a building, or a sign near the merging point. And the arrangement information in.

  The information processing device estimates the merging point by estimating the scenery seen by the user based on the surrounding information and the description of the merging point of the user, and assists the moving body to arrive at the position.

[overall structure]
FIG. 1 is a diagram illustrating a use environment of the information processing apparatus 100. The information processing apparatus 100 communicates with the mobile unit V and the user device UD via the network NW, for example. The information processing apparatus 100 receives, from the user P via the user device UD, features near the merging point, names of streets facing the merging point, characteristics of buildings seen from the merging point, information on the current location of the user P, and the like. Is estimated on the basis of. Note that the information processing device 100 may be mounted on the moving object V.

  The user P informs the information processing device 100 of the merging point, such as “Please come to A department store to pick me up.” The information processing apparatus 100 communicates the merging point to the moving body V and causes the moving body V to move to the vicinity of the merging point. When the moving object V arrives near the merging point, the moving object V captures an image around the moving object V. The information processing apparatus 100 acquires the image from the moving object V, and notifies the moving object V of the estimation result of the waiting position based on the information of the merging point and the image. The waiting position is information of a position (higher granularity) that is more detailed than the merging point, and is represented by coordinates such as latitude, longitude and altitude. Hereinafter, this detailed position is referred to as a coordinate.

  In addition, the information processing device 100 is also notified by the user P together with information for specifying the merging point, such as "There is a building with a signboard of company B in front of the merging point." In that case, the information is used for estimating the waiting position. Thereby, even if the merging point has a plurality of entrances and exits, and it is a facility (for example, a terminal station or a large commercial facility) where it is difficult to specify the position specified by the user, the meeting position can be estimated.

  FIG. 2 is a block diagram of the information processing apparatus 100. The information processing device 100 includes, for example, a communication unit 110, a first acquisition unit 120, a second acquisition unit 130, a position estimation unit 140, a waiting support unit 150, an alternative position setting unit 160, a storage unit 170, Is provided.

  The communication unit 110 communicates with the user device UD and the mobile unit V. The communication unit 110 receives, for example, information about the merging point from the user device UD as information of a natural language (for example, voice data or text data), and outputs the information to the first acquisition unit 120 and the position estimation unit 140. The communication unit 110 outputs the image received from the moving object V to the second acquisition unit 130. Further, communication unit 110 transmits the estimation result of position estimation unit 140 to mobile unit V.

  The first acquisition unit 120 acquires information on the merging point via the communication unit 110 as described above, and outputs the information to the position estimation unit 140 and the meeting support unit 150.

  The second acquisition unit 130 acquires an image around the moving object V from the moving object V via the communication unit 110 as described above. The image acquired by the second acquisition unit 130 is stored in the peripheral information 172a stored in the storage unit 170. The second acquisition unit 130 outputs a part or all of the acquired image to the position estimation unit 140. When there is related information stored in the peripheral information 172a (for example, an image of the same point captured in the past from another angle), the second acquisition unit 130 also includes the relevant information together with the position estimating unit 140. May be output.

  The position estimating unit 140 calculates the coordinates of the point intended by the user as the merging point based on the audio data or text data of the information related to the merging point output by the communication unit 110 and the image output by the second acquiring unit 130. (Hereinafter, a meeting position) is estimated.

  The position estimating unit 140 includes, for example, a semantic interpreting unit 142, an image analyzing unit 144, and a merging point estimating unit 146.

  The meaning interpreting unit 142 interprets the meaning of the information about the merging point by reading the information about the merging point from the information about the merging point as the location information on the map such as the address of the merging point or the information (for example, the facility name) corresponding to the address of the merging point. I do. When the information about the merging point is audio data, the semantic interpretation unit 142 converts the audio data into text data by performing morphological analysis. The semantic interpretation unit 142 performs processing such as extraction of proper nouns on the text data, and acquires the first area from the map information 172 stored in the storage unit 170 based on the extracted address or information corresponding to the address. I do. The first area is indicated by an address indicating an integrated area including a merging point such as “Nihonbashi, Chuo-ku, Tokyo” or “Burbank, Los Angeles County, California, USA”. The map information 172 is, for example, information expressing a link indicating a road, a node connected by the link, and a target (a building or the like) around the link.

  For example, when the information on the merging point provided by the user P is an explanation such as “Please come to the front of the cafe facing the street on the north side of Station C.” By interpreting the positions indicated by “Station C”, “North Street”, and “Cafe”, the address corresponding to the merging point indicated by the user P is acquired from the map information 172.

  In addition, the meaning interpreting unit 142 determines candidates for the queuing position by the user P interpreting the information describing the merging point. The semantic interpreting unit 142 determines the waiting position indicated by the user P based on information provided by the user P, for example, “There is a building with a signboard of the company B in front of the merging point.” Interpret the positional relationship that the mark "building with the signboard of Company B" exists "in front" of the building.

  Note that if the interpreted waiting position candidate does not exist in the area indicated by the first area, that is, if there is a high possibility that the extraction result of the first area is incorrect, the semantic interpreting unit 142 determines the interpreted waiting position. The destination of the mobile unit V may be corrected by transmitting the candidate to the mobile unit V via the communication unit 110. The meaning interpreting unit 142 outputs the information on the interpreted position to the waiting support unit 150.

  The image analysis unit 144 analyzes the image output by the second acquisition unit 130, and determines a candidate for a waiting position. For example, when “A department store” is designated as the merging point, the image analysis unit 144 analyzes the image and specifies an area where the A department store is reflected in the image. Further, the image analysis unit 144 specifies an area in which the target listed by the user P for explaining the meeting position is reflected. Note that the image analysis unit 144 may also analyze an image included in the related information stored in the peripheral information 172a.

  The image analysis unit 144, for example, distinguishes between a region necessary for the process of estimating the merging point in the image and an unnecessary region, and determines a feature point (for example, an edge or a contour of a target such as a building) in the necessary region. ) Is extracted. Further, the image analysis unit 144 may recognize characters such as signs and signboards in the image.

  The merging point estimating unit 146 estimates a waiting position based on the interpretation result of the meaning interpreting unit 142 and the analysis result of the image analyzing unit 144. When there are a plurality of merging point candidates, the merging point estimating unit 146 estimates the most likely waiting position from the candidates. The processing of the merging point estimation unit 146 is realized by, for example, a GQN (Generative Query Network).

  GQN is a rendering method for estimating an invisible part from visible information. Rendering is a method of generating an image or the like from abstract high-order information. For example, when the tabletop and three legs of a table with a four-legged structure are visible, it is visually recognized by rendering based on visible information (tabletop and three legs of the table) and the angle of the viewpoint. The position of another leg that is not present can be estimated, and an image in which the leg can be seen can be derived as a calculation result.

  The merging point estimating unit 146 may predict a landscape viewed from the moving object V at the waiting position as a result of estimating the waiting position, and generate the predicted scene. Note that the merging point estimating unit 146 may use the related data of the peripheral information 172a to generate the predicted scenery.

  The merging point estimating unit 146 derives the waiting position and the accuracy of the waiting position. When the accuracy is equal to or higher than the predetermined threshold, the merging point estimation unit 146 transmits the estimation result to the mobile unit V via the communication unit 110. When the accuracy is less than the predetermined threshold, the merging point estimation unit 146 outputs the merging point candidates to the alternative position setting unit 160. In addition, when the merging point specified by the user P is unsuitable for getting on and off (for example, when a waiting position exists in an area where getting on and off is restricted), the merging point estimation unit 146 substitutes a merging point candidate. Output to position setting section 160.

  The queuing support unit 150 supports queuing by controlling the transfer of information required for queuing between the mobile unit V and the user. The queuing support unit 150 sends the first area to the mobile unit V via the communication unit 110, thereby moving the mobile unit V to the first area. In addition, when the moving object V approaches the periphery of the first area, the waiting support unit 150 issues an instruction to capture an image of the periphery acquired by the second acquisition unit 130.

  The meeting support unit 150 notifies the user that the moving object V has arrived at the meeting position. In addition, when the alternative position setting unit 160, which will be described later, sets an alternative merging point, the queuing support unit 150 transmits the location to the user.

  When the accuracy of the waiting position estimated by the position estimating unit 140 is less than a predetermined threshold, the substitute position setting unit 160 sets a substitute waiting position (substitute position) from the surrounding information, and Send to device UD. The alternative position setting unit 160 may set an alternative position from the merging point candidates output by the position estimating unit 140, or may set a new merging point.

  The user device UD accepts a user P's response input as to whether or not he / she agrees to change to the alternative position transmitted from the information processing apparatus 100. If the user P does not agree with the change to the alternative position, the user device UD further accepts the designation of a third merging point (a place that is not the first designated merging point and is not a substitute position), and It is transmitted to the processing device 100. The consent to the alternative position transmitted from the user device UD or the information of the third merging point is input to the queuing support unit 150 via the communication unit 110 and is processed.

  Note that the communication unit 110 transmits the information about the merging point to the coordinate information (for example, a result of positioning by the GNSS receiver provided in the user device UD at the waiting position, a predetermined area (for example, an area that can be limited to about 10 square meters). ) May be received. In that case, the coordinate information received by the communication unit 110 may be used as a candidate for the first area and the waiting position.

[First area]
Hereinafter, a method of interpreting the first area by the semantic interpreting unit 142 will be described. FIG. 3 is a diagram illustrating a method of interpreting the first area by the semantic interpreting unit 142. For example, when the information regarding the merging point is “Please come to A department store.”, The meaning interpreting unit 142 interprets “A department store” from the information and outputs the map information 172 of the area including the A department store. An arbitrary place is set as the first area.

  For example, when acquiring the map information 172 including the A department store as illustrated in FIG. 3, the semantic interpreting unit 142 selects one of the coordinates of C1 to C4 that is the end of the map information 172 as the first area, or the A department store. Any one of the coordinates of the rectangular center of gravity C5 surrounding the map information 172 including is set as the first area.

  Further, if the meaning interpreting unit 142 has history information that has been used as a waiting position at the department store A in the past, it may select any one of the places (C6 to C8) and set it as the first area.

[Prediction scenery]
Hereinafter, an example of a method of estimating the waiting position by the merging point estimation unit 146 will be described. FIG. 4 is a diagram schematically illustrating a method of predicting a landscape from a specific position using GQN.

  The GQN estimates, for example, a view seen from a new viewpoint other than the imaging position, based on images captured from one or more viewpoints (viewpoints S1 to S3 in FIG. 4). The landscape estimation method using GQN is realized in three steps: an image acquisition step, a model generation step, and a prediction step.

  In the image acquiring step, the merging point estimating unit 146 acquires, for example, an image captured from the viewpoints S1 to S3 in FIG. 4 and information indicating the viewpoint of the image (information on the angle and direction indicating the direction of the line of sight). Then, the process proceeds to the model generation step. Note that a combination of a large rectangle and a small rectangle at the viewpoints S1 to S3 indicates a camera. A large rectangle indicates the imaging position of the camera, and a small rectangle indicates the imaging direction of the camera (a direction not in contact with the large rectangle). Image pickup).

  In the model generation step, the merging point estimation unit 146 generates an operation model. The arithmetic model is configured by, for example, a multi-stage neural network. When the operation model is composed of a two-stage neural network of a first network and a second network, the first network extracts a characteristic target of the landscape (for example, a sphere, a cube, a cone, Recognize each wall). The first network is also called an expression network. In addition, the second network generates an image of a landscape that can occur from a new viewpoint (for example, the position WP in the figure). The resolution of the image generated by the second network is equal to or lower than the input image. A reproducibility of the landscape (accuracy information) may be set together with the image of the possible landscape. The second network is also called a generating network.

  In the prediction step, the merging point estimation unit 146 inputs a parameter of information indicating a viewpoint corresponding to the position WP to the calculation model generated in the model generation step, and applies the parameter to the calculation model to generate an image of the predicted scenery. Generate

[Coordinate estimation]
The merging point estimation unit 146 further estimates a waiting position based on the image of the predicted scenery and the image around the moving object V acquired by the second acquisition unit 130. FIG. 5 is a diagram illustrating a method of estimating a waiting position using an image of a predicted scene by the position estimating unit 140.

  The left diagram in FIG. 5 shows the position on the map, the imaging direction, and the combination of the images captured at regular intervals (time or travel distance) after the moving object V arrives around the first area. FIG. The image G2 (t2) in the lower left diagram of FIG. 5 is obtained at a time t2 after a certain interval has elapsed from the timing (t1) when the image G1 (t1) in the upper left diagram of FIG.

  The image analysis unit 144 recognizes the targets OB1 to OB7 around the first area based on the image analysis results of the image G1 (t1) and the image G2 (t2), for example. Further, the image analysis unit 144 generates a peripheral information image NI that is arrangement information of the recognized targets OB1 to OB7 in a three-dimensional space.

  The lower right diagram of FIG. 5 is a diagram illustrating a predicted scenery PL at the waiting position. For example, when the user has received a position designation "a building with a signboard of company B is in front of the junction point" by the user, the merging point estimation unit 146 adds the signboard (PO1) of company B. A scene including the building (PO2) that has been set is predicted.

  The lower right diagram of FIG. 5 is a diagram illustrating a result of estimating the waiting position by the merging point estimating unit 146. The merging point estimating unit 146 includes the predicted scenery PL generated by the merging point estimating unit 146, the image G1 (t1) and the image G2 (t2) around the moving object V acquired by the second acquiring unit 130, and the image analyzing unit. The waiting position specified by the user is estimated based on the peripheral information image NI generated by the user. The merging point estimating unit 146, for example, when receiving from the user the position designation "There is a building with a signboard of Company B in front of the merging point", the predicted scenery PL and the surrounding information image NI Is performed, and the position WP around the first area and where the signboard of the company B can be seen is estimated. The matching process includes, for example, the presence or absence of the signboard (OB7) of the peripheral information image NI corresponding to the signboard (PO1) of the company B of the predicted scenery PL, and the PO1 and OB7 existing on the buildings (PO2 and OB4). Is to derive and evaluate the similarity based on the arrangement location of. Further, the matching process checks whether a known landscape is completed by matching the insufficient portion of the predicted landscape PL and the peripheral information image NI, that is, whether the predicted landscape PL and the peripheral information image NI correspond to each other. It may include evaluating.

  In addition, the process of creating the predicted scenery PL may be performed not only based on the information acquired by the first acquisition unit 120 but also by taking into account the analysis result of the image analysis unit 144. In this case, the process of creating the predicted scenery PL and the image analysis process may be performed in parallel so as to complement each other.

  In the right diagram of FIG. 5, the position WP is a position where the target (signboard) OB7 can be seen and a position close to the target OB1 corresponding to “A department store” input as the junction point, so that the junction point is estimated. The unit 146 determines that the accuracy is equal to or greater than the predetermined threshold, and estimates that the position WP is a waiting position.

  The merging point estimation unit 146 determines whether the specified position WP is an appropriate position. The merging point estimation unit 146 causes the alternative position setting unit 160 to set alternative coordinates, for example, when the estimated position WP is a parking prohibited area around the position WP. In addition, for example, in the right diagram of FIG. 5, when the merging point estimation unit 146 estimates that the position WP is a position distant from the target OB1 (for example, another adjacent block), the accuracy is less than a predetermined threshold. It is determined that there is, and the substitute position setting unit 160 sets the substitute coordinates.

[Processing flow]
FIG. 6 is a flowchart illustrating an example of a processing flow of the information processing apparatus 100. First, the communication unit 110 receives information on the merging point (S100). Next, the first obtaining unit 120 obtains information on the merging point and outputs the information to the position estimating unit 140 (S102). Next, the semantic interpreting unit 142 interprets information on the first area and the merging point (S104).

  Next, the communication unit 110 receives an image around the first area captured by the moving object V (S106). Next, the image analysis unit 144 analyzes the image around the first area acquired by the second acquisition unit 130 (S108). Next, the merging point estimation unit 146 estimates a merging point (S110).

  Next, the merging point estimating unit 146 obtains the accuracy of the waiting position, and determines whether or not the accuracy is equal to or greater than a threshold (S112). If the merging point estimating unit 146 determines that the accuracy is equal to or higher than the threshold, the merging point estimating unit 146 determines whether the merging point is at an appropriate position (S114). If it is determined that the position is an appropriate position, the queuing support unit 150 transmits the position information of the merging point to the moving object V, moves the moving object V to the merging point (S116), and ends the processing of this flowchart. I do.

  If it is not determined in S112 that the accuracy is equal to or greater than the threshold, or if it is not determined in S114 that the position is appropriate, the queuing support unit 150 causes the alternative position setting unit 160 to set an alternative joining point ( (S118), the process returns to S100. This is the end of the description of the processing in this flowchart.

  According to the information processing apparatus 100 of the embodiment described above, the first acquisition unit 120 that acquires, as information in natural language, information about a merging point where the first merging body and the second merging body merge, and an image about the merging point. Position estimation for estimating the position of the merging point based on the information about the merging point acquired by the second acquiring unit 130 and the information about the merging point acquired by the first acquiring unit 120 and the image about the merging point acquired by the second acquiring unit 130. By providing the unit 140, the convenience at the time of meeting can be improved.

  As described above, the embodiments for carrying out the present invention have been described using the embodiments. However, the present invention is not limited to these embodiments at all, and various modifications and substitutions may be made without departing from the gist of the present invention. Can be added.

  Reference Signs List 100 information processing device, 120 first acquisition unit, 130 second acquisition unit, 140 position estimation unit, 142 meaning interpretation unit, 160 alternative position setting unit, UD user device, V mobile

Claims (11)

A first acquisition unit that acquires information about a merging point where the first merging body and the second merging body merge;
A second acquisition unit that acquires an image related to the merging point;
A position estimating unit that estimates a position of the merging point based on the information acquired by the first acquiring unit and the image acquired by the second acquiring unit;
An information processing device comprising: The position estimating unit estimates a more detailed position of the merging point than information on the merging point acquired by the first acquiring unit,
The information processing device according to claim 1. The merging point is a point where the first merging body and the second merging body meet.
The information processing apparatus according to claim 1. The first acquisition unit acquires information about the merging point as information in a natural language.
The information processing apparatus according to claim 1. Further comprising a semantic interpreting unit that interprets the meaning of the information acquired by the first acquiring unit, and reads the interpreted information into position information on a map including the merging point;
The information processing device according to claim 4. The position estimating unit generates the peripheral information from the image acquired by the second acquiring unit, when there is no peripheral information that is information on the arrangement of the target in the three-dimensional space in the position information, Estimating the position based on information;
The information processing device according to claim 5. The position estimating unit, when estimating the position, predicts by generating a landscape at the position by rendering realized by a Generative Query Network,
The information processing device according to claim 6. The position estimating unit estimates the position by comparing the predicted presence or absence of a target of the landscape and the surrounding information, and the arrangement of the target,
The information processing device according to claim 7. When the position cannot be specified by the position estimating unit, a new merging point candidate is set from the image, and an alternative position setting unit is further provided.
The information processing device according to claim 1. Computer
Obtain information about the merging point where the first merging body and the second merging body merge,
Obtaining an image of the merging point,
Estimating the position of the merging point based on the information and the image,
Information processing method. On the computer,
Information about the merging point where the first merging body and the second merging body merge,
Causing an image of the merging point to be acquired,
Based on the information and the image, the position of the merging point is estimated,
program.

Images