WO2021125190A1 - Information processing device, information processing system, and information processing method - Google Patents

Abstract

[Problem] To provide an information processing device, an information processing system, and an information processing method that make it possible to increase the quality of a service using AR and the satisfaction level of a user. [Solution] An information processing device according to the present disclosure comprises: a map generation unit that generates a first map on the basis of an image; a map acquisition unit that acquires a second map from another device; an object generation unit that uses the first map or the second map to generate a virtual object; and a display unit that displays the virtual object generated by the object generation unit as superposed on a background.

Description

Information processing equipment, information processing system and information processing method

This disclosure relates to an information processing device, an information processing system, and an information processing method.

Augmented Reality (AR) can be used to provide users with images in which virtual objects such as characters, texts, icons, items, and animations are superimposed in the real world. In addition, information and communication technology has realized services and applications that can show the same virtual object to a plurality of users.

Japanese Unexamined Patent Publication No. 2011-186808 Japanese Unexamined Patent Publication No. 2014-203175

Even if the virtual object is displayed at the same coordinates in the system, the appearance of the virtual object may be different depending on the user. Therefore, there is a problem that the quality of the service using AR and the satisfaction of the user are lowered.

In view of the above-mentioned problems, the present disclosure provides an information processing device, an information processing system, and an information processing method that can improve the quality of services using AR and the satisfaction of users.

The information processing device according to one aspect of the present disclosure includes a map generation unit that generates a first map based on an image, a map acquisition unit that acquires a second map from another device, and the first map or the second map. It may have an object generator that creates a virtual object using.

An evaluation data generation unit that generates first evaluation data based on the first map and a third map used for managing services, and the first map or the first map that the object generation unit uses to generate the virtual object. 2 A map selection unit for selecting a map based on the first evaluation data may be further provided.

The map selection unit further includes a self-position estimation unit that estimates the self-position, acquires a first evaluation value corresponding to the self-position from the first evaluation data, and generates the object based on the first evaluation value. The first map or the second map that the unit uses to generate the virtual object may be selected.

The map selection unit may select the first map when the first evaluation value is equal to or less than the threshold value.

The map acquisition unit acquires the second evaluation data corresponding to the second map from the other device, and the map selection unit acquires the second evaluation value corresponding to the self-position from the second evaluation data. Then, the second map may be selected when the second evaluation value is larger than the first evaluation value.

The map selection unit may select the first map when the second evaluation value is equal to or less than the first evaluation value.

The object generation unit may generate the virtual object having a size larger than when the first evaluation value is equal to or less than the threshold value when the first evaluation value is larger than the threshold value.

The evaluation data generation unit may generate the first evaluation data based on the degree of similarity between the first map and the third map.

The map acquisition unit may acquire the third map from the server.

An imaging unit for capturing the image may be further provided.

A sensor unit that measures at least one of the direction in which the image pickup unit is facing or the inclination angle of the image pickup unit may be further provided.

A display unit that superimposes and displays a virtual object generated by the object generation unit on the background may be further provided.

The display unit includes a transmissive display and may be a wearable computer.

The information processing system according to one aspect of the present disclosure includes a first information processing device that generates a first map based on a first image, and a second information processing device that generates a second map based on a second image. The first information processing device generates a virtual object based on the first map or the second map acquired from the second information processing device, and the second information processing device is the second map or the second map. The virtual object may be generated based on the first map acquired from the first information processing apparatus.

The first information processing apparatus further includes a server that manages services using the third map, generates first evaluation data based on the first map and the third map, and uses the first map or the third map. The second map is selected based on the first evaluation data and used to generate a virtual object, and the second information processing apparatus generates the second evaluation data based on the second map and the third map. , The second map or the first map may be selected based on the first evaluation data and used to generate the virtual object.

The first information processing apparatus estimates the first self-position, acquires the first evaluation value corresponding to the first self-position from the first evaluation data, and based on the first evaluation value, of the virtual object. The first map or the second map to be used for generation is selected, the second information processing apparatus estimates the second self-position, and the third evaluation value corresponding to the second self-position is the second evaluation data. The first map or the second map which is acquired from the above and used for generating the virtual object may be selected based on the third evaluation value.

The first information processing apparatus acquires the second evaluation data from the second information processing apparatus, acquires the second evaluation value corresponding to the first self-position from the second evaluation data, and obtains the second evaluation value. When the value is larger than the first evaluation value, the second map is selected, and the second information processing apparatus acquires the first evaluation data from the first information processing apparatus and obtains the first evaluation data from the first evaluation data. The first map may be selected when the fourth evaluation value corresponding to the second self-position is acquired and the fourth evaluation value is larger than the third evaluation value.

When the first evaluation value is larger than the threshold value, the first information processing apparatus generates the virtual object having a size larger than that when the first evaluation value is equal to or less than the threshold value, and the second information processing device generates the virtual object. The information processing apparatus may generate the virtual object having a size larger when the third evaluation value is larger than the threshold value as compared with the case where the third evaluation value is equal to or less than the threshold value.

The information processing method according to one aspect of the present disclosure includes a step of generating a first map based on an image, a step of acquiring a second map from another device, and a virtual method using the first map or the second map. You may want the computer to perform steps to create the object.

The step of generating the first evaluation data based on the first map and the third map used for managing the service, and the first map or the second map used for generating the virtual object are used as the first evaluation data. It may further include a step of selection based on.

The figure which showed the example of the information processing system by this disclosure. The block diagram which showed the example of the information processing apparatus by this disclosure. A block diagram showing an example of data stored in a storage unit. The figure which showed the example of the relationship between the coordinate systems. The figure which showed the example of the global map. The figure which showed the example of the global map data. The figure which showed the example of the local map. The figure which showed the example of the local map data. The figure which showed the example of the deviation of the appearance of a virtual object by a user. A flowchart showing an example of evaluation data generation processing. The figure which showed the example of the similarity between a global map and a local map. The figure which showed the example of the data which specifies the display area of a virtual object. The figure which showed the example of the priority for each area. The figure which showed the example of the evaluation data. A flowchart showing an example of the map selection process. The figure which shows the example of the appearance of the virtual object in the information processing system by this disclosure. The flowchart which shows the example of the display processing of the virtual object in the modification 1. The figure which shows the example of the appearance of the virtual object in the modification 1. The figure which showed the wearing example of the head mounted display. The figure which showed the example of the structure of the display part. The figure which showed the example of the processing in the head-mounted display.

Hereinafter, preferred embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. In the present specification and the drawings, components having substantially the same functional configuration are designated by the same reference numerals, so that duplicate description will be omitted.

FIG. 1 shows an example of an information processing system according to the present disclosure. The information processing system 100 of FIG. 1 provides a user with a service using augmented reality (AR). Services provided using AR include, for example, games, attractions, training systems, work support systems or sales promotion systems.

An example of a game using AR is a multiplayer action game. However, the type of game does not matter. Examples of attractions using AR include watching sports events or watching live events. An example of a training system using AR is a system for improving skills in medical, manufacturing or mobile driving. An example of a work support system using AR is a system used at a construction site or a machine maintenance site. In addition, as an example of a sales promotion system using AR, a system that allows the customer to experience the scenery after the interior is installed, a system that allows the customer to experience the scenery after the remodeling of the building, or a system that allows the customer to experience the appearance after wearing clothes is presented. There is a system. However, these are just examples of services using AR. Therefore, the information processing system 100 may provide services different from these.

The information processing system 100 of FIG. 1 includes a wearable computer 1A, a wearable computer 1B, a management server 30, and a base station 41. The management server 30 is connected to the base station 41 via the network 40. The network 40 is, for example, an information communication network capable of communication by TCP / IP. The base station 41 is, for example, a wireless LAN access point or a base station of a mobile communication system. Both the wearable computers 1A and 1B have a wireless communication function. Therefore, the wearable computers 1A and 1B can perform data communication with the management server 30 via the network 40 and the base station 41. Further, it is assumed that the wearable computer 1A and the wearable computer 1B can perform data communication with each other directly or via the base station 41.

Wearable computers 1A and 1B are information processing devices equipped with a function (SLAM: Simultaneous Localization and Mapping) that performs self-position estimation and map creation in parallel, and allow users to experience services using AR. A wearable computer is a device that is worn and used on any part of the user's body, such as a head-mounted display (HMD), smart glasses, AR helmet, or smart watch. In the following, a case where the information processing device according to the present disclosure is a wearable computer will be described as an example.

However, the information processing device according to the present disclosure may be a device other than a wearable computer. For example, the information processing device according to the present disclosure may be a device portable to a user such as a game device, a smartphone, a tablet, or a laptop computer, and the information processing device according to the present disclosure may be an automobile, an aircraft, a ship, or a railway. It may be a device mounted in the cockpit (driver's seat) or console of a moving body such as a vehicle.

The wearable computer itself generates data corresponding to a map (local map). The wearable computer 1A of FIG. 1 generates local map data 21A. Further, the wearable computer 1B generates the local map data 21B. The local map data is, for example, point cloud data (point cloud) that expresses the shape and position of an object in space by a plurality of feature points. The wearable computer according to the present disclosure can generate point cloud data by, for example, SFM (Structure from Motion). However, the method of generating the point cloud data does not matter. For example, the wearable computer according to the present disclosure may generate point cloud data by LIDAR (Laser Imaging Detection And Ringing). In this case, the wearable computer includes a light source and a distance measuring sensor.

The information processing system 100 of FIG. 1 includes two wearable computers (information processing devices). However, the information processing system according to the present disclosure may include more information processing devices. Details of the information processing apparatus according to the present disclosure will be described later.

The management server 30 is a computer that manages services using AR. The management server 30 includes a communication circuit 32, a processor 33, and a storage 35. The communication circuit 32, the processor 33, and the storage 35 are connected to each other via the bus 31. The communication circuit 32 is, for example, a NIC (network interface card). The processor 33 is a hardware circuit such as a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), an ASIC (Application Specific Integrated Circuit), or an FPGA (Field-Programmable Gate Array). The storage 35 provides a storage area in which data can be stored. As the storage 35, a volatile memory (for example, DRAM or SRAM), a non-volatile memory (for example, NAND flash memory, NOR flash memory, resistance change type memory, magnetoresistive memory), a hard disk, SSD (Solid State Drive) or these Combinations can be used.

As shown in FIG. 1, the global map data 36 is stored in the storage 35. As the global map data 36, for example, point cloud data (point cloud) expressing the shape and position of an object in space by a plurality of feature points can be used. The global map data 36 corresponds to a map of the space in which the service is provided. The service program 34 executed on the processor 33 can specify the position where the virtual object is generated and the position where the event is generated in the service based on the coordinates defined in the global map data 36. Further, the service program 34 may manage the position of the user who is using the service based on the coordinates defined in the global map data 36.

A virtual object is content that is displayed superimposed on the background of the real world, and is also called an AR object. Examples of virtual objects include characters, text, icons, and items. However, it does not limit the types of virtual objects. The virtual object may be an animation with motion or it may be stationary.

As the global map data 36, the point cloud data generated in advance can be used. However, as the global map data 36, point cloud data updated with the passage of time may be used. The global map data 36 may be data generated by the same method as the local map data described later, or may be data generated by a method different from the local map data. For example, the global map data 36 of the space may be generated by a technique such as SFM or LIDAR. Further, the global map data 36 may be generated by using an automobile equipped with a laser scanner or an unmanned aerial vehicle (UAV: Unmanned Aerial Vehicle) equipped with a laser scanner.

Figure 2 shows an example of a global map. The global map of FIG. 2 shows the shape and position of objects along the streets of the city. In the example of FIG. 2, a global map including a wide area where the service is expected to be used is prepared in advance. Therefore, if you refer to the global map, you can see that the buildings are continuous along the road to a long distance. FIG. 3 shows an example of global map data including the global map of FIG. As shown in FIG. 3, as the global map data 36, data expressing the shape and position of an object in space in a three-dimensional Cartesian coordinate system can be used. However, the data format of the global map data 36 is not limited.

Next, the details of the information processing device according to the present disclosure will be described.

The block diagram of FIG. 4 shows an example of the information processing apparatus according to the present disclosure. The information processing device of FIG. 4 corresponds to the wearable computer 1A or 1B of FIG. The information processing device 1 includes an image pickup unit 2, a sensor unit 3, a processing circuit 4, a storage unit 5, a communication circuit 6, a display unit 7, and an audio circuit 8. As the processing circuit 4, for example, a hardware circuit including a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or a combination thereof can be used. it can. The processing circuit 4 includes a local map generation unit 10, a self-position estimation unit 11, a map acquisition unit 12, a map selection unit 13, an evaluation data generation unit 14, and an object generation unit 15 as internal components. Includes.

The block diagram of FIG. 5 shows an example of data stored in the storage unit. In the storage unit 5 of FIG. 5, the image data 20, the first local map data 21, the second local map data 22, the global map data 23, the first evaluation data 24, the second evaluation data 25, and the like. The map processing program 26 and the application program 27 are stored. As the storage unit 5, a volatile memory (for example, DRAM or SRAM), a non-volatile memory (for example, NAND flash memory, NOR flash memory, resistance change type memory, magnetoresistive memory), a hard disk, SSD (Solid State Drive) or these You can use the combination of. However, the type of memory / storage used as the storage unit 5 does not matter.

In the following, each component will be described with reference to FIGS. 4 and 5.

The image pickup unit 2 includes, for example, an image pickup element such as a CMOS image sensor or a CCD image sensor. For example, an image sensor that detects visible light and outputs a color image can be used. However, the image pickup unit 2 may include an image pickup element that detects electromagnetic waves in other wavelength bands such as infrared rays. The image taken by the imaging unit 2 is stored in the storage unit 5 as image data 20. The imaging unit 2 captures, for example, an image in the same direction as the line of sight of the user wearing the information processing device 1. However, the direction in which the imaging unit 2 performs shooting and the direction in which the user's line of sight does not necessarily have to match. Further, the field of view (FOV) of the imaging unit 2 may be different from the field of view of the user. The imaging unit 2 may be integrally mounted with other components in the information processing device 1, or may be an external device.

The sensor unit 3 includes, for example, at least one of a sensor capable of estimating the position of the information processing device 1 and a sensor capable of detecting the posture of the information processing device 1. The sensor unit 3 may include, for example, a sensor such as an acceleration sensor, a gyro sensor, an electronic compass, a GPS receiver, or a tilt sensor. The information processing device 1 may acquire at least one information of its own position or posture via the sensor unit 3. Here, the posture may include at least one of the direction in which the imaging unit 2 is facing or the tilt angle of the information processing device 1.

The sensor unit 3 may perform measurement in conjunction with an image shooting operation by the imaging unit 2. For example, the sensor unit 3 may measure at least one of the position or the posture of the information processing device 1 at the image shooting timing when the image capturing unit 2 takes an image. Then, the sensor unit 3 may store information including at least one of the positions and postures of the information processing device 1 in the storage unit 5 in association with the image captured by the imaging unit 2. That is, the image data 20 of the storage unit 5 may include at least information on the position or posture of the information processing device 1 when each image is taken. In addition, the sensor unit 3 may include a sensor that measures other types of physical information, such as a temperature sensor or an illuminance sensor.

The communication circuit 6 is a circuit for wireless data communication. The communication circuit 6 may include an antenna. Each component in the processing circuit 4 can perform data communication with an external device via the communication circuit 6. As the communication circuit 6, for example, a circuit corresponding to a wireless LAN (IEEE802.11 series or a higher standard thereof) or various mobile communication standards can be used. However, the type of communication standard used in the communication circuit 6 does not matter.

The display unit 7 is a device that displays contents such as graphics and characters. The display unit 7 may include a transmissive display that displays the content superimposed on the background. Further, the display unit 7 may include a display capable of displaying the image captured by the imaging unit 2 and superimposing the content on the image. The display unit 7 may include, for example, a liquid crystal display, an OLED (Organic Light Emitting Diode) display, or a MicroLED display. However, the type of display used in the display unit 7 is not particularly limited. As will be described later, the display unit 7 may include a dimming element. For example, the display unit 7 visually presents the content specified by the object generation unit 15 to the user. However, the content displayed on the display unit 7 may be the content specified by the service program 34 of the management server 30. The display unit 7 may be integrally mounted with other components in the information processing device 1, or may be an external device.

The audio circuit 8 is a circuit that reproduces the voice used in the service. For example, when the audio used in the service is digital format data, the audio circuit 8 converts the digital format data into an analog format audio signal. Then, the audio circuit 8 outputs an analog-format audio signal to an audio device such as an earphone, a headphone, or a speaker. The sound reproduced by the audio circuit 8 may be based on the data specified by the object generation unit 15. Further, the audio circuit 8 may reproduce audio based on the data specified by the service program 34 of the management server 30. The audio circuit 8 may have a voice recording (microphone) function.

The local map generation unit 10 causes the imaging unit 2 to capture an image at a plurality of timings. For example, the imaging unit 2 periodically captures an image. However, the timing of taking an image is not limited. For example, the imaging unit 2 may capture an image at a timing specified by the user. As a result, it is possible to obtain image data 20 including a plurality of images taken at different positions and angles according to the movement of the user wearing the information processing device 1. As described above, each image in the image data 20 may be associated with information including at least one of the positions or orientations of the information processing apparatus 1.

Then, the local map generation unit 10 can extract feature points from the image in the image data 20 and generate point cloud data (point cloud). The local map generation unit 10 may generate point cloud data using information including at least one of the positions and orientations of the information processing device 1. Here, the point cloud data is only an example of a data format for expressing the shape and position of an object in space. Therefore, the local map generation unit 10 may generate data in a format different from the point cloud data in order to express the shape and position of the object in the space. The local map generation unit 10 stores the first local map data 21 in the storage unit 5. The first local map data 21 may include coordinate information in space. The local map generation unit 10 may divide the space in which the service is provided into a plurality of spatial meshes and generate point cloud data for each spatial mesh.

The first local map data 21 generated by the local map generation unit 10 may include data obtained by converting the point cloud data. For example, the local map generation unit 10 may convert point cloud data, generate data such as polygons or NURBS (non-uniform rational B-spline), and store it as the first local map data 21. Further, the local map generation unit 10 may generate the first local map data 21 including the color information (for example, RGB) by performing the surface formation processing of the point cloud data and projecting the image on the surface. ..

FIG. 6 shows an example of a local map. The local map of FIG. 6 shows the shape and position of an object along the road in the same city as that of FIG. The local map generation unit 10 generates a local map from an area where a sufficient number of images can be captured. Therefore, as in the area 28 of FIG. 6, information on the shape and position of the object cannot be sufficiently obtained, and there is a possibility that an area for which a local map has not been created remains in the space. On the other hand, the information on the shape and position of the object in the vicinity of the user in the local map is updated in real time. Therefore, it is possible to acquire detailed information about the area in the space that the user is observing for a long time.

With reference to FIG. 6, information on the shape of the building on the right side of the road is obtained with higher accuracy than the global map of FIG. In addition, it can be seen that real-time information such as passersby on the right side of the road and pebbles on the road is obtained. In this way, local maps can be used to reflect information about the current environment, such as weather, sunlight orientation, and plant growth, in the process. Especially for location-based services or applications, using local maps can improve the display accuracy of virtual objects. On the other hand, in the local map, there may be a difference in the information on the shape and position of the object obtained depending on the height or body shape of the user who wears the information processing device 1.

FIG. 7 shows an example of local map data. As the local map data, as shown in FIG. 7, data expressing the shape and position of an object in space in a three-dimensional Cartesian coordinate system can be used. However, it does not limit the type of coordinate system used in the local map data.

As will be described later, the information processing device according to the present disclosure can use not only the local map generated by the own device but also the local map generated by another information processing device. The local map generated by the own device is called the first local map, and the local map generated by another information processing device is called the second local map. For example, the information processing device 1 can use the first local map when estimating its own position. Further, the information processing device 1 can arbitrarily use the second local map in some processes such as generation of a virtual object. However, the map used in each process is not limited. The first local map data 21 in FIG. 5 is data including the first local map. On the other hand, the second local map data 22 in FIG. 5 is data including the second local map generated by another information processing apparatus.

The self-position estimation unit 11 estimates the position of the information processing device 1 in space. The self-position estimation unit 11 can estimate the self-position based on the image captured by the imaging unit 2, for example. The self-position estimation unit 11 may estimate the self-position based on the measured value of the sensor in the sensor unit 3. When the self-position estimation unit 11 estimates the position in space, the coordinates in the first local map indicating the current position of the information processing device 1 are updated. The self-position estimation unit 11 may calculate the coordinates in at least one of the second local map and the global map when estimating the self-position. As the coordinates, for example, offset values (x _l , y _l , z _l ) from the origin in the Cartesian coordinate system can be used. However, the method of expressing the coordinates is not limited. Other types of coordinate systems, such as spherical or polar coordinates, may be used.

The coordinate system used in the first local map may be of the same type as the coordinate system used in the management server 30 (global map), or may be of a different type. Further, the origin of the coordinate system used in the first local map may be the same as the origin of the coordinate system used in the management server 30, or may be set at a different position. The coordinates (system) used in the first local map are called local coordinates (system), and are distinguished from the global coordinates (system) used in the management server 30. Further, the coordinates (system) used in the second local map are referred to as the second local coordinates (system).

The self-position estimation unit 11 may estimate the self-position at a predetermined cycle. Further, the self-position estimation unit 11 may estimate the self-position when a predetermined condition is satisfied. For example, the self-position estimation unit 11 may calculate the coordinates in space on the condition that the image pickup unit 2 detects an image including a specific object (called a key object). This image is called a keyframe. The key frame may be selected by the user or may be selected by the self-position estimation unit 11 based on the feature points. The self-position estimation unit 11 or the user may add a label to the image corresponding to the key frame in the image data 20.

Further, when the global coordinate value of the key object is notified from the management server 30, the self-position estimation unit 11 may correct the deviation of the first local coordinate based on the global coordinate value. For example, the self-position estimation unit 11 converts the first local coordinate in which the key object is detected into the global coordinate, and when the value of the global coordinate is different from the value notified from the management server 30, the management server 30 informs the user. The value obtained by converting the notified global coordinates to the first local coordinates may be set as the self-position. As a result, the consistency between the first local coordinate system and the global coordinate system can be ensured.

Here, the method of coordinate conversion between the first local coordinate and the global coordinate will be described. For example, it is possible to compare the global map data 36 of the management server 30 with the feature points in the data of the first local map data 21 of the information processing device 1 and determine whether or not they are at the same position in the space. Is. In the following, a case where conversion to the other coordinate system is performed when the first local coordinates or global coordinates corresponding to the same position in the space are obtained will be described.

First, the notation used in the explanation will be explained. The position vector of the position v represented by the coordinate system u is expressed as _u P _v . Further, the rotation matrix for converting the vector represented by the coordinate system v into the vector represented by the coordinate system u is represented by _u R _v . When a three-dimensional Cartesian coordinate system is used, _u P _v is a vector with three elements, and _u R _v is a 3 × 3 rotation matrix. _{Further, the uT v} of the following equation (1) is a matrix for converting the position vector represented by the coordinate system v into the coordinate system u. _u T _v is a 4 × 4 matrix representing _u P _v and _u R _{v in a simultaneous coordinate system.}

Here, 0 ₃ is assumed to be a matrix of [0, 0, 0].

FIG. 8 shows an example in which the position c is expressed using each coordinate system when the global coordinate system is a and the first local coordinate system is b. In FIG. 8, _a P _c is a vector indicating the position of c in the coordinate system a. Further, _b P _c is a vector indicating the position of c in the coordinate system b. Further, _a R _b is a rotation matrix for converting a vector represented by the coordinate system b into a vector represented by the coordinate system a. _a P _b is a vector indicating the position of b in the coordinate system a. Further, _a T _b is a representation of the rotation matrix _a R _b and the vector _a P _b in 4 × 4 of the homogeneous coordinate system.

When the above notation is used, the relationship of the following equation (2) is established between the coordinate system a and the coordinate system b.

Further, by using the following equation (3), the position and orientation of the information processing apparatus 1 represented by the coordinate system b can be converted into the coordinate system a.

When the above equations (2) and (3) are expressed in a homogeneous coordinate system by a 4 × 4 matrix, the following equations (4) and (5) are obtained.

When there are a plurality of images of the same object (for example, a key object) in the image data 20, a determinant is obtained using each of the plurality of images including the same object, and the average value of the plurality of determinants is calculated. You may calculate. By using the matrix equation that defines the relationship between the position vector _a P _c and the position vector _b P _c described above, between the global coordinates (coordinates of the coordinate system a) and the local coordinates (coordinates of the coordinate system b). Can be converted. The conversion between the first local coordinate and the second local coordinate used in other information processing devices and the conversion between the second local coordinate and the global coordinate can also be performed by the same method as described above. is there.

In the following, each component will be described with reference to FIGS. 4 and 5 again.

The map acquisition unit 12 acquires map data from an external device. For example, when the information processing device 1 is the wearable computer 1A of FIG. 1, the map acquisition unit 12 can acquire the local map data 21B from the wearable computer 1B. The map acquisition unit 12 stores the local map data 21B as the second local map data 22 in the storage unit 5. Further, the map acquisition unit 12 acquires the global map data 36 from the management server 30. The map acquisition unit 12 stores the global map data 36 as the global map data 23 in the storage unit 5. Further, the map acquisition unit 12 stores the evaluation data data acquired from another information processing device as the second evaluation data 25 in the storage unit 5.

The object generation unit 15 executes AR-related processing such as generation of a virtual object and processing of a service provided to the user. The application program 27 of the storage unit 5 can be executed on the processing circuit 4 to provide the function of the object generation unit 15 to the user. When the service is implemented as a client-server system, the object generation unit 15 executes the client-side processing of the service. In this case, the service program 34 of the management server 30 executes the processing on the server side of the service.

For example, the service program 34 of the management server 30 specifies a position where a virtual object is generated and a position where an event is generated based on the global coordinate system. The object generation unit 15 can convert the global coordinates specified by the service program 34 into at least one of the first local coordinates and the second local coordinates, and execute the processing on the client side.

For example, when the service program 34 of the management server 30 transmits a command requesting the generation of a virtual object at a predetermined global coordinate, the object generation unit 15 generates a virtual object at the first local coordinate obtained by converting the global coordinate. Then, the user can visually recognize the virtual object through the display unit 7.

Note that the object generation unit 15 may generate a virtual object at a position shifted by a predetermined distance from the first local coordinate obtained by converting the global coordinates. Further, when the service program 34 of the management server 30 transmits a command to display a virtual object within the global coordinates (for example, the spatial mesh) of a predetermined range, the object generation unit 15 causes the first local coordinates corresponding to the spatial mesh. Calculate the range of. Then, the object generation unit 15 may adjust the display position of the virtual object within the range of the spatial mesh. Further, the object generation unit 15 may determine the display position of the virtual object by using the second local coordinates.

The map selection unit 13 selects the map used by the object generation unit 15 to display the virtual object based on the evaluation data described later. When the map is selected by the map selection unit 13, the coordinate system used for displaying the virtual object is also determined.

FIG. 9 shows an example of the deviation of the appearance of the virtual object by the user. FIG. 9 shows a user U1 wearing a wearable computer 1A, an object OBJ, and a user U2 wearing a wearable computer 1B. In FIG. 9, two users are shown. However, the number of users who use the service may be larger than this. The object OBJ in FIG. 9 is a jar having an opening at the top. Although the user U1 and the user U2 stand at different positions, they both face the direction of the object OBJ. View V1 shows an example of how the user U1 wearing the wearable computer 1A sees it. On the other hand, the view V2 shows an example of how the user U2 who wears the wearable computer 1B sees it.

View V1 and view V2 of FIG. 9 may be realized by a combination of an image or video displayed on the display unit 7 and a background transmitted through the display unit 7. Further, the view V1 and the view V2 may be realized only by the image or the video displayed on the display unit 7.

In FIG. 9, the service program 34 of the management server 30 transmits a command to generate the virtual object A1 at predetermined global coordinates to the wearable computers 1A and 1B. The virtual object A1 is a character that appears in the service. The virtual object A1 may be a still image or an animation. Within the service, it is assumed that there is a demand for a visual effect in which the character protrudes from the opening of the jar. In this way, the position where the virtual object is generated and the movement of the virtual object are controlled not only by the absolute coordinates in the map but also by the relative positional relationship with the object in space. Good.

Here, each of the wearable computers 1A and 1B converts the global coordinates into the local coordinates and displays the virtual object A1 which is a character on the display unit 7. Referring to FIG. 9, in the view V1 of the wearable computer 1A, a visual effect in which the character protrudes from the opening of the jar is realized. On the other hand, in the view V2 of the wearable computer 1B, since the virtual object A1 which is a character is displayed at a position shifted from above the object OBJ, the user U2 cannot recognize that the character is protruding from the opening of the jar. ..

It is expected that the body shapes and heights of users who use the service will be diverse, and since each user will take different behavior patterns, there will be differences in the area in the space where the user's line of sight is directed. Therefore, a different local map is generated for each wearable computer. If different local maps are used depending on the wearable computer, the appearance of the virtual object may differ depending on the user as shown in the example of FIG. 9, and the quality of service may deteriorate. For example, when the virtual object is an enemy character, the appearance of the enemy character differs depending on the user. In this case, it becomes difficult for a plurality of users who play the action game to cooperate and fight against the enemy character, and the user's satisfaction level is lowered. Therefore, it is desirable to suppress the deviation of the appearance of the virtual object among a plurality of users according to the service situation and situation.

The evaluation data generation unit 14 generates evaluation data. The flowchart of FIG. 10 shows an example of evaluation data generation processing. Hereinafter, the evaluation data generation process will be described with reference to the flowchart of FIG.

First, the local map generation unit 10 updates the first local map (first local map data 21) (step S101). Then, the map acquisition unit 12 acquires the global map data 23 from the management server 30 (step S102). For example, when the management server 30 updates the global map data, the information processing apparatus 1 can obtain the updated global map data 23. However, if the global map data 23 pre-installed in the information processing device 1 is used, or if the management server 30 does not update the global map data, the process of step S102 may be skipped.

Next, the self-position estimation unit 11 specifies the global coordinates in the global map corresponding to the first local coordinates in the first local map (step S103). For example, the self-position estimation unit 11 can select the first local coordinates in the area where the evaluation data is generated, and calculate the global coordinates corresponding to the first local coordinates by the coordinate transformation.

Then, the evaluation data generation unit 14 divides the point cloud data in the vicinity of the first local coordinates in the first local map into the point cloud data in the vicinity of the global coordinates corresponding to the first local coordinates in the global map. The similarity is calculated based on this (step S104). In step S104, for example, the similarity value can be calculated for each mesh that divides the space based on the first local coordinate system. Data D1 in FIG. 11 shows an example of the degree of similarity between the global map and the local map calculated in step S104. In the example of FIG. 11, the similarity value is shown for each two-dimensional region. However, the actually calculated similarity may be associated with the coordinates in the three-dimensional space or the mesh in the three-dimensional space.

For example, using point cloud data in a sphere with a radius r centered on the first local coordinate and point cloud data in a sphere with a radius r centered on the global coordinates corresponding to the first local coordinate. The similarity can be calculated. However, the shape and size of the area in the space for which the similarity is calculated are not limited. The similarity can be calculated using techniques such as template matching, feature point matching or histogram comparison. In addition, the similarity may be calculated by a machine learning method such as a distance between features or clustering of features. Further, the similarity may be calculated by another method.

Next, the evaluation data generation unit 14 acquires the attribute information in the vicinity of the first local coordinates from the program that uses the map (step S105). For example, the evaluation data generation unit 14 may acquire attribute information in the vicinity of the first local coordinates from the object generation unit 15 in step S105. Similar to the similarity data calculated in step S104, data in which a value is set for each mesh in which the space is divided based on the first local coordinate system can be used as the attribute information.

For example, as the attribute information, data indicating whether or not it is the display target area of the virtual object can be used as shown in FIG. In the data D2 of FIG. 12, the value of the mesh that is the display target of the virtual object is set to 1.0, and the value of the mesh that is not the display target of the virtual object is set to 0.0. Further, as the attribute information, data for designating a map to be preferentially used as shown in FIG. 13 may be used. In the data D3 of FIG. 13, the positive value indicates the region in which the reduction of the deviation in appearance among a plurality of users is prioritized, and the negative value indicates the region in which the consistency with the global map is prioritized. In the data D3, for example, the value of the mesh on which the boss character is displayed or the mesh in which cooperation between a plurality of users is required can be set large. By using the attribute information exemplified in the data D2 and the data D3, it is possible to reflect the presence / absence of the virtual object display in the service, the purpose of the virtual object, the importance of the display accuracy of the virtual object, and the like.

Here, the case where the evaluation data generation unit 14 acquires the attribute information from the object generation unit 15 has been described as an example. However, the evaluation data generation unit 14 may acquire attribute information from other components. For example, the evaluation data generation unit 14 may acquire attribute information from the service program 34 of the management server 30. However, in this case, it is necessary to convert the attribute information expressed in global coordinates to the first local coordinates.

Then, the evaluation data generation unit 14 calculates the evaluation data based on the similarity and the attribute information (step S106). For example, it is possible to calculate the evaluation data using the data D1 to D3 shown in FIGS. 11 to 13. For example, for the corresponding meshes of the data D1 to D3, _Dr 1 × _Dr 2 + _Dr 3 can be calculated and the evaluation data can be calculated. Here, _Dr 1 to _Dr 3 indicate the respective meshes in the data D1 to D3. The data D4 in FIG. 14 is an example of the evaluation data calculated in step S106. As the evaluation data, as in the data D4 of FIG. 14, data including a value normalized by 1 may be used. In the data D4, the evaluation value for each mesh D _{r 4} is set.

When at least one of the first local map data 21 and the global map data 23 is updated, the evaluation data generation unit 14 may update the above-mentioned data D1. Further, as the data D2 and the data D3, the data set by the object generation unit 15 or the service program 34 can be used. However, the evaluation data generation unit 14 may correct the data D2 or the data D3 according to the movement of the user, the direction in which the user is facing, or the time.

Further, when it is desired to avoid an increase in load and a delay in processing due to switching of the map used in the object generation unit 15, the evaluation data generation unit 14 may correct the value of the data D3 to be small. For example, when the map used by the object generation unit 15 is switched, the display position or shape of the virtual object may suddenly change. The process of correcting the value of the data D3 to be small may be executed by the object generation unit 15 or the service program 34, or may be executed according to the operation of the user.

The evaluation data generation unit 14 stores the calculated evaluation data as the first evaluation data 24 in the storage unit 5. The first evaluation data 24 is, for example, data including evaluation values set for each mesh that divides the space based on the first local coordinate system. However, the first evaluation data 24 may be data in a format different from this. For example, the first evaluation data 24 may be data including evaluation values set for each coordinate. Further, the coordinate system used in the first evaluation data 24 may be a coordinate system different from the first local coordinate system.

This is the end of the description of the processing of the flowchart of FIG. For example, by executing the map processing program 26 of the storage unit 5 on the processing circuit 4, the above-mentioned local map generation unit 10, self-position estimation unit 11, map acquisition unit 12, map selection unit 13, or evaluation data generation unit 14 Functions can be realized. However, at least one of the processes of the local map generation unit 10, the self-position estimation unit 11, the map acquisition unit 12, the map selection unit 13, and the evaluation data generation unit 14 may be implemented by the hardware circuit.

The information processing device according to the present disclosure can perform map selection processing based on the evaluation data. The map selection process will be described below.

The flowchart of FIG. 15 shows an example of the map selection process. Hereinafter, the process will be described with reference to the flowchart of FIG. In the explanation of the flowchart, when it is stated that data is to be acquired, it shall include not only the case of acquiring the entire relevant data but also the acquisition of a part of the relevant data.

First, the self-position estimation unit 11 estimates the self-position of the information processing device 1 (step S201). For example, the self-position estimation unit 11 can obtain the first local coordinates corresponding to the self-position of the information processing device 1 in step S201. Next, the map selection unit 13 obtains the first evaluation value of the self-position from the first evaluation data (first evaluation data 24) (step S202). For example, the map selection unit 13 searches for a mesh in the first evaluation data 24 including the first local coordinates corresponding to its own position. As a result, the map selection unit 13 can obtain the evaluation value set in the mesh. Here, the evaluation value obtained based on the first local coordinates is referred to as the first evaluation value.

Then, the map selection unit 13 determines whether or not the first evaluation value of the self-position in the first evaluation data is larger than the threshold value (step S203). For example, when the data D4 (FIG. 14) is used as the first evaluation data, a value larger than 0 and smaller than 1 can be used as the threshold value. However, a different threshold may be used.

The process to be executed branches according to the determination result in step S203. When the first evaluation value of the self-position in the evaluation data is equal to or less than the threshold value (NO in step S203), the object generation unit 15 uses the first local map (first local map data 21) when the virtual object is generated. Use (step S208). If the first evaluation value is less than or equal to the threshold value, it is not highly necessary to create a virtual object using the same local map as other users, or even if the first local map is used, a virtual object is created between users. It is assumed that the deviation of the appearance of the object will not increase. Therefore, when the determination in step S203 is negative, the object generation unit 15 executes the virtual object generation process using the first local map (step S208).

When the first evaluation value of the self-position in the evaluation data is larger than the threshold value (YES in step S203), the map acquisition unit 12 has the second local map (second local map data 22) and the second local map data 22 from other information processing devices. The second evaluation data (second evaluation data 25) is acquired (step S204). If the first evaluation value is larger than the threshold value, it is assumed that there is a large difference in the appearance of the virtual object between users, or it is highly necessary to generate the virtual object using the same local map as other users. The object.

The information processing device from which the second local map data 22 and the second evaluation data 25 are acquired is not particularly limited. The map acquisition unit 12 can acquire the second local map data 22 and the second evaluation data 25 from at least one of the information processing devices in the information processing system 100 in step S204. Further, the map acquisition unit 12 may acquire the second local map data 22 and the second evaluation data 25 from the plurality of information processing devices in step S204.

For example, the map acquisition unit 12 may acquire the second local map data 22 and the second evaluation data 25 from the information processing device designated by the service program 34 of the management server 30. For example, the service program 34 may specify an information processing device that is the acquisition source of the second local map data 22 and the second evaluation data 25 based on the user's identifier. This makes it possible to select an information processing device that uses a common local map according to the situation in the service.

Further, the information processing device 1 may select another information processing device that is the acquisition source of the second local map data 22 and the second evaluation data 25. For example, the map acquisition unit 12 may acquire the second local map data 22 and the second evaluation data 25 from the information processing device having the highest intensity of the beacon signal received via the communication circuit 6. Further, the map acquisition unit 12 may acquire the second local map data 22 and the second evaluation data 25 from one or more information processing devices selected based on the strength of the beacon signal. Further, the map acquisition unit 12 may acquire the second local map data 22 and the second evaluation data 25 from the user or the information processing device designated by the object generation unit 15. This makes it possible to select the information processing device that is the acquisition source of the local map based on the position of each user and the processing in the object generation unit 15.

When the map acquisition unit 12 acquires the second local map data 22 and the second evaluation data 25 from another information processing device, the map selection unit 13 uses the second local map (second local map data 22) and the second evaluation data. (Second evaluation data 25) is used to obtain the second evaluation value of the self-position (step S205). Here, the evaluation value obtained based on the second local coordinates is referred to as the second evaluation value. For example, the map selection unit 13 obtains the second local coordinates corresponding to the self-position based on the first local coordinates corresponding to the self-position.

For example, the map selection unit 13 performs the above-mentioned coordinate transformation to calculate the second local coordinates corresponding to its own position. Further, the map selection unit 13 may calculate the second local coordinates corresponding to its own position based on the shape and position of the object included in the first local map data 21 and the second local map data 22. For example, pattern matching may be performed based on the feature amount, and the second local coordinates corresponding to the self-position in the second local map data 22 may be searched. However, the method in which the map selection unit 13 calculates the second local coordinates corresponding to its own position does not matter.

The map selection unit 13 searches for the second local coordinates corresponding to the self-position in the second evaluation data 25. As a result, the map selection unit 13 can obtain the second evaluation value of the self-position. Then, the map selection unit 13 determines whether or not the second evaluation value of the self-position is larger than the first evaluation value of the self-position (step S206).

The process to be executed branches according to the determination result in step S206. When the second evaluation value is equal to or less than the first evaluation value (NO in step S206), the object generation unit 15 executes a virtual object generation process using the first local map (first local map data 21) (NO in step S206). Step S208). That is, if the determination in step S206 is negative, the object generation unit 15 may execute the virtual object generation process using the first local coordinate system.

When the second evaluation value is larger than the first evaluation value (YES in step S206), the object generation unit 15 executes a virtual object generation process using the second local map (second local map data 22) (step). S207). That is, if the determination in step S206 is affirmative, the object generation unit 15 may execute the virtual object generation process using the second local coordinate system.

When the map acquisition unit 12 acquires the second local map data 22 and the second evaluation data 25 from the plurality of information processing devices in step S204, the map selection unit 13 acquires the second local from the plurality of information processing devices. The processes of steps S205 and S206 can be executed for the map data 22 and the second evaluation data 25. In this case, the object generation unit 15 generates a virtual object using the second local map data 22 of the information processing apparatus having the largest value obtained by subtracting the first evaluation value from the second evaluation value in step S207. You may. Further, when an information processing device for which the determination in step S206 is affirmative is detected, a virtual object may be generated using the second local map data 22 acquired from the information processing device.

It is possible to execute the processing of the flowchart of FIG. 15 in the information processing device used by each of a plurality of users using the same service. For example, as shown in FIG. 9 above, it is assumed that both the user U1 and the user U2 are facing the direction of the object OBJ. In this case, the wearable computer 1A worn by the user U1 and the wearable computer 1B worn by the user U2 can generate a virtual object using the same local map.

For example, if the first evaluation value is less than or equal to the threshold value in the wearable computer 1A and the first evaluation value is larger than the threshold value in the wearable computer 1B, the same local map is used for both when creating the virtual object. there is a possibility. As a result, it is possible to reduce the deviation of the appearance of the virtual object between users. Further, depending on the conditions, the wearable computer 1A and the wearable computer 1B may generate virtual objects using different local maps.

FIG. 16 shows an example of how a virtual object looks in the information processing system according to the present disclosure. In FIG. 16, the user U1 wearing the wearable computer 1A and the user U2 wearing the wearable computer 1B are facing the object OBJ at the same positions and postures as in FIG. Similar to FIG. 9, the object OBJ is a jar having an opening at the top. View V1 shows an example of how it looks from user U1. Further, the view V2 shows an example of how the user U2 sees the view. In FIG. 16, it is assumed that the service program 34 of the management server 30 transmits a command to generate the virtual object A1 at predetermined global coordinates to the wearable computers 1A and 1B as in FIG.

In the example of FIG. 16, the wearable computer 1A and the wearable computer 1B execute the processing of the flowchart of FIG. 15 and select the local coordinates (local map) to be used when the virtual object A1 is generated. Therefore, the wearable computer 1A and the wearable computer 1B generate the virtual object A1 by using either the first local map or the second local map.

With reference to FIG. 16, in view V2 of the wearable computer 1B, a character appears at the upper part of the jar, and the problem of display position deviation shown in FIG. 9 is solved. Therefore, not only the user U1 but also the user U2 can visually recognize how the character is popping out from the opening of the jar. It is also possible for three or more users to use the same local map to suppress the deviation of the appearance of virtual objects.

By using the information processing device and the information processing system according to the present disclosure, it is possible to eliminate the difference in the appearance of virtual objects between users and improve the quality of services using AR. Multiple users will be able to share their experiences in augmented reality space. For example, when the virtual object is an enemy character, a plurality of users playing an action game can cooperate to fight the enemy character.

As described above, the information processing method according to the present disclosure includes a step of generating a first map based on an image, a step of acquiring a second map from another device, and a virtual object using the first map or the second map. It may be the one that causes the computer to perform the steps to generate. Further, the information processing method according to the present disclosure includes a step of generating the first evaluation data based on the first map and the third map used for managing the service, and the first map or the second map used for generating the virtual object. It may further include a step of selection based on the first evaluation data. The first map corresponds to, for example, the first local map data 21. The second map corresponds to, for example, the second local map data 22. The third map corresponds to, for example, global map data 23 and 36.

Here, the outline of the information processing device according to this disclosure is organized.

The information processing device according to the present disclosure includes a map generation unit that generates a first map based on an image, a map acquisition unit that acquires a second map from another device, and a virtual object using the first map or the second map. It may be provided with an object generation unit that generates. Here, the first local map data 21 is an example of the first map. The second local map data 22 is an example of the second map. For example, in view of the above-mentioned wearable computer 1A, the wearable computer 1B corresponds to another device.

Further, the information processing apparatus according to the present disclosure is used by an evaluation data generation unit that generates the first evaluation data based on the first map and the third map used for service management, and an object generation unit that generates a virtual object. It may further include a map selection unit that selects the first map or the second map based on the first evaluation data. The global map data 23 and 36 described above are examples of the third map. The evaluation data generation unit may generate the first evaluation data based on the degree of similarity between the first map and the third map. The map acquisition unit may acquire the third map from the server. The management server 30 described above is an example of a server.

Further, the information processing apparatus according to the present disclosure may further include a self-position estimation unit that estimates the self-position. In this case, the map selection unit acquires the first evaluation value corresponding to the self-position from the first evaluation data, and the object generation unit uses the first map or the second map to generate the virtual object based on the first evaluation value. Select.

As illustrated in the flowchart of FIG. 15, the map selection unit may select the first map when the first evaluation value is equal to or less than the threshold value. Further, the map acquisition unit acquires the second evaluation data corresponding to the second map from another device, and the map selection unit acquires the second evaluation value corresponding to the self-position from the second evaluation data, and the second evaluation value is obtained. The second map may be selected when the evaluation value is larger than the first evaluation value. Further, the map selection unit may select the first map when the second evaluation value is equal to or less than the first evaluation value.

The information processing device according to the present disclosure may include an imaging unit that captures an image. Further, the information processing apparatus according to the present disclosure may further include a sensor unit that measures at least one of the direction in which the imaging unit is facing and the tilt angle of the imaging unit. In this case, the map generation unit may generate the first map using the measured values of the sensor unit.

Next, the outline of the information processing system according to this disclosure will be summarized.

Even if the information processing system according to the present disclosure includes a first information processing device that generates a first map based on a first image and a second information processing device that generates a second map based on a second image. Good. The first information processing device generates a virtual object based on the first map or the second map acquired from the second information processing device. The second information processing device generates a virtual object based on the second map or the first map acquired from the first information processing device. For example, the first information processing device corresponds to the wearable computer 1A, and the second information processing device corresponds to the wearable computer 1B.

The first map is, for example, the first local map data 21 generated by the local map generation unit 10 of the first information processing apparatus. The second map is, for example, the second local map data 22 stored in the storage unit 5 of the first information processing apparatus. The second map also corresponds to the first local map data 21 generated by the local map generation unit 10 of the second information processing apparatus.

Further, the information processing system according to the present disclosure may further include a server that manages services using the third map. In this case, the first information processing apparatus generates the first evaluation data based on the first map and the third map, selects the first map or the second map based on the first evaluation data, and generates a virtual object. May be used for. On the other hand, the second information processing apparatus generates the second evaluation data based on the second map and the third map, selects the second map or the first map based on the first evaluation data, and generates a virtual object. You may use it. Global map data 23 and 36 are examples of the third map.

Further, as illustrated in the flowchart of FIG. 15, the first information processing apparatus estimates the first self-position, acquires the first evaluation value corresponding to the first self-position from the first evaluation data, and performs the first evaluation. You may choose the first or second map to use to create the virtual object based on the value. On the other hand, the second information processing device estimates the second self-position, acquires the third evaluation value corresponding to the second self-position from the second evaluation data, and uses it to generate a virtual object based on the third evaluation value. You may choose the first map or the second map. Here, the third evaluation value corresponds to the first evaluation value in steps S202 and S203 of FIG. The first self-position and the second self-position are estimated by, for example, the self-position estimation unit 11 in each information processing device.

As illustrated in the flowchart of FIG. 15, the first information processing apparatus acquires the second evaluation data from the second information processing apparatus, and acquires the second evaluation value corresponding to the first self-position from the second evaluation data. , The second map may be selected when the second evaluation value is larger than the first evaluation value. On the other hand, the second information processing apparatus acquires the first evaluation data from the first information processing apparatus, acquires the fourth evaluation value corresponding to the second self-position from the first evaluation data, and the fourth evaluation value is the third. The first map may be selected when it is larger than the evaluation value. For example, the third evaluation value corresponds to the first evaluation value in steps S202 and S203 of FIG. In this case, the fourth evaluation value corresponds to the second evaluation value in steps S205 and S206 of FIG.

In the above, by selecting the first local map generated by the own device or the second local map generated by another information processing device as the local map used when the virtual object is generated, the appearance of the virtual object among users can be seen. The information processing device and the information processing system that eliminate the deviation have been described. However, in the information processing device and the information processing system according to the present disclosure, the deviation in the appearance of the virtual object between users may be eliminated by a method different from this.

The flowchart of FIG. 17 shows an example of the display processing of the virtual object in the modification 1. Hereinafter, the process will be described with reference to the flowchart of FIG.

First, the self-position estimation unit 11 estimates the self-position of the information processing device 1 (step S211). For example, the self-position estimation unit 11 can obtain the first local coordinates corresponding to the self-position of the information processing device 1 in step S211. Next, the map selection unit 13 obtains the first evaluation value of the self-position from the first evaluation data (first evaluation data 24) (step S212). For example, the map selection unit 13 searches for a mesh in the first evaluation data 24 including the first local coordinates corresponding to its own position. As a result, the map selection unit 13 can obtain the evaluation value set in the mesh.

Then, the map selection unit 13 determines whether or not the first evaluation value of the self-position in the evaluation data is larger than the threshold value (step S213). For example, when the data D4 (FIG. 14) is used as the first evaluation data, a value larger than 0 and smaller than 1 can be used as the threshold value. However, a different threshold may be used.

The process to be executed branches according to the determination result in step S213. When the first evaluation value of the self-position in the evaluation data is equal to or less than the threshold value (NO in step S213), the object generation unit 15 displays the generated virtual object on the display unit 7 in a specified size (step S215). ..

When the first evaluation value of the self-position in the evaluation data is larger than the threshold value (YES in step S213), the object generation unit 15 displays the generated virtual object on the display unit 7 larger than the specified size (step S214). ..

FIG. 18 shows an example of how the virtual object looks in the first modification. In FIG. 18, the user U1 wearing the wearable computer 1A and the user U2 wearing the wearable computer 1B are facing the object OBJ at the same positions and postures as in FIGS. 9 and 16. The object OBJ is a jar having an opening at the top. View V1 shows an example of how it looks from user U1. Further, the view V2 shows an example of how the user U2 sees the view. Further, in FIG. 18, it is assumed that the service program 34 of the management server 30 transmits a command to generate the virtual object A1 at predetermined global coordinates to the wearable computers 1A and 1B, as in FIGS. 9 and 16.

In the example of FIG. 18, the wearable computer 1A and the wearable computer 1B execute the processing of the flowchart of FIG. 17 and determine whether or not the generated virtual object is displayed larger than the specified size. In the wearable computer 1A, since it is determined that the first evaluation value of the self-position in the first evaluation data is equal to or less than the threshold value, the virtual object A1 is displayed in a predetermined size. In the view V1 of the wearable computer 1A, the display deviation is small, and the virtual object A1 which is a character appears at a position corresponding to the upper part of the jar. Therefore, the user U1 can visually recognize how the character is popping out of the opening of the jar, as intended by the service.

On the other hand, in the wearable computer 1B, since it is determined that the first evaluation value of the self-position in the evaluation data is larger than the threshold value, the virtual object A1 is displayed larger than the specified size. With reference to view V2 of the wearable computer 1B, the virtual object A1 is enlarged 1.5 times as large as the specified size (size of FIG. 9). Therefore, in FIG. 16, as in FIG. 9, the display position of the virtual object A1 which is a character is displaced, but it is difficult for the user U2 to recognize this displacement.

In the example of FIG. 18, the virtual object A1 is enlarged in three directions of the x-axis, the y-axis, and the z-axis. However, the object generation unit 15 may expand the virtual object in the direction of a part of the axes. Further, the object generation unit 15 may execute a predetermined arithmetic process on the virtual object to change the shape of the virtual object and display it. As described above, the information processing apparatus according to the present disclosure controls at least one of the size and shape of the virtual object displayed on the display unit 7 according to the first evaluation value of the self-position in the first evaluation data. May be good.

The processing of the flowchart of FIG. 17 also makes it possible to eliminate the deviation in the appearance of virtual objects between users and improve the quality of services using AR.

As illustrated in the flowchart of FIG. 17 and FIG. 18, the object generation unit is a virtual object having a size larger when the first evaluation value is larger than the threshold value as compared with the case where the first evaluation value is less than or equal to the threshold value. May be generated.

Further, in the information processing system according to the present disclosure, when the first evaluation value is larger than the threshold value, the first information processing apparatus produces a virtual object having a larger size than when the first evaluation value is equal to or less than the threshold value. It may be generated. Further, the second information processing apparatus may generate a virtual object having a larger size when the third evaluation value is larger than the threshold value as compared with the case where the third evaluation value is equal to or less than the threshold value.

Note that the information processing device 1 may execute the processing of the flowchart of FIG. 15 in combination with the flowchart of FIG. That is, the information processing apparatus according to the present disclosure may execute a combination of a process of selecting a local map to be used when displaying a virtual object and a process of controlling at least one of the size and shape of the virtual object to be displayed.

FIG. 19 shows an example of wearing a head-mounted display. FIG. 19 shows an example in which the information processing apparatus according to the present disclosure is a head-mounted display 1C which is a kind of wearable computer. A display unit 7 is provided on a surface facing the user's eyes when the head-mounted display 1C is worn. As the display unit 7, for example, a transmissive display capable of displaying content so as to overlap the background can be used. A transmissive member 17 is provided on the surface of the head-mounted display 1C opposite to the display unit 7. As the transmitting member 17, for example, a material having transparency to visible light such as polyimide, acrylic, epoxy or glass can be used. However, the type of material used as the transparent member 17 does not matter.

The transmitting member 17 may be coated with a coating that prevents transmission of a specific wavelength band such as an infrared component or an ultraviolet component among electromagnetic waves. Further, a film that prevents transmission of a specific wavelength band of electromagnetic waves may be attached to the transmission member 17. Further, the head-mounted display 1C of FIG. 19 includes an earphone 16. The earphone 16 reproduces voice based on the voice signal output by the audio circuit 8.

The configuration of the head-mounted display shown in FIG. 19 is only an example. Therefore, a head-mounted display having a different shape may be used. Further, a head-mounted display in which an audio device such as an earphone is omitted may be used. For example, the user may use external earphones or headphones that receive audio signals transmitted by the head-mounted display via wireless communication (eg, Bluetooth).

FIG. 20 shows an example of the configuration of the display unit. The display unit 7 of FIG. 20 includes an illuminance sensor 3A, a dimming element 18, and a display 19. The illuminance sensor 3A detects the ambient brightness of the head-mounted display 1C. The dimming element 18 is, for example, a plate-shaped element containing an electrochromic material. The voltage applied to the dimming element 18 is adjusted according to the measured value of the illuminance sensor 3A. For example, the applied voltage of the dimming element 18 can be increased when the measured value of the illuminance sensor 3A becomes large, and the applied voltage of the dimming element 18 can be decreased when the measured value of the illuminance sensor 3A becomes small. .. By performing such control, it is possible to increase the transmittance of the dimming element 18 in an environment with low illuminance and suppress the transmittance of the dimming element 18 in an environment with high illuminance. The display 19 is a transmissive display. By adopting the display unit 7 having the configuration shown in FIG. 20, it is possible to provide the user with a head-mounted display 1C that can be used in a bright environment such as outdoors.

FIG. 21 shows an example of AR-related processing executed by the processing circuit 4 of the head-mounted display 1C. As shown in FIG. 21, AR-related processes are classified into three processes: a process for understanding the environment, a process for determining a response, and a process for superimposing display. An example of processing for understanding the environment includes self-positioning processing. The details of the self-position determination process are as described in the above-mentioned explanation of the self-position estimation unit 11. An example of processing for determining a response includes application logic processing. As an example of the application logic process, there is a virtual object display position determination process by the object generation unit 15 described above.

The processing for superimposed display includes, for example, image generation processing, delay compensation processing, sound image generation processing, and the like. When the process for superimposed display is completed, it becomes possible to generate an output signal to a display or an audio device.

The information processing device according to the present disclosure may include a display unit that superimposes and displays a virtual object generated by the object generation unit on the background. The display unit may include a transmissive display. Further, the information processing device according to the present disclosure may be a wearable computer.

By using the information processing device, information processing system, and information processing method according to the present disclosure, it is possible to reduce the deviation in the appearance of virtual objects among a plurality of users. As a result, the quality of service using AR and the satisfaction of users can be improved. Multiple users will be able to share their experiences in augmented reality space. Specifically, in an action game, a plurality of users can cooperate to fight against an enemy character. Since the map to be used at each position in the space can be selected, the map can be used according to the service situation and timing. In addition, since the map to be used can be switched at the timing required on the service side, it is possible to suppress the influence on the processing load and the response speed.

The present technology can have the following configurations.
(1)
A map generator that generates the first map based on the image,
A map acquisition unit that acquires a second map from another device,
An information processing device including an object generation unit that creates a virtual object using the first map or the second map.
(2)
An evaluation data generation unit that generates the first evaluation data based on the first map and the third map used for managing the service.
The information processing apparatus according to (1), further comprising a map selection unit for selecting the first map or the second map used by the object generation unit for generating the virtual object based on the first evaluation data.
(3)
It also has a self-position estimation unit that estimates the self-position.
The map selection unit acquires a first evaluation value corresponding to the self-position from the first evaluation data, and the object generation unit uses the first map to generate the virtual object based on the first evaluation value. Or select the second map,
The information processing device according to (2).
(4)
The map selection unit selects the first map when the first evaluation value is equal to or less than a threshold value.
The information processing device according to (3).
(5)
The map acquisition unit acquires the second evaluation data corresponding to the second map from the other device, and obtains the second evaluation data.
The map selection unit acquires a second evaluation value corresponding to the self-position from the second evaluation data, and selects the second map when the second evaluation value is larger than the first evaluation value.
The information processing device according to (3) or (4).
(6)
The map selection unit selects the first map when the second evaluation value is equal to or less than the first evaluation value.
The information processing device according to (5).
(7)
When the first evaluation value is larger than the threshold value, the object generation unit generates the virtual object having a larger size than when the first evaluation value is equal to or less than the threshold value, (3) to (3). The information processing apparatus according to any one of 6).
(8)
The evaluation data generation unit generates the first evaluation data based on the degree of similarity between the first map and the third map.
The information processing device according to any one of (2) to (7).
(9)
The map acquisition unit acquires the third map from the server.
(2) The information processing device according to any one of 8 to 8.
(10)
An image pickup unit for capturing the image is further provided.
The information processing device according to any one of (1) to (9).
(11)
Further, a sensor unit for measuring at least one of the direction in which the image pickup unit is facing or the inclination angle of the image pickup unit is provided.
The map generation unit generates the first map using the measured values of the sensor unit.
The information processing device according to (10).
(12)
A display unit that superimposes and displays a virtual object generated by the object generation unit on the background is further provided.
The information processing device according to any one of (1) to (11).
(13)
The display includes a transmissive display.
A wearable computer,
The information processing device according to (12).
(14)
A first information processing device that generates a first map based on a first image,
It is equipped with a second information processing device that generates a second map based on the second image.
The first information processing device generates a virtual object based on the first map or the second map acquired from the second information processing device.
The second information processing device generates the virtual object based on the second map or the first map acquired from the first information processing device.
Information processing system.
(15)
Further equipped with a server that manages services using the third map
The first information processing apparatus generates first evaluation data based on the first map and the third map, selects the first map or the second map based on the first evaluation data, and virtualizes the first map. Used to create an object
The second information processing device generates second evaluation data based on the second map and the third map, selects the second map or the first map based on the first evaluation data, and the above. Used to create virtual objects,
The information processing system according to (14).
(16)
The first information processing apparatus estimates the first self-position, acquires the first evaluation value corresponding to the first self-position from the first evaluation data, and based on the first evaluation value, of the virtual object. Select the first map or the second map to be used for generation,
The second information processing device estimates the second self-position, acquires the third evaluation value corresponding to the second self-position from the second evaluation data, and based on the third evaluation value, of the virtual object. Select the first map or the second map to be used for generation,
The information processing system according to (15).
(17)
The first information processing apparatus acquires the second evaluation data from the second information processing apparatus, acquires the second evaluation value corresponding to the first self-position from the second evaluation data, and obtains the second evaluation value. When the value is larger than the first evaluation value, the second map is selected.
The second information processing apparatus acquires the first evaluation data from the first information processing apparatus, acquires the fourth evaluation value corresponding to the second self-position from the first evaluation data, and obtains the fourth evaluation value. Select the first map when the value is greater than the third evaluation value.
The information processing system according to (16).
(18)
When the first evaluation value is larger than the threshold value, the first information processing device generates the virtual object having a size larger than that when the first evaluation value is equal to or less than the threshold value.
When the third evaluation value is larger than the threshold value, the second information processing device generates the virtual object having a larger size than when the third evaluation value is equal to or less than the threshold value.
The information processing system according to (16) or (17).
(19)
Steps to generate the first map based on the image,
The step of getting the second map from another device,
Have the computer perform the steps of creating a virtual object using the first map or the second map.
Information processing method.
(20)
The step of generating the first evaluation data based on the first map and the third map used for managing the service, and
The information processing method according to (19), further including a step of selecting the first map or the second map used for generating the virtual object based on the first evaluation data.

The aspects of the present disclosure are not limited to the individual embodiments described above, but also include various modifications that can be conceived by those skilled in the art, and the effects of the present disclosure are not limited to the above-mentioned contents. That is, various additions, changes and partial deletions are possible without departing from the conceptual idea and purpose of the present disclosure derived from the contents defined in the claims and their equivalents.

1 Information processing device 1A, 1B Wearable computer 1C Head-mounted display 2 Imaging unit 3 Sensor unit 3A Illumination sensor 4 Processing circuit 5 Storage unit 6, 32 Communication circuit 7 Display unit 8 Audio circuit 10 Local map generator 11 Self-position estimation unit 12 Map acquisition unit 13 Map selection unit 14 Evaluation data generation unit 15 Object generation unit 16 Earphones 17 Transmission member 18 Dimming element 19 Display 30 Management server 31 Bus 33 Processor 34 Service program 35 Storage 36 Global map data 40 Network 41 Base station

Claims (20)

1. A map generator that generates the first map based on the image,
   A map acquisition unit that acquires a second map from another device,
   An information processing device including an object generation unit that creates a virtual object using the first map or the second map.
2. An evaluation data generation unit that generates the first evaluation data based on the first map and the third map used for managing the service.
   The information processing apparatus according to claim 1, further comprising a map selection unit for selecting the first map or the second map used by the object generation unit for generating the virtual object based on the first evaluation data.
3. It also has a self-position estimation unit that estimates the self-position.
   The map selection unit acquires a first evaluation value corresponding to the self-position from the first evaluation data, and the object generation unit uses the first map to generate the virtual object based on the first evaluation value. Or select the second map,
   The information processing device according to claim 2.
4. The map selection unit selects the first map when the first evaluation value is equal to or less than a threshold value.
   The information processing device according to claim 3.
5. The map acquisition unit acquires the second evaluation data corresponding to the second map from the other device, and obtains the second evaluation data.
   The map selection unit acquires a second evaluation value corresponding to the self-position from the second evaluation data, and selects the second map when the second evaluation value is larger than the first evaluation value.
   The information processing device according to claim 3.
6. The map selection unit selects the first map when the second evaluation value is equal to or less than the first evaluation value.
   The information processing device according to claim 5.
7. The object generation unit according to claim 3, wherein when the first evaluation value is larger than the threshold value, the object generation unit generates the virtual object having a larger size than when the first evaluation value is equal to or less than the threshold value. Information processing device.
8. The evaluation data generation unit generates the first evaluation data based on the degree of similarity between the first map and the third map.
   The information processing device according to claim 2.
9. The map acquisition unit acquires the third map from the server.
   The information processing device according to claim 2.
10. An image pickup unit for capturing the image is further provided.
    The information processing device according to claim 1.
11. Further, a sensor unit for measuring at least one of the direction in which the image pickup unit is facing or the inclination angle of the image pickup unit is provided.
    The map generation unit generates the first map using the measured values of the sensor unit.
    The information processing device according to claim 10.
12. A display unit that superimposes and displays a virtual object generated by the object generation unit on the background is further provided.
    The information processing device according to claim 1.
13. The display includes a transmissive display.
    A wearable computer,
    The information processing device according to claim 12.
14. A first information processing device that generates a first map based on a first image,
    It is equipped with a second information processing device that generates a second map based on the second image.
    The first information processing device generates a virtual object based on the first map or the second map acquired from the second information processing device.
    The second information processing device generates the virtual object based on the second map or the first map acquired from the first information processing device.
    Information processing system.
15. Further equipped with a server that manages services using the third map
    The first information processing apparatus generates first evaluation data based on the first map and the third map, selects the first map or the second map based on the first evaluation data, and virtualizes the first map. Used to create an object
    The second information processing device generates second evaluation data based on the second map and the third map, selects the second map or the first map based on the first evaluation data, and the above. Used to create virtual objects,
    The information processing system according to claim 14.
16. The first information processing apparatus estimates the first self-position, acquires the first evaluation value corresponding to the first self-position from the first evaluation data, and based on the first evaluation value, of the virtual object. Select the first map or the second map to be used for generation,
    The second information processing device estimates the second self-position, acquires the third evaluation value corresponding to the second self-position from the second evaluation data, and based on the third evaluation value, of the virtual object. Select the first map or the second map to be used for generation,
    The information processing system according to claim 15.
17. The first information processing apparatus acquires the second evaluation data from the second information processing apparatus, acquires the second evaluation value corresponding to the first self-position from the second evaluation data, and obtains the second evaluation value. When the value is larger than the first evaluation value, the second map is selected.
    The second information processing apparatus acquires the first evaluation data from the first information processing apparatus, acquires the fourth evaluation value corresponding to the second self-position from the first evaluation data, and obtains the fourth evaluation value. Select the first map when the value is greater than the third evaluation value.
    The information processing system according to claim 16.
18. When the first evaluation value is larger than the threshold value, the first information processing device generates the virtual object having a size larger than that when the first evaluation value is equal to or less than the threshold value.
    When the third evaluation value is larger than the threshold value, the second information processing device generates the virtual object having a larger size than when the third evaluation value is equal to or less than the threshold value.
    The information processing system according to claim 16.
19. Steps to generate the first map based on the image,
    The step of getting the second map from another device,
    Have the computer perform the steps of creating a virtual object using the first map or the second map.
    Information processing method.
20. The step of generating the first evaluation data based on the first map and the third map used for managing the service, and
    The information processing method according to claim 19, further comprising a step of selecting the first map or the second map used for generating the virtual object based on the first evaluation data.

Images