WO2015093130A1

WO2015093130A1 - Information processing device, information processing method, and program

Info

Publication number: WO2015093130A1
Application number: PCT/JP2014/076620
Authority: WO
Inventors: 裕也花井
Original assignee: ソニー株式会社
Priority date: 2013-12-17
Filing date: 2014-10-03
Publication date: 2015-06-25
Also published as: JP2015118442A

Abstract

[Problem] To propose an information processing device, an information processing method, and a program that make it possible for a processing device to perform processing in accordance with the position of a manipulation device without putting restrictions on the positional relationship between the processing device and the manipulation device. [Solution] An information processing device that has the following: an inference unit that, on the basis of a taken image taken by an imaging unit that images real space, uses inference to generate first orientation information indicating the position and orientation of a manipulation device containing the imaging unit in an environment map that represents the position of an object in real space; and a communication unit that transmits the first orientation information generated by the inference unit to a processing device that performs processing in accordance with the position and orientation of the manipulation device.

Description

Information processing apparatus, information processing method, and program

This disclosure relates to an information processing apparatus, an information processing method, and a program.

As information processing devices such as smartphones, PCs (Personal Computers), and game machines have become widespread and highly functional, various UIs (User Interfaces) have been developed to enable more intuitive operation of information processing devices. Yes. As one of such UIs, there is a gesture input realized by a user wearing and moving an operation device. In order to realize the gesture input by the operation device, a technique is required for the processing device that accepts the gesture input to estimate the position and angle of the operation device operated by the user.

For example, in the field of game controllers, the following patent document 1 describes gesture input according to a controller position by estimating a controller position by tracking feature points provided in the controller by a statically installed camera. Techniques that enable this are disclosed.

JP 2012-135642 A

However, in the technique described in Patent Document 1, when the operating device is deviated from the angle of view of the installed camera, the feature point cannot be tracked and it is difficult to estimate the position of the operating device. That is, there is a restriction on the positional relationship between the processing device (camera) and the operation device. Therefore, in the present disclosure, a new and improved information processing apparatus and information processing method that enable processing according to the position of the operation device by the processing device without providing restrictions on the positional relationship between the processing device and the operation device. And suggest programs.

According to the present disclosure, the position and orientation of the operating device having the imaging unit in the environment map representing the position of the object existing in the real space based on the captured image captured by the imaging unit that captures the real space. An estimation unit that estimates first posture information to be shown, a communication unit that transmits the first posture information estimated by the estimation unit to a processing device that performs processing according to the position and posture of the controller device; An information processing apparatus is provided.

Further, according to the present disclosure, the position of the operating device having the imaging unit in the environment map representing the position of the object existing in the real space based on the captured image captured by the imaging unit that captures the real space, and Estimating first posture information indicating a posture, and transmitting the estimated first posture information to a processing device that performs processing according to a position and a posture of the operating device. An information processing method executed by a processor of a processing device is provided.

In addition, according to the present disclosure, the operation device having the imaging unit in the environment map representing the position of the object existing in the real space based on a captured image captured by the imaging unit that captures the real space. An estimation unit that estimates first posture information indicating the position and posture of the controller, and the first posture information estimated by the estimation unit is transmitted to a processing device that performs processing according to the position and posture of the controller device And a communication unit functioning program are provided.

As described above, according to the present disclosure, it is possible to perform processing in accordance with the position of the operation device by the processing device without limiting the positional relationship between the processing device and the operation device.
Note that the above effects are not necessarily limited, and any of the effects shown in the present specification, or other effects that can be grasped from the present specification, together with or in place of the above effects. May be played.

It is explanatory drawing for demonstrating the outline | summary of the display system which concerns on one Embodiment. It is a block diagram which shows the structural example of the display system which concerns on 1st Embodiment. It is a figure which shows the example of AR display by the display apparatus which concerns on 1st Embodiment. It is a sequence diagram which shows an example of the flow of the AR display process performed in the display system which concerns on 1st Embodiment. It is a block diagram which shows the structural example 1 of the display system which concerns on 2nd Embodiment. It is a figure which shows the example of AR display by the display apparatus which concerns on 2nd Embodiment. It is a block diagram which shows the structural example 2 of the display system which concerns on 2nd Embodiment. It is a block diagram which shows the structural example 3 of the display system which concerns on 2nd Embodiment. It is explanatory drawing for demonstrating the outline | summary of the display system which concerns on 3rd Embodiment. It is a block diagram which shows the structural example of the display system which concerns on 3rd Embodiment.

Hereinafter, preferred embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. In addition, in this specification and drawing, about the component which has the substantially same function structure, duplication description is abbreviate | omitted by attaching | subjecting the same code | symbol.

The description will be made in the following order.
1. Overview 2. First embodiment 2-1. Configuration example of operation device 2-2. Configuration example of display device 2-3. 2. Operation processing Second embodiment 3-1. Configuration example 1
3-2. Configuration example 2
3-3. Configuration example 3
4). Third embodiment 4-1. Outline 4-2. Configuration example 5. Summary

<1. Overview>
First, an overview of a display system according to an embodiment of the present disclosure will be described with reference to FIG.

FIG. 1 is an explanatory diagram for explaining an overview of a display system according to an embodiment. As shown in FIG. 1, the user holds a smartphone 1 as a display device and a smartphone 2 as an operation device. The display system includes a display device 1 and an operation device 2, and the display device 1 performs a display 100 according to the position of the operation device 2.

Here, a technology called augmented reality (AR) that superimposes additional information on the real space and presents it to the user will be described. Information presented to the user in AR technology is visualized using various forms of virtual objects such as text, icons or animations. The virtual object is arranged in the AR space according to the position of the associated real object. In addition, the virtual object can perform an operation such as movement, collision, or deformation in the AR space. The display device 1 according to the present embodiment arranges a virtual object in the AR space according to the position of the controller device 2 and displays an AR image representing an action such as movement, collision, or deformation.

As shown in FIG. 1, the user is holding the display device 1 with his right hand and looking at the display unit 17, and the camera disposed on the back side of the display unit 17 is pointed at the desk. The display device 1 superimposes and displays an AR image showing a state in which the ball 8 that is a virtual object is ejected from a position corresponding to the operation device 2 on a through image captured in real time on a desk by a camera. Yes. The display device 1 can inject the ball 8 from a position corresponding to the operation device 2 by estimating the position of the operation device 2 in the through image.

There are various ways in which the display device 1 estimates the position of the operation device 2. As an example, a comparative example in which the operation device 2 is imaged by a camera provided in the display device 1 and a relative position between the display device 1 and the operation device 2 is estimated by image recognition can be considered. However, in this comparative example, when the controller device 2 is out of the angle of view of the camera, the controller device 2 is not captured in the captured image, so it is difficult to estimate the relative position. That is, in order for the display device 1 to succeed in estimating the relative position, there is a constraint that the operation device 2 exists within the imaging range of the camera provided in the display device 1.

Accordingly, the operation device 2 (information processing device) according to the present embodiment has been created with the above circumstances taken into consideration. The operation device 2 according to the present embodiment enables processing according to the position of the operation device 2 by the display device 1 without providing a restriction on the positional relationship between the display device 1 and the operation device 2.

Specifically, the controller device 2 according to the present embodiment estimates the position and orientation (angle) of the controller device 2 in the world coordinate system, and transmits the estimation result to the display device 1. The world coordinate system is a coordinate system set in the real space. Similarly, the display device 1 also estimates the position and orientation of the display device 1 in the same world coordinate system. Then, the display device 1 estimates the relative position between the display device 1 and the operation device 2 by integrating the estimation result received from the operation device 2 and its own estimation result. Reference numeral 9 shown in FIG. 1 schematically shows the position in the through image of the operating device 2 estimated by the display device 1 by such integration processing. The display device 1 superimposes an AR image indicating the ball 8 at a position corresponding to the position indicated by reference numeral 9. As a result, the superimposed display of the AR image at the position corresponding to the controller device 2 in the through image is realized. Since the display device 1 does not need to track the operation device 2 by image recognition, the relative position can be estimated even when the operation device 2 is out of the angle of view of the camera. That is, the positional relationship between the display device 1 and the operation device 2 is not limited.

In general, the display device 1 performs calculation of the position and orientation of a virtual object, display processing of an AR image, recognition of gesture input according to the movement of the operation device 2, calculation of output based on the recognition result, and the like. . In addition, according to the above-described comparative example, the display device needs to perform image recognition to estimate the position of the controller device, and the processing load becomes great. On the other hand, according to the present embodiment, the display device 1 does not need to perform image recognition, so the processing load is reduced.

In addition, in FIG. 1, although the example which the display apparatus 1 which concerns on one Embodiment of this indication is implement | achieved as a smart phone was shown, the technique which concerns on this indication is not limited to this. For example, the display device 1 may be an HMD (Head Mounted Display), a digital camera, a digital video camera, a tablet terminal, a mobile phone terminal, or the like. When the display device 1 is realized as an HMD, the HMD may display an AR image superimposed on a through image captured by a camera, or a display unit formed in a transparent or translucent through state. An AR image may be displayed.

Similarly, FIG. 1 illustrates an example in which the operation device 2 according to an embodiment of the present disclosure is realized as a smartphone, but the technology according to the present disclosure is not limited thereto. For example, the controller device 2 may be a digital camera, a digital video camera, a tablet terminal, a mobile phone terminal, or the like. In addition, the operation device 2 may be an operation-dedicated device having a rod shape, a spherical shape, or any other shape.

1 illustrates an example in which the processing device according to an embodiment of the present disclosure is realized as a display device, but the technology according to the present disclosure is not limited thereto. For example, in addition to the display of the AR image, the processing device performs a process for expressing an arbitrary interaction between the controller device 2 and the real world, such as vibration according to the position of the controller device 2, sound reproduction, and light emission. May be.

The overview of the display system according to this embodiment has been described above. Hereinafter, each embodiment will be described in detail.

<2. First Embodiment>
First, the configuration of the display system according to the present embodiment will be described with reference to FIG. FIG. 2 is a block diagram illustrating a configuration example of the display system according to the first embodiment. As shown in FIG. 2, the display system according to the present embodiment includes a display device 1-1 and an operation device 2-1, and shares the same environment map 3.

[2-1. Example of operation device configuration]
As illustrated in FIG. 2, the controller device 2-1 includes an imaging unit 21, a sensor 22, a posture estimation unit 23, a posture information transmission unit 24, a user operation acquisition unit 25, and a control signal transmission unit 26.

(1) Imaging unit 21
The imaging unit 21 photoelectrically converts imaging light obtained by a lens system including an imaging lens, a diaphragm, a zoom lens, and a focus lens, a drive system that causes the lens system to perform a focus operation and a zoom operation, and the lens system. A solid-state imaging device array that generates an imaging signal. The solid-state imaging device array may be realized by, for example, a CCD (Charge Coupled Device) sensor array or a CMOS (Complementary Metal Oxide Semiconductor) sensor array. The imaging unit 21 according to the present embodiment images a real space and outputs a captured image to the posture estimation unit 23.

(2) Sensor 22
The sensor 22 has a function of detecting the attitude of the controller device 2-1. For example, the sensor 22 is realized by an acceleration sensor, an angular velocity (gyro) sensor, or a geomagnetic sensor. The sensor 22 outputs a sensing result indicating the detected gravity direction, inclination, or angle to the posture estimation unit 23.

(3-1) Environmental map 3
The environment map 3 is information that represents a three-dimensional position of an object existing in the real space. More specifically, the environment map 3 is a set of data representing the position and orientation of one or more objects existing in the real space. The environment map can include, for example, an object name corresponding to the object, a three-dimensional position of a feature point corresponding to the object, and polygon information constituting the shape of the object. The environment map can be constructed, for example, by obtaining the three-dimensional position of each feature point from the position of the feature point on the imaging surface.

In FIG. 2, the environment map 3 is expressed as not belonging to either the display device 1-1 or the operation device 2-1, but can be accessed from both the display device 1-1 and the operation device 2-1. It is expressed that there is. This indicates that the display device 1-1 and the operation device 2-1 share the same environment map 3. As a result, the display device 1-1 and the operation device 2-1 each estimate the position and angle in the same world coordinate system, and the posture information integration process by the display device 1-1 described later becomes possible. The environment map 3 may be included in the display device 1-1, and the operation device 2-1 may access the environment map 3 through communication with the display device 1-1. Similarly, the environment map 3 may be included in the operation device 2-1, or may be a home server connected to the display device 1-1 and the operation device 2-1, or a server on the cloud. It may be included in the information processing apparatus.

Also, if the display device 1-1 and the operation device 2-1 can estimate the position and angle in the same world coordinate system, what is shared need not be the environment map 3. For example, the display device 1-1 and the operation device 2-1 may have a common so-called planar marker recognition dictionary as a starting point capable of defining a common coordinate system.

(3-2) Attitude estimation unit 23
The posture estimation unit 23 has a function as an estimation unit that estimates posture information (first posture information) indicating the position and angle (posture) of the controller device 2-1 in the environment map 3 (world coordinate system). Specifically, the posture estimation unit 23 estimates the position and posture of the controller device 2-1 by estimating the position and posture of the imaging unit 21. As one of the position estimation techniques in the world coordinate system, for example, there is a technique called SLAM (Simultaneous Localization And Mapping) that can simultaneously estimate the position and orientation of a camera and the position of a feature point reflected in a camera image. The basic principle of SLAM technology using a monocular camera is described in “Andrew J. Davison,“ Real-Time Simultaneous Localization and Mapping with a Single Camera ”, Proceedings of the 9th IEEE International Conference on Computer Vision Volume 2, 2003, pp. .1403-1410 ". The SLAM technique for visually estimating a position using a camera image is also referred to as VSLAM (visual SLAM). The posture estimation unit 23 uses the SLAM technology to estimate the position and posture of the controller device 2-1 based on the matching result between the feature points in the environment map 3 and the feature points reflected in the captured image captured by the imaging unit 21. In addition, the posture estimation unit 23 estimates the position and posture of the imaging unit 21 in the environment map 3 by using a posture estimation technique using a marker, DTAM (Dense Tracking and Mapping in Real-Time), and a technique such as Kinect Fusion. May be. The posture estimation unit 23 may correct the estimation result of the position and posture by the SLAM technique using the various sensing results output from the sensor 22. The posture estimation unit 23 outputs the estimated posture information to the posture information transmission unit 24.

(4) Posture information transmission unit 24
The posture information transmission unit 24 has a function of transmitting the posture information estimated by the posture estimation unit 23 to the display device 1-1. Thereby, the display device 1-1 can perform various processes according to the position and posture of the operation device 2-1. The attitude information transmission unit 24 is a communication module for transmitting and receiving data to and from the display device 1-1 by wire / wireless. For example, the attitude information transmission unit 24 is directly connected to the display device 1-1 by a method such as wireless LAN (Local Area Network), Wi-Fi (Wireless Fidelity (registered trademark)), infrared communication, Bluetooth (registered trademark), or the like. Wireless communication is performed via a network access point.

In FIG. 2, the posture estimation unit 23 and the posture information transmission unit 24 are included in the controller device 2-1, but the technology according to the present disclosure is not limited to this. For example, the posture estimation unit 23 and the posture information transmission unit 24 may be included in other information processing devices connected to the display device 1-1 and the operation device 2-1. In this case, the other information processing apparatus estimates the posture information of the controller device 2-1 based on the captured image and the sensing result received from the controller device 2-1, and the estimated posture information is displayed on the display device 1-1. You may send it. Similarly, the posture estimation unit 23 may be included in the display device 1-1.

(5) User operation acquisition unit 25
The user operation acquisition unit 25 has a function of acquiring a user operation as control information for controlling the display device 1-1 according to the user operation. For example, the user operation acquisition unit 25 is realized by a button, a keyboard, a mouse, a trackball, a touch pad, a touch panel, or the like. The user operation acquisition unit 25 outputs control information indicating the acquired user operation to the control signal transmission unit 26.

(6) Control signal transmitter 26
The control signal transmission unit 26 has a function of transmitting the control information output by the user operation acquisition unit 25 to the display device 1-1. Thereby, the display device 1-1 can perform processing according to the control information. The control signal transmission unit 26 is a communication module for transmitting and receiving data to and from the display device 1-1 by wire / wireless. For example, the control signal transmission unit 26 uses a wireless LAN, Wi-Fi (registered trademark), infrared communication, Bluetooth (registered trademark), or the like to perform wireless communication directly with the display device 1-1 or via a network access point. To do.

(7) Supplement Although not explicitly shown in FIG. 1, the operation device 2-1 functions as an arithmetic processing device and a control device, and has a control unit that controls the overall operation in the operation device 2-1 according to various programs. You may do it. The control unit is realized by, for example, a CPU (Central Processing Unit) and a microprocessor. The control unit may include a ROM (Read Only Memory) that stores programs to be used, calculation parameters, and the like, and a RAM (Random Access Memory) that temporarily stores parameters that change as appropriate.

[2-2. Example of configuration of display device]
As illustrated in FIG. 2, the display device 1-1 includes an imaging unit 11, a sensor 12, a posture estimation unit 13, a posture information reception unit 14, a control information reception unit 15, a control unit 16, and a display unit 17.

(1) Imaging unit 11
The imaging unit 11 photoelectrically converts imaging light obtained by a lens system including an imaging lens, a diaphragm, a zoom lens, and a focus lens, a drive system that causes the lens system to perform a focus operation and a zoom operation, and the lens system. A solid-state imaging device array that generates an imaging signal. The solid-state image sensor array may be realized by, for example, a CCD sensor array or a CMOS sensor array. The imaging unit 11 according to the present embodiment images a real space and outputs a captured image to the posture estimation unit 13.

(2) Sensor 12
The sensor 12 has a function of detecting the attitude of the display device 1-1. For example, the sensor 12 is realized by an acceleration sensor, an angular velocity (gyro) sensor, or a geomagnetic sensor. The sensor 12 outputs a sensing result indicating the detected gravity direction, inclination, or angle to the posture estimation unit 13.

(3) Posture estimation unit 13
The posture estimation unit 13 has a function as an estimation unit that estimates posture information (second posture information) indicating the position and angle (posture) of the display device 1-1 in the environment map 3 (world coordinate system). Specifically, the posture estimation unit 13 estimates the position and posture of the display device 1-1 by estimating the position and posture of the imaging unit 11. Similar to the posture estimation unit 23 described above, the posture estimation unit 13 uses the SLAM technology to display the display device 1 based on the matching result between the feature points in the environment map 3 and the feature points reflected in the captured image captured by the imaging unit 11. -1 position and orientation are estimated. In addition, the posture estimation unit 13 may estimate the position and posture of the imaging unit 11 in the environment map 3 by a posture estimation technology using a marker, or a technology such as DTAM or Kinect Fusion. The posture estimation unit 13 may correct the posture estimation result by the SLAM technology using various sensing results output by the sensor 12. The posture estimation unit 13 outputs the estimated posture information to the control unit 16.

In FIG. 2, the posture estimation unit 13 is included in the display device 1-1, but the technology according to the present disclosure is not limited to this. For example, the posture estimation unit 13 may be included in another information processing apparatus connected to the display device 1-1 and the operation device 2-1. In this case, the other information processing device estimates the posture information of the display device 1-1 based on the captured image and the sensing result received from the display device 1-1, and sends the estimated posture information to the display device 1-1. You may send it. Similarly, the posture estimation unit 13 is included in the operation device 2-1, estimates posture information of the display device 1-1 based on a captured image and a sensing result received from the display device 1-1, and estimates the posture. Information may be returned to the display device 1-1.

(4) Attitude information receiver 14
The posture information receiving unit 14 has a function of receiving the posture information of the controller device 2-1 transmitted by the posture information transmitting unit 24. The posture information receiving unit 14 is a communication module for transmitting / receiving data to / from the controller device 2-1 by wire / wireless. For example, the attitude information receiving unit 14 wirelessly communicates with the controller device 2-1 directly or via a network access point using a method such as wireless LAN, Wi-Fi (registered trademark), infrared communication, Bluetooth (registered trademark), or the like. To do. The posture information receiving unit 14 outputs the received posture information to the control unit 16.

(5) Control information receiver 15
The control information receiving unit 15 has a function of receiving the control information transmitted by the control signal transmitting unit 26. The control information receiving unit 15 is a communication module for transmitting / receiving data to / from the controller device 2-1 by wire / wireless. For example, the control information receiving unit 15 wirelessly communicates with the controller device 2-1 directly or via a network access point using a wireless LAN, Wi-Fi (registered trademark), infrared communication, Bluetooth (registered trademark), or the like. To do. The control information receiving unit 15 outputs the received control information to the control unit 16.

(6) Control unit 16
The control unit 16 functions as an arithmetic processing device and a control device, and controls the overall operation in the display device 1-1 according to various programs. The control unit 16 is realized by, for example, a CPU or a microprocessor. The control unit 16 may include a ROM that stores programs to be used, calculation parameters, and the like, and a RAM that temporarily stores parameters that change as appropriate.

The control unit 16 according to the present embodiment integrates the posture information of the display device 1-1 output from the posture estimation unit 13 and the posture information of the operation device 2-1 received by the posture information reception unit 14 to display The relative position between the device 1-1 and the controller device 2-1 is calculated. Since these posture information are position information in the world coordinate system defined by the same environment map 3, the relative position can be calculated by integrating them. Here, the relative position means the position of the operating device 2-1 in the coordinate system of the display device 1-1. The relative position may be regarded as the position of the operating device 2-1 in the through image captured by the operating device 2. This relative position is used in determining the virtual object drawing method. Hereinafter, display control of the virtual object by the control unit 16 will be described. First, the control unit 16 acquires the position of the operation device 2-1 in the world coordinate system from the posture information of the operation device 2-1 received by the posture information reception unit. Next, the control unit 16 determines the position of the virtual object in the world coordinate system. Next, the control unit 16 determines the positions of the operation device 2-1 and the virtual object in the coordinate system of the display device 1-1. Then, the control unit 16 determines a virtual object drawing method according to the positional relationship between the controller device 2-1 and the virtual object in the coordinate system of the display device 1-1. As a drawing method, for example, partial occlusion is expressed according to the context of the operation device 2-1 and the virtual object as viewed from the display device 1-1. Since the control unit 16 can calculate the relative position, it is possible to perform expression according to such a context. Since the display device 1 does not need to track the operation device 2 by image recognition, the relative position can be estimated even when the operation device 2 is out of the angle of view of the camera. Further, since the control unit 16 does not need to perform posture estimation of the operation device 2-1 by image recognition performed in the technique described in Patent Document 1, the processing load on the display device 1 side is reduced. The

In addition, the control unit 16 performs various processes according to the control information received by the control information receiving unit 15. Specifically, in addition to the calculated relative position between the display device 1-1 and the operation device 2-1, the control unit 16 displays the display unit 17 so as to display a visual effect corresponding to the user operation indicated by the control information. Control. For example, when a predetermined user operation is acquired by the user operation acquisition unit 25, the control unit 16 displays an AR image indicating that the ball 8 illustrated in FIG. 1 is ejected from the operation device 2-1.

The control unit 16 may display a virtual operation assisting device as an example of a virtual object reflecting the relative position and the control information. The virtual operation assisting device is a virtual object indicating an action point defined by a relative position with respect to the operation device 2-1. With the virtual operation assisting device, the operating point of the operation device 2-1 can be installed at a position away from the main body of the operation device 2-1, or the operation content can be visually expressed in a user-friendly manner. . Hereinafter, a display example of the virtual operation assistance device will be described with reference to FIG.

FIG. 3 is a diagram showing an example of AR display by the display device 1-1 according to the first embodiment. FIG. 3 shows an example in which the display device 1-1 displays the AR image 8-1 showing the virtual object of the light bulb in front of the operation device 2-1 in the through image. The light bulb 8-1 moves along with the operating device 2-1, as if it were integrated with the operating device 2-1. This is realized by the control unit 16 estimating the position of the operation device 2-1 in the through image and displaying the AR image 8-1 fixedly in front of the operation device 2-1. The user can use the light bulb 8-1 as a part of the operation device 2-1. Specifically, the display device 1-1 causes the light bulb 8-1 to function as an action point, and displays various AR images with the light bulb 8-1 as a reference point superimposed on the through image. For example, when a horizontal swipe operation is acquired by a user operation acquisition unit 25 that is configured integrally with the display unit as a touch panel, the control unit 16 may switch the color of the light bulb 8-1. In addition, when the tap operation is detected by the user operation acquisition unit 25, the control unit 16 may display the neon sign 8-2 at the three-dimensional position through which the light bulb 8-1 has passed. In addition, when the swipe operation in the vertical direction is acquired by the user operation acquisition unit 25, the control unit 16 may move the bulb 8-1 away from or close to the operation device 2-1. In FIG. 3, the display (reference numeral 9 in FIG. 1) showing the result of the posture estimation of the controller device 2-1 by the display device 1-1 is omitted.

In addition to the light bulb 8-1 shown in FIG. 3, various virtual objects are conceivable for the virtual operation assisting device. For example, scissors, a cutter, a pen, a ruler, etc. can be virtual operation assisting devices and can act on virtual objects and real objects in real space. These multiple types of virtual operation assistance devices may be switched by a swipe operation in the user operation acquisition unit 25 formed as a touch panel, for example. Further, for example, the user may wear the display device 1-1 formed as an HMD, hold the operation device 2-1 with both hands, and operate the virtual auxiliary device with both hands.

Note that the control unit 16 may express various visual effects such as displaying text and images in accordance with the position of the operation device 2-1, in addition to displaying the AR image. In addition, the control unit 16 performs processing for expressing any interaction between the controller device 2 and the real world, such as vibration, sound reproduction, and light emission according to the position of the controller device 2-1. -1 or the controller device 2-1 may be controlled.

In FIG. 2, the attitude information receiving unit 14, the control information receiving unit 15, and the control unit 16 are included in the display device 1-1. However, the technology according to the present disclosure is not limited to this. For example, the posture information receiving unit 14, the control information receiving unit 15, and the control unit 16 may be included in another information processing apparatus connected to the display device 1-1 and the operation device 2-1. In this case, the other information processing device generates display control information indicating the content of the AR image based on the posture information and control information received from the display device 1-1 and the operation device 2-1, and performs display control. Information may be transmitted to the display device 1-1 to control display by the display unit 17. Similarly, the posture information receiving unit 14, the control information receiving unit 15, and the control unit 16 may be included in the controller device 2-1.

(7) Display unit 17
The display unit 17 displays a through image captured in real time by the imaging unit 11 based on control by the control unit 16, displays various AR images in a superimposed manner, and stores stored image data (still image data / Moving image data) is displayed (reproduced). The display unit 17 is realized by, for example, an LCD (Liquid Crystal Display) or an OLED (Organic Light-Emitting Diode). When the processing apparatus according to the present embodiment is realized as an HMD, the display unit 17 is formed as a transparent or translucent through state, and displays an AR image in a real space reflected on the display unit 17 in the through state. Also good.

The configuration example of the display system according to the present embodiment has been described above. Subsequently, an operation process of the display system according to the present embodiment will be described with reference to FIG.

[2-3. Operation processing]
FIG. 4 is a sequence diagram illustrating an example of the flow of the AR display process executed in the display system according to the first embodiment. The display device 1-1 and the operation device 2-1 are involved in the sequence shown in FIG. The sequence shown in FIG. 4 shows an example in which each process is executed by either the display device 1-1 or the operation device 2-1, but each process may be executed by either of them. It may be centrally executed by another information processing apparatus such as a server.

As shown in FIG. 4, first, in step S102-1, the display device 1-1 performs posture estimation. Specifically, the posture estimation unit 13 uses SLAM technology to determine the position and posture of the display device 1-1 based on the matching result between the feature points in the environment map 3 and the feature points reflected in the captured image captured by the imaging unit 11. presume. At this time, the posture estimation unit 13 may correct the posture estimation result by the SLAM technique using various sensing results output by the sensor 12. Similarly, in step S102-2, the controller device 2-1 performs posture estimation. In particular, the posture estimation unit 23 estimates the position and posture of the controller device 2-1 using the same environment map 3 as the posture estimation unit 13.

Next, in step S104, the controller device 2-1 transmits the estimated posture information to the display device 1-1. Specifically, the posture information transmission unit 24 transmits the posture information estimated in step S102-2 by the posture estimation unit 23 to the display device 1-1.

Next, in step S106, the display device 1-1 integrates the posture information of the display device 1-1 and the posture information of the operation device 2-1. Specifically, the control unit 16 integrates the posture information of the display device 1-1 output by the posture estimation unit 13 and the posture information of the operation device 2-1 received by the posture information reception unit 14 to display the display device 1. −1 and the operation device 2-1 are calculated.

On the other hand, in step S108, the controller device 2-1 acquires a user operation. Specifically, the user operation acquisition unit 25 acquires a user operation such as a touch on the touch panel or a button press by the user as control information for controlling the display device 1-1 according to the user operation.

Next, in step S110, the controller device 2-1 transmits the acquired control information to the display device 1-1. Specifically, the control signal transmission unit 26 transmits control information indicating the user operation acquired in step S108 to the display device 1-1.

Next, in step S112, the display device 1-1 calculates visual information. Specifically, the control unit 16 calculates visual information corresponding to the user operation indicated by the control information in addition to the relative position between the display device 1-1 and the operation device 2-1 calculated in step S106. Specifically, as described with reference to FIGS. 1 and 3, the control unit 16 calculates the position of the operating device 2-1 in the through image captured by the imaging unit 11, and operates the operating device 2- Display control information for displaying an AR image indicating a virtual object reflecting the control information at a position corresponding to 1 is generated.

In step S114, the display device 1-1 performs AR display. Specifically, the display unit 17 performs display based on the display control information generated by the control unit 16 in step S112.

The operation process of the display system according to this embodiment has been described above.

<3. Second Embodiment>
This embodiment is a form which avoids the situation where the posture estimation of the controller device 2 becomes difficult by determining the approach between the controller device 2 and the real object and notifying the user. Various configurations of the present embodiment are conceivable. As an example, a configuration example of a display system according to the present embodiment will be described with reference to FIGS.

[3-1. Configuration Example 1]
FIG. 5 is a block diagram illustrating a configuration example 1 of the display system according to the second embodiment. As shown in FIG. 5, the display system according to the present embodiment includes a display device 1-2 and an operation device 2-2. The display system according to the first embodiment described with reference to FIG. In addition, it has three-dimensional data 4, an approach determination unit 27, and an approach notification unit 28.

(1) Three-dimensional data 4
The three-dimensional data 4 is data composed of the position information of the vertices of the object in the real space, the line segments connecting the vertices, and the surface surrounded by the line segments, and is information expressing the three-dimensional shape (Surface) of the real space. is there. Here, the position information included in the three-dimensional data 4 is information indicating a position in the world coordinate system in the environment map 3. The three-dimensional data 4 may be dynamically generated using the captured image captured by the imaging unit 21 and the posture information estimated by the posture estimation unit 23, or may be generated in advance. The three-dimensional data is realized as CAD (Computer Assisted Drafting) data, for example.

In FIG. 5, the three-dimensional data 4 is expressed as belonging to neither the display device 1-2 nor the operation device 2-2, and is expressed as accessible from the operation device 2-2. . This indicates that the three-dimensional data 4 may be included in any device. For example, the three-dimensional data 4 may be included in the display device 1-2 or the operation device 2-2, or may be a home server connected to the operation device 2-2 via a network or a server on the cloud. It may be included in another information processing apparatus.

(2) Approach determination unit 27
The approach determination unit 27 has a function of determining whether or not the distance between the imaging surface of the imaging unit 21 and the real object is equal to or less than a threshold value. As described above, the posture estimation unit 23 uses the SLAM technique to determine the position of the controller device 2-1 based on the matching result between the feature points in the environment map 3 and the feature points reflected in the captured image captured by the imaging unit 21. Estimate posture. In order for the posture estimation unit 23 to succeed in estimating posture information, it is necessary that at least a feature point is included in the imaging range and that the feature point is reflected in the captured image to the extent that it can be identified. Therefore, the approach determination unit 27 determines whether or not the imaging surface of the imaging unit 21 and the real object are excessively close. When the approach determination unit 27 determines that the distance between the imaging surface of the imaging unit 21 and the real object is equal to or less than the threshold, the posture estimation unit 23 may not satisfy the condition for successfully estimating the posture information. Therefore, a notification by an approach notification unit 28 described later is made. In addition, the approach determining unit 27 may perform notification by the approach notifying unit 28 when no feature point is included in the imaging range, such as when the imaging surface of the imaging unit 21 faces the sky.

The approach determination unit 27 calculates the distance between the imaging surface of the imaging unit 21 and the real object based on the attitude information of the controller device 2-2 estimated by the attitude estimation unit 23 and the three-dimensional data 4. Specifically, the approach determination unit 27 obtains the position and angle of the controller device 2-2 in the world coordinate system indicated by the posture information by taking the correspondence between the three-dimensional shape indicated by the three-dimensional data 4 and the posture information. The position and angle of the operation device 2-2 with respect to the three-dimensional shape are converted. Then, the approach determination unit 27 calculates the distance between the surface of the real object indicated by the three-dimensional shape and the imaging surface of the imaging unit 21 included in the controller device 2-2. In addition, the approach determination unit 27 simply generates a mesh by applying Delaunay triangulation to the SLAM feature point information, and regards this mesh as a pseudo three-dimensional shape and approaches it. A determination may be made. In this case, the approach determination unit 27 can perform the approach determination without using the three-dimensional data 4.

In FIG. 5, the approach determination unit 27 is included in the operation device 2-2, but the technology according to the present disclosure is not limited to this. For example, the approach determination unit 27 may be included in another information processing apparatus connected to the controller device 2-2. In this case, the other information processing device calculates the distance between the imaging surface of the imaging unit 21 and the real object based on the three-dimensional data 4 and the posture information received from the operation device 2-2, and makes an approach determination. The notification by the approach notification unit 28 may be controlled. Similarly, the approach determination unit 27 may be included in the display device 1-2.

(3) Approach notification unit 28
The approach notification unit 28 has a function as a notification unit that performs notification indicating that the distance between the imaging surface of the imaging unit 21 and the real object is equal to or less than a threshold based on the determination result by the approach determination unit 27. For example, the approach notification unit 28 is configured by a display unit, a vibration motor, a speaker, an LED, or the like, and may perform notification by screen display, vibration, sound reproduction, light emission, or the like. The notification by the approach notification unit 28 prompts the user to move the operating device 2-2 away from the real object, and can prevent a situation in which it is difficult to estimate the posture of the operating device 2-2.

(4) Supplement 1
In addition, the control unit 16 may assist the virtual operation assisting device to avoid a situation in which posture estimation by the operation device 2-2 becomes difficult. For example, in the through image, the control unit 16 displays the virtual operation assistance device at a position where the distance between the imaging surface of the imaging unit 21 and the real object can be equal to or less than a threshold value. With this display, the control unit 16 can prompt the user to prevent the virtual operation assisting device from colliding with the real object, and can avoid a situation where it is difficult to estimate the posture of the operation device 2-2. Of course, when these collide, the display system may perform notification by warning display, vibration, sound reproduction, light emission, or the like. Hereinafter, a display example of such a virtual operation assistance device will be described with reference to FIG.

FIG. 6 is a diagram showing an AR display example by the display device 1-2 according to the second embodiment. In FIG. 6, a virtual operation assisting device for assisting the display device 1-2 to avoid a situation where posture estimation by the operation device 2-2 is difficult in front of the operation device 2-2 in a through image. An example of displaying 8-3 is shown. Similar to the light bulb 8-1 described above with reference to FIG. 3, the virtual operation assisting device 8-3 is attached to the operation device 2-2 as if it is integrated with the operation device 2-2. Move. With such a virtual operation assisting device 8-3, the control unit 16 makes it difficult for the distance between the imaging surface of the imaging unit 21 and the real object to be equal to or smaller than the threshold, that is, the posture estimation of the operating device 2-2. The situation can be avoided in advance.

Furthermore, the control unit 16 allows the virtual operation assisting device to move to an arbitrary position in accordance with a user operation, thereby enabling an action at a position close to the real object. Specifically, the control unit 16 approaches the virtual operation assisting device instead of the main body of the operation device 2-2 even in a place where it is difficult to estimate the posture by the operation device 2-2 because the distance to the real object is too short. be able to. For example, the control unit 16 moves the virtual operation assisting device in the up / down / left / right direction or the direction away / approaching from the operation device 2-2 by a swipe operation in the user operation acquisition unit 25 formed as a touch panel. The action point is guided to a position corresponding to the user operation. Thereby, the control part 16 can express the interaction to the position close | similar to a real object, for example, the position which approached the real object itself or a real object.

(5) Supplement 2
There are various methods for avoiding a situation in which it is difficult to estimate the posture of the controller device 2 itself. As an example, the posture estimation unit 23 improves the robustness of posture estimation by the SLAM technology by selecting the captured image in the imaging direction that is easy to estimate the posture and estimating the posture information of the controller device 2-2. Can be considered. For example, the imaging unit 21 may be formed as two wide-angle cameras provided on the back and front surfaces of the casing of the controller device 2-2, or a 360-degree omnidirectional camera. In this case, the posture estimation unit 23 can perform posture estimation by selectively using a region where feature points are easily recognized from the captured image. In addition, the image capturing unit 21 is provided on, for example, a pan head that moves with six degrees of freedom, and the posture estimating unit 23 moves the pan head so that the image capturing surface is continuously directed in a direction in which posture estimation can be performed most stably. You may control.

[3-2. Configuration example 2]
FIG. 7 is a block diagram illustrating a configuration example 2 of the display system according to the second embodiment. As shown in FIG. 7, the display system according to the present embodiment includes a display device 1-3 and an operation device 2-3. The depth information acquisition unit 29 is replaced with the three-dimensional data 4 described with reference to FIG. Have

(1) Depth information acquisition unit 29
The depth information acquisition unit 29 has a function of acquiring depth information in the imaging direction of the imaging unit 21. For example, the depth information acquisition unit 29 may be realized by a stereo camera or an infrared camera. The depth information acquisition unit 29 outputs the acquired depth information to the approach determination unit 27.

The approach determination unit 27 uses the depth information output by the depth information acquisition unit 29 to determine whether the distance between the imaging surface of the imaging unit 21 and the real object is equal to or less than a threshold value. When the approach determining unit 27 determines that the distance between the imaging surface of the imaging unit 21 and the real object is equal to or less than the threshold value, the approach notifying unit 27 notifies the fact to that effect. This notification prompts the user to move the controller device 2-3 away from the real object, thereby avoiding a situation where it is difficult to estimate the posture of the controller device 2-3.

[3-3. Configuration Example 3]
FIG. 8 is a block diagram illustrating a configuration example 3 of the display system according to the second embodiment. As shown in FIG. 8, the display system according to the present embodiment includes a display device 1-4 and an operation device 2-4. Instead of the approach determination unit 27 and the approach notification unit 28 described with reference to FIG. An approach determining unit 18 and an approach notifying unit 19 are provided.

(1) Approach determination unit 18
The approach determination unit 18 has the same function as the approach determination unit 27. Specifically, the approach determination unit 18 determines the distance between the imaging surface of the imaging unit 21 and the real object based on the three-dimensional data 4 and the posture information of the controller device 2-4 received by the posture information receiving unit 14. Is calculated. And the approach determination part 18 determines whether the distance of the imaging surface of the imaging part 21 and a real object is below a threshold value. The approach determination unit 18 outputs the determination result to the approach notification unit 19.

(2) Approach notification unit 19
The approach notification unit 19 has the same function as the approach notification unit 28. In particular, the approach notification unit 19 enables notification on the display device 1-4 side. For example, the approach notification unit 19 can prompt the user to pay more attention by displaying a warning on the display unit 17.

The configuration example of the display system according to the present embodiment has been described above.

<4. Third Embodiment>
In the present embodiment, the display device 1-5 corrects the posture estimation error due to SLAM based on the marker recognition result displayed on the operation device 2-5. First, an overview of the display system according to the present embodiment will be described with reference to FIG.

[4-1. Overview]
FIG. 9 is an explanatory diagram for explaining an overview of a display system according to the third embodiment. FIG. 9 shows a through image captured by the operation device 2-5 and displayed on the display unit 17 of the display device 1-5 according to the present embodiment. As shown in FIG. 9, the operating device 2-5 according to the present embodiment displays the marker 5 made of a white rectangle while displaying black in the entire area of the display. The display device 1-5 according to the present embodiment, based on the recognition result of the marker 5, causes the relative position by integrating the posture information of the display device 1-5 and the operation device 2-5 due to the posture estimation error by SLAM. The estimation error is corrected.

As shown in FIG. 9, the marker 5 faces the display unit 17 of the display device 1-5. Specifically, each square side indicating the marker 5 is projected in parallel with each side of the screen. The operation device 2-5 displays the marker 5 so that the marker 5 faces the screen, that is, so as to face the imaging unit 11 of the display device 1-5. Thereby, since the shape of the marker 5 in the through image is known (fixed), the recognition load of the marker 5 by the display device 1-5 is reduced. In FIG. 9, as an example, the marker 5 is a square, but may take any shape such as a polygon, a circle, and an ellipse.

The overview of the display system according to this embodiment has been described above. Next, a configuration example of the display system according to the present embodiment will be described with reference to FIG.

[4-2. Configuration example]
FIG. 10 is a block diagram illustrating a configuration example of a display system according to the third embodiment. As shown in FIG. 10, the display system according to this embodiment includes a display device 1-5 and an operation device 2-5. In addition to the display system according to the first embodiment described with reference to FIG. , A function for notifying posture information in both directions, and a function for displaying a marker facing each other. A function for notifying posture information bidirectionally is realized by the posture information transmitting unit 141, the posture information receiving unit 142, the posture information transmitting unit 241, and the posture information receiving unit 242. The function for displaying the marker in a face-to-face relationship is realized by the position correction unit 143, the correction matrix transmission unit 144, the correction matrix reception unit 243, the marker generation unit 244, and the marker display unit 245.

(1) Posture information transmission unit 141
The posture information transmission unit 141 has the same function as the posture information transmission unit 24 described above with reference to FIG. The posture information transmission unit 141 according to the present embodiment transmits the posture information of the display device 1-5 estimated by the posture estimation unit 13 to the operation device 2-5.

(2) Attitude information receiving unit 142
The posture information receiving unit 142 has the same function as the posture information receiving unit 14 described above with reference to FIG. The posture information receiving unit 142 according to the present embodiment receives posture information of the controller device 2-5 transmitted by a posture information transmitting unit 241 described later, and outputs the posture information to the position correcting unit 143.

(3) Posture information transmission unit 241
The posture information transmission unit 241 has the same function as the posture information transmission unit 24 described above with reference to FIG. The posture information transmission unit 241 according to the present embodiment transmits the posture information of the operation device 2-5 estimated by the posture estimation unit 23 to the display device 1-5.

(4) Attitude information receiving unit 242
The posture information receiving unit 242 has the same function as the posture information receiving unit 14 described above with reference to FIG. The posture information receiving unit 242 according to the present embodiment receives the posture information of the display device 1-5 transmitted by the posture information transmitting unit 141 and outputs the posture information to the marker generating unit 244.

(5) Position correction unit 143
The position correction unit 143 has a function of recognizing the marker from the through image captured by the imaging unit 11 and calculating the relative position between the display device 1-5 and the operation device 2-5 using the recognition result. Specifically, first, the position correction unit 143 integrates the posture information of the display device 1-5 output by the posture estimation unit 13 and the posture information of the operation device 2-5 received by the posture information reception unit 142, The relative position between the display device 1-5 and the operation device 2-5 is calculated. The arithmetic processing here is the same as the processing in the control unit 16 described above with reference to FIG. In synchronization with this, the imaging unit 11 outputs a through image obtained by imaging a marker displayed by a marker display unit 245 described later. Then, the position correcting unit 143 estimates the position of the controller device 2-5 in the through image by recognizing the marker from the through image. Here, even if a posture estimation error by SLAM is included, the position of the controller device 2-5 in the through image is known by the calculation of the relative position based on the posture information. That is, the position of the marker in the through image is known. Therefore, the position correction unit 143 may recognize a marker from a limited area in the through image. Even if the identification of the marker in the region fails, the position correction unit 143 can efficiently identify the marker by searching the peripheral region. For this reason, the processing load is reduced as compared with the case where the marker is recognized from the entire through image. The position correction unit 143 corrects the calculation result of the relative position by integrating the posture information based on the position estimation result based on the marker.

The position correction unit 143 recognizes the marker using, for example, a Haar-like feature amount. The Haar-like feature amount is a scalar amount obtained as a difference value of the average brightness of the rectangular area. A white region in the black region such as the marker 5 shown in FIG. 9 has a clear brightness difference from the black region, and is easily recognized by the Haar-like feature value. Since the position correction unit 143 only needs to recognize the marker, the processing load is reduced as compared with the case of recognizing the controller device 2-5 itself.

The control unit 16 according to the present embodiment performs various processes according to the relative position between the display device 1-5 and the operation device 2-5 calculated and corrected by the position correction unit 143. For example, the control unit 16 estimates the position of the controller device 2-5 in the through image according to the corrected relative position, and displays the virtual object at a position corresponding to the estimated position. Since the calculation result of the relative position is corrected based on the recognition result of the marker, an error between the estimated position and the actual position is reduced as compared with the case where correction is not performed, and a more natural AR display is realized. .

When the rotational component of the posture estimation error by SLAM is large, the marker is not displayed facing the imaging unit 21, and the marker recognition load by the position correction unit 143 can be great. Therefore, the position correction unit 143 generates a correction matrix for correcting the display control information (shape, angle, and position) of the marker so as to reduce the error of the rotation component. Specifically, the position correction unit 143 generates a correction matrix that corrects a difference from a marker that is assumed to be recognized when facing a marker recognized in the through image. The position correction unit 143 outputs the generated correction matrix to the correction matrix transmission unit 144. Assuming that the offset of the posture estimation error by SLAM is constant, the correction matrix also has a constant value. Therefore, the position correction unit 143 may apply a low-pass filter or the like to the correction matrix. In this case, since the correction width is stable, stable marker display is realized by a marker display unit 245 described later.

(6) Correction matrix transmission unit 144
The correction matrix transmission unit 144 has a function of transmitting information indicating the correction matrix output from the position correction unit 143 to the controller device 2-5. The correction matrix transmission unit 144 is a communication module for transmitting / receiving data to / from the controller device 2-5 by wire / wireless. For example, the correction matrix transmission unit 144 uses a wireless LAN, Wi-Fi (registered trademark), infrared communication, Bluetooth (registered trademark), or the like to perform wireless communication directly with the controller device 2-5 or via a network access point. To do.

(7) Correction matrix receiver 243
The correction matrix reception unit 243 has a function of receiving information indicating the correction matrix transmitted by the correction matrix transmission unit 144. The correction matrix receiving unit 243 is a communication module for transmitting / receiving data to / from the display device 1-5 by wire / wireless. For example, the correction matrix receiving unit 243 wirelessly communicates with the display device 1-5 directly or via a network access point using a wireless LAN, Wi-Fi (registered trademark), infrared communication, Bluetooth (registered trademark), or the like. To do. The correction matrix reception unit 243 outputs the received correction matrix to the marker generation unit 244.

(8) Marker generation unit 244
The marker generation unit 244 generates a marker used for the calculation of the relative position between the display device 1-5 and the operation device 2-5 by the position correction unit 143, and controls the marker display unit 245 to display the generated marker. Functions as a marker control unit. Specifically, first, the marker generation unit 244 integrates the posture information of the operation device 2-5 output by the posture estimation unit 23 and the posture information of the display device 1-5 received by the posture information reception unit 242; The relative position between the display device 1-5 and the operation device 2-5 is calculated. The arithmetic processing here is the same as the processing in the control unit 16 described above with reference to FIG. Next, the marker generation unit 244 generates display control information that defines the shape, angle, and position of the marker facing the display device 1-5 based on the calculation result of the relative position. Then, the marker generation unit 244 outputs the generated display control information to the marker display unit 245, and controls the marker display unit 245 so as to display a marker facing the imaging unit 11 of the display device 1-5. Accordingly, since the marker faces the imaging unit 11 of the display device 1-5, the marker is easily recognized by the position correcting unit 143, and the recognition load is reduced. Note that since the marker is meaningful only after being imaged by the imaging unit 11 of the display device 1-5, the marker generation unit 244 is configured so that the operation device 2-5 is outside the angle of view of the imaging unit 11. The marker display may be stopped. Further, the marker generation unit 244 may reduce the marker generation load by limiting the correction content, such as correcting only the position when the posture estimation error by SLAM is small.

The marker generation unit 244 corrects the shape, position, and angle of the marker so as to face the imaging unit 11 of the display device 1-5 according to the posture estimation error by SLAM. Specifically, the marker display unit 245 is controlled to display a marker facing the imaging unit 11 of the display device 1-5 based further on the marker recognition result by the position correction unit 143 of the display device 1-5. Specifically, the marker generation unit 244 corrects the display control information of the marker generated based on the calculation result of the relative position using the correction matrix received by the correction matrix reception unit 243. As a result, the marker further faces the imaging unit 11 of the display device 1-5, so that the marker recognition load by the position correction unit 143 is further reduced.

The marker generated by the marker generation unit 244 can be considered variously. For example, as shown in FIG. 9, the marker generation unit 244 may generate a rectangle made of a single color with a clear brightness difference from the surrounding area, which can be easily recognized by the Haar-like feature value, as a marker. Good. In addition, the marker generation unit 244 may generate a planar marker with a low detection cost, such as Cyber Code. In this case, even if it is difficult to identify the marker by the Haar-like feature amount, it is possible to identify the marker.

(9) Marker display section 245
The marker display unit 245 has a function of displaying the marker generated by the marker generation unit 244. The marker display unit 245 is realized by, for example, an LCD or an OLED.

(10) Supplement In FIG. 10, the position correction unit 143 is included in the display device 1-5 and the marker generation unit 244 is included in the operation device 2-5. However, the technique according to the present disclosure is not limited thereto. It is not limited. For example, the position correction unit 143 and the marker generation unit 244 may be included in another information processing apparatus connected to the display device 1-5 and the operation device 2-5. In this case, another information processing apparatus may perform marker recognition and relative position calculation to control display on the display unit 17, generate a correction matrix, and control marker display by the marker display unit 245. . Similarly, the position correction unit 143 and the marker generation unit 244 may be included in either the display device 1-5 or the operation device 2-5.

<5. Summary>
So far, the embodiments of the technology according to the present disclosure have been described in detail with reference to FIGS. 1 to 10. According to the above-described embodiment, it is possible to perform processing according to the position of the operating device by the processing device without restricting the positional relationship between the processing device and the operating device. Specifically, the display system according to the present embodiment estimates the postures of the display device 1 and the operation device 2 using the same environment map 3, and estimates their relative positions by integrating them later. For this reason, the display system can estimate the relative position even when the controller device 2 is out of the angle of view of the imaging unit 11 of the display device 1. That is, the display system can estimate the relative position without restricting the positional relationship between the display device 1 and the controller device 2, and can perform various processes according to the estimation result.

Also, since the display device 1 does not need to track the operation device 2 itself by image recognition for position estimation, the load for the relative position estimation processing is reduced. Further, the display device 1 can estimate the relative position more accurately by recognizing the marker displayed on the operation device 2. In addition, since the operation device 2 can display the marker on the display device 1 by calculating the relative position and further using the correction matrix, the marker of the marker by the display device 1 can be displayed. The recognition load is reduced.

In addition, the display device 1 can display a virtual operation auxiliary device and express an interaction with the real world. Since the controller device 2 performs posture estimation by SLAM, posture estimation becomes difficult when the imaging surface of the imaging unit 21 and the real object approach excessively. In this respect, the display device 1 can avoid a situation where it is difficult to estimate the posture of the operation device 2 by displaying a virtual operation auxiliary device. It can represent interactions with real objects.

Further, the display device 1 and the operation device 2 have many of the same or similar functions. For example, posture estimation unit 13 and posture estimation unit 23, posture information transmission unit 141 and posture information transmission unit 241, posture information reception unit 142, and posture information reception unit 242 realize the same or similar functions. For this reason, since the display device 1 and the operation device 2 are interchangeable programmatically, the development efficiency when developing the display system is good.

The preferred embodiments of the present disclosure have been described in detail above with reference to the accompanying drawings, but the technical scope of the present disclosure is not limited to such examples. It is obvious that a person having ordinary knowledge in the technical field of the present disclosure can come up with various changes or modifications within the scope of the technical idea described in the claims. Of course, it is understood that it belongs to the technical scope of the present disclosure.

For example, as has been described from time to time in the description of the embodiment, each component of the display device 1 and the operation device 2 may be included in any of the display device 1, the operation device 2, and other information processing devices. Good. For example, the display device 1 includes the imaging unit 11, the sensor 12, and the display unit 17, the operation device 2 includes the imaging unit 21, the sensor 22, and the user operation acquisition unit 25. You may have another component. In this case, the server may estimate the posture information of the display device 1 and the controller device 2 by the SLAM technology based on the captured image and the sensing result, and may estimate the relative position by integrating them. And the display apparatus 1 may display AR image according to control by a server. Moreover, the controller device 2 may display a marker according to control by the server.

Further, the embodiments of the first to third embodiments of the present disclosure can be appropriately combined.

Note that a series of control processing by each device described in this specification may be realized using any of software, hardware, and a combination of software and hardware. For example, the program constituting the software is stored in advance in a storage medium (non-transitory medium) provided inside or outside each device. Each program is read into a RAM at the time of execution, for example, and executed by a processor such as a CPU.

In addition, the effects described in this specification are merely illustrative or illustrative, and are not limited. That is, the technology according to the present disclosure can exhibit other effects that are apparent to those skilled in the art from the description of the present specification in addition to or instead of the above effects.

The following configurations also belong to the technical scope of the present disclosure.
(1)
1st attitude | position information which shows the position and attitude | position of the operating device which has the said imaging part in the environment map expressing the position of the object which exists in the said real space based on the captured image imaged by the imaging part which images real space An estimation unit for estimating
A communication unit that transmits the first posture information estimated by the estimation unit to a processing device that performs processing according to the position and posture of the controller device;
An information processing apparatus comprising:
(2)
The first posture information is used to calculate a relative position between the operation device and the processing device by integrating with second posture information indicating the position and posture of the processing device in the environment map. The information processing apparatus according to (1).
(3)
The information processing apparatus includes:
The information processing apparatus according to (2), further including a marker control unit that controls a display unit of the operation device to display a marker that is recognized by the processing device and used for the calculation of the relative position.
(4)
The marker control unit is configured to display the marker facing the imaging unit of the processing device based on a calculation result of the relative position obtained by integrating the first posture information and the second posture information. The information processing apparatus according to (3), which controls a unit.
(5)
The information according to (4), wherein the marker control unit controls the display unit to display the marker facing the imaging unit of the processing device based further on a recognition result of the marker by the processing device. Processing equipment.
(6)
The information processing apparatus according to any one of (2) to (5), wherein the first posture information is used by the processing apparatus to display a virtual object at a position corresponding to the relative position.
(7)
The information processing apparatus includes:
The information processing apparatus according to any one of (1) to (6), further including a notification unit that performs notification indicating that a distance between an imaging surface of the imaging unit and a real object is equal to or less than a threshold value.
(8)
The information processing apparatus according to any one of (1) to (7), wherein the estimation unit estimates a first orientation information by selecting a captured image in an imaging direction that allows easy orientation estimation.
(9)
1st attitude | position information which shows the position and attitude | position of the operating device which has the said imaging part in the environment map expressing the position of the object which exists in the said real space based on the captured image imaged by the imaging part which images real space Estimating
Transmitting the estimated first posture information to a processing device that performs processing according to the position and posture of the operating device;
An information processing method executed by a processor of the information processing apparatus.
(10)
Computer
1st attitude | position information which shows the position and attitude | position of the operating device which has the said imaging part in the environment map expressing the position of the object which exists in the said real space based on the captured image imaged by the imaging part which images real space An estimation unit for estimating
A communication unit that transmits the first posture information estimated by the estimation unit to a processing device that performs processing according to the position and posture of the controller device;
Program to function as.

DESCRIPTION OF SYMBOLS 1 Display apparatus 11 Imaging part 12 Sensor 13 Attitude estimation part 14 Attitude information receiving part 141 Attitude information transmission part 142 Attitude information receiving part 143 Position correction part 144 Correction matrix transmission part 15 Control information receiving part 16 Control part 17 Display part 18 Approach determination Unit 19 approach notification unit 2 operation device 21 imaging unit 22 sensor 23 posture estimation unit 24 posture information transmission unit 241 posture information transmission unit 242 posture information reception unit 243 correction matrix reception unit 244 marker generation unit 245 marker display unit 25 user operation acquisition unit 26 control signal transmission unit 27 approach determination unit 28 approach notification unit 29 depth information acquisition unit 3 environment map 4 three-dimensional data 5 marker

Claims

1st attitude | position information which shows the position and attitude | position of the operating device which has the said imaging part in the environment map expressing the position of the object which exists in the said real space based on the captured image imaged by the imaging part which images real space An estimation unit for estimating
A communication unit that transmits the first posture information estimated by the estimation unit to a processing device that performs processing according to the position and posture of the controller device;
An information processing apparatus comprising:
The first posture information is used to calculate a relative position between the operation device and the processing device by integration with second posture information indicating the position and posture of the processing device in the environment map. Item 4. The information processing apparatus according to Item 1.
The information processing apparatus includes:
The information processing apparatus according to claim 2, further comprising a marker control unit that controls a display unit of the operation device to display a marker that is recognized by the processing device and is used for the calculation of the relative position.
The marker control unit is configured to display the marker facing the imaging unit of the processing device based on a calculation result of the relative position obtained by integrating the first posture information and the second posture information. The information processing apparatus according to claim 3, wherein the information processing apparatus controls the unit.
5. The information processing according to claim 4, wherein the marker control unit controls the display unit to display the marker facing the imaging unit of the processing device based further on a recognition result of the marker by the processing device. apparatus.
The information processing apparatus according to claim 2, wherein the first posture information is used by the processing apparatus to display a virtual object at a position corresponding to the relative position.
The information processing apparatus includes:
The information processing apparatus according to claim 1, further comprising a notification unit that performs notification indicating that a distance between an imaging surface of the imaging unit and a real object is equal to or less than a threshold value.
The information processing apparatus according to claim 1, wherein the estimation unit selects a captured image in an imaging direction that is easy to estimate a posture and estimates the first posture information.
1st attitude | position information which shows the position and attitude | position of the operating device which has the said imaging part in the environment map expressing the position of the object which exists in the said real space based on the captured image imaged by the imaging part which images real space Estimating
Transmitting the estimated first posture information to a processing device that performs processing according to the position and posture of the operating device;
An information processing method executed by a processor of the information processing apparatus.
Computer
1st attitude | position information which shows the position and attitude | position of the operating device which has the said imaging part in the environment map expressing the position of the object which exists in the said real space based on the captured image imaged by the imaging part which images real space An estimation unit for estimating
A communication unit that transmits the first posture information estimated by the estimation unit to a processing device that performs processing according to the position and posture of the controller device;
Program to function as.