JP2021064332A - Information processing device, information processing system, and information processing method - Google Patents

Abstract

To easily improve the versatility of a device used for user operation.SOLUTION: A measurement value acquisition unit 52 of an information processing device 10 acquires measured values from a motion sensor 41 of a controller 14 and a motion sensor 44 of an attachment 18. A connection determination unit 54 compares waveforms representing time-variation of the measured values, and when the difference is smaller than a threshold, determines that the controller 14 and the attachment 18 are connected. An information processing unit 58 combines a user operation to the controller 14 and positional information obtained from a pickup image of the attachment 18 to obtain input value and performs a series of information processing.SELECTED DRAWING: Figure 4

Description

The present invention relates to an information processing device, an information processing system, and an information processing method that perform information processing according to a user operation.

With the development of information processing technology, a user interface that can realize advanced processing with more intuitive operations has been put into practical use. For example, a technique of capturing a moving image of a marker attached to a user, analyzing the marker in real time to acquire the movement of the user, and using it as an input value for information processing is used in a wide range of fields. By applying this, a virtual reality can be realized by drawing an image of a virtual world that reflects the movement of the user in a field of view according to the orientation of the face and displaying it on a head-mounted display.

On the other hand, in order to realize more complicated information processing such as electronic games in a virtual world, not only user movements but also controllers and input devices capable of various and detailed inputs are often required. When information processing is performed using a plurality of types of information in this way, it is conceivable to introduce a device capable of obtaining all the information as an input device. However, due to the diversification of information processing content, it is difficult to prepare a device that can handle all of them. In addition, if a new information processing technology is developed and the device is replaced every time the required input information changes, the burden on the user becomes heavy.

The present invention has been made in view of these problems, and an object of the present invention is to provide a technique capable of easily improving the versatility of a device used for user operation.

One aspect of the present invention relates to an information processing device. In this information processing device, the measurement value acquisition unit that acquires the measurement value of the motion sensor provided by each of the plurality of devices is compared with the waveform representing the time change of the measurement value, so that the plurality of devices are combined. It is characterized by including a coupling determination unit for determination and an information processing unit for performing information processing based on a combination of information obtained via each of a plurality of coupled devices.

Another aspect of the present invention relates to an information processing system. This information processing system includes a plurality of devices equipped with motion sensors and an information processing device that acquires input values via a combination of the devices and performs information processing according to the input values. , A coupling determination unit that determines that multiple devices are coupled by comparing the measured value acquisition unit that acquires the measured values of the motion sensors provided by each of the multiple devices with the waveform that represents the time change of the measured values. It is characterized by including an information processing unit that performs information processing based on a combination of information obtained via each of a plurality of coupled devices.

Yet another aspect of the present invention relates to an information processing method. In this information processing method, the information processing device combines a plurality of devices by comparing the step of acquiring the measured value of the motion sensor provided in each of the plurality of devices with the waveform representing the time change of the measured value. It is characterized by including a step of determining the above and a step of performing information processing based on a combination of information obtained via each of a plurality of coupled devices.

It should be noted that any combination of the above components and the conversion of the expression of the present invention between a method, a device, a system, a computer program, a recording medium on which a computer program is recorded, and the like are also effective as aspects of the present invention. ..

According to the present invention, the versatility of the device used for user operation can be easily improved.

It is a figure which illustrates a plurality of devices to be coupled in this embodiment. It is a figure which shows the configuration example of the information processing system to which this embodiment can be applied. It is a figure which shows the internal circuit structure of the information processing apparatus in this embodiment. It is a figure which shows the structure of the functional block of an information processing apparatus, a controller, and an attachment in this embodiment. In this embodiment, it is a figure for demonstrating the process which the waveform comparison part compares the waveform of the measured value of a motion sensor. In this embodiment, it is a figure which shows an example of the screen which is displayed when the user notification part requests a user to give a predetermined motion to a controller. In this embodiment, it is a flowchart showing a processing procedure in which an information processing apparatus performs information processing by combining information related to a controller and an attachment. It is a figure which illustrates the information which can be used as an input value by an information processing unit in this embodiment.

In the present embodiment, information processing is performed by combining a plurality of devices and combining information obtained from each device into an input value. FIG. 1 illustrates a plurality of devices to be coupled in the present embodiment. In this example, the controller 14 and the attachment 18 are coupled as a device for user operation. As shown in (a), the controller 14 and the attachment 18 originally have different housings. By combining it as shown in (b), the user uses the combined body as an input device.

Here, the controller 14 is, for example, a game controller, and includes general operation means such as various operation buttons and a joystick. The attachment 18 includes, for example, a reflecting member or a light emitting body, and serves as a marker for identifying a position in a three-dimensional space by analyzing a photographed image thereof. By combining these, even the controller 14 having a configuration in which position information cannot be obtained can track the position based on the captured image.

By combining a plurality of devices having different functions in this way, complicated processing can be performed by using a large amount of information that cannot be obtained by a single unit as an input value. In addition, by making it removable, it is possible to replace the devices to be combined according to the situation, optimize the information obtained, or use it alone as a lightweight and compact input device. However, if the situation where the device is freely replaced or removed is allowed, information such as whether or not multiple devices are combined as shown in (b), and if so, which devices are combined, etc. It needs to be recognized by the processing device.

For example, when a plurality of users play a game using the combination shown in (b) as an input device, the information obtained from each cannot be combined unless the combination of the plurality of controllers 14 and the plurality of attachments 18 is known. .. For that purpose, for example, it is conceivable that the controller 14 and the attachment 18 are electrically connected, the controller 14 recognizes which attachment 18 is connected, and then the information is transmitted to the information processing apparatus. However, in this case, it is necessary to provide a connector and a communication mechanism for connection, which is disadvantageous in terms of cost.

Further, when the function of the original controller 14 is expanded by the attachment 18, it is necessary to provide a connection mechanism in the controller 14 itself in anticipation of future expansion, and the expandability is limited. Therefore, in the present embodiment, motion sensors are provided in the controller 14 and the attachment 18, and it is determined whether or not the two are combined based on the respective movements obtained from the measured values.

That is, when a predetermined condition that can be regarded as the same movement is satisfied, it is determined that the two movements are combined. This eliminates the need for electrical connections and limits the means of coupling as long as multiple devices can be considered as one rigid body, such as hook-and-loop fasteners, cable ties, snaps, buttons, covers, and screws. .. As a result, the degree of freedom of the connection point is increased, so that design restrictions are reduced, and a highly versatile and expandable input device can be realized at a low manufacturing cost by simple connection.

In the illustrated example, the connection between the controller 14 and the attachment 18 is shown, but the type and number of devices to be combined are not limited to this. That is, as long as each device has a built-in motion sensor and different information can be acquired by using each device, the present embodiment can be applied to any combination of devices. For example, the device may be a controller, a marker, a mobile terminal, a wearable display, or the like.

FIG. 2 shows a configuration example of an information processing system to which the present embodiment can be applied. In this example, as shown in FIG. 1, by attaching the attachment 18 having the mechanism to the controller 14 which does not have the mechanism for detecting the position, the operation content of the controller 14 and the position information thereof are combined to provide information. It is supposed to be used for processing. As a result, for example, when the button press of the controller 14 corresponds to the shooting operation in the virtual reality, the timing and direction of shooting the gun are acquired from the controller 14, and the position is acquired from the attachment 18, so that the bullet hits in the virtual world. It is possible to find the position accurately.

The controller 14 includes at least one of general input means such as a button, a key, a joystick, a touch pad, a camera, and a microphone, and also incorporates a motion sensor. The attachment 18 also has a built-in motion sensor and a mechanism for detecting its position. For example, the attachment 18 may have a mechanism for emitting light in a predetermined wavelength band such as a predetermined color or infrared rays, or may be provided with a member for reflecting the irradiated light in a predetermined wavelength band on the outer surface. Good.

The motion sensor built into the controller 14 and the attachment 18 is a sensor that measures at least one of physical values representing the movement or state of an object such as acceleration, angular velocity, and attitude, and is an IMU (Inertial Measurement Unit) sensor, an acceleration sensor, or the like. Any or combination of practical sensors such as a gyro sensor and a geomagnetic sensor may be used. The information processing system 8 includes a controller 14 to which an attachment 18 is coupled, an image pickup device 12 that captures a real space, an information processing device 10 that performs information processing based on a user operation with respect to the controller 14 and the position of the attachment 18, and information processing. A display device 16 for displaying the resulting image is included.

The information processing device 10 and other devices may be connected by a cable or may be connected by a known wireless communication technology such as Bluetooth (registered trademark). The image pickup apparatus 12 captures an image for acquiring the position of the attachment 18 in the real space at a predetermined frame rate. Therefore, the imaging device 12 includes at least a CCD (Charge Coupled Device) sensor and a CMOS (Complementary Metal Oxide Semiconductor) sensor that detect light from the attachment 18 (light including infrared light and visible light). The image pickup apparatus 12 may be a stereo camera in which two cameras are arranged on the left and right at a known interval as shown in the figure.

In this case, the information processing apparatus 10 extracts the images of the attachment 18 from the images taken at the same time from the left and right viewpoints, and obtains the position of the attachment 18 in the three-dimensional space from the positional relationship. A method of obtaining position information of an object by the principle of triangulation using a stereo image is widely known. However, even if the image pickup device 12 is a monocular camera, if the size of the light emitting portion or the reflecting portion of the attachment 18 and the geometrical positional relationship of the plurality of light emitting portions are known, the three-dimensional space is determined from the size of the image. You will be asked for your position.

Alternatively, by using the image pickup device 12 as a depth camera that irradiates an object with reference light such as infrared rays and captures the reflected light, the information processing device 10 obtains the position of the attachment 18 by TOF (Time of Flight). May be good. As a modification, the information processing device 10 may specify the position by using the image pickup device 12 as a measuring device for a magnetic field or an ultrasonic wave and measuring the magnetic field or the ultrasonic wave generated by the attachment 18. Alternatively, a camera may be provided on the attachment 18 itself, and the information processing apparatus 10 may estimate the position from the captured image by an existing technique such as SLAM (Simultaneous Localization and Mapping).

That is, any of the practical techniques may be adopted for the position acquisition of the attachment 18, and the configuration of the attachment 18 may be various depending on the technique. The information processing device 10 acquires the contents from the controller 14 each time a user operation is performed on the controller 14. As described above, the information processing device 10 further acquires the data of the captured image from the image pickup device 12, analyzes the data, and acquires the position of the attachment 18 in the real space at a predetermined rate.

Then, the information processing device 10 combines the user operation with respect to the controller 14 and the information on the position of the attachment 18, advances the information processing accordingly, and outputs an image or sound representing the result to the display device 16. Here, in order to determine whether the position of the attachment 18 may be regarded as the position of the controller 14, the information processing device 10 determines the connection between the controller 14 and the attachment 18 as an initial process based on the waveform of the measured value of the motion sensor. To do.

The content of information processing performed by the information processing apparatus 10 by combining the user operation obtained from the controller 14 and the position information of the attachment 18 is not particularly limited. Further, in the present embodiment, the types of information used in combination are not limited as long as the functions of the devices determined to be combined are combined to perform information processing. Naturally, the measured value of the motion sensor may be used not only for determining the coupling but also for the subsequent information processing.

The display device 16 may be a television receiver having a general display and a speaker, and may be, for example, a liquid crystal television, an organic EL television, a plasma television, a PC display, or the like. Alternatively, the display device 16 may be a head-mounted display that the user 6 wears on the head to display an image in front of the eyes. At this time, the images for the left eye and the image for the right eye can be viewed stereoscopically by displaying the images for the left eye and the right eye in the left and right regions formed by dividing the display screen into two.

If a motion sensor such as an IMU sensor is mounted on the head-mounted display and the information processing device 10 acquires the measured value to generate a display image of the virtual world in a field of view corresponding to the posture of the head-mounted display, virtual reality Can be realized. The form of each device constituting the information processing system 8 is not limited to the one shown in the figure.

For example, the controller 14 and the display device 16 may be combined to form a head-mounted display, and the attachment 18 may be attached to the head-mounted display. As a result, the movement of the user can be tracked even if the head-mounted display does not have a mechanism for specifying the position. As a result, the information processing device 10 determines the field of view of the displayed image based on the orientation of the user's face specified by the IMU sensor built in the head-mounted display and the position of the user, and provides a more realistic image expression. realizable.

FIG. 3 shows the internal circuit configuration of the information processing device 10. The information processing device 10 includes a CPU (Central Processing Unit) 22, a GPU (Graphics Processing Unit) 24, and a main memory 26. Each of these parts is connected to each other via a bus 30. An input / output interface 28 is further connected to the bus 30. The input / output interface 28 includes a communication unit 32 composed of a peripheral device interface such as USB or IEEE1394, a wired or wireless LAN network interface, a storage unit 34 such as a hard disk drive or a non-volatile memory, and an output for outputting data to an external device. A unit 36, an input unit 38 for inputting data from an external device, and a recording medium driving unit 40 for driving a removable recording medium such as a magnetic disk, an optical disk, or a semiconductor memory are connected.

The CPU 22 controls the entire information processing apparatus 10 by executing the operating system stored in the storage unit 34. The CPU 22 also executes various programs read from the removable recording medium, loaded into the main memory 26, or downloaded via the communication unit 32. The GPU 24 has a geometry engine function and a rendering processor function, performs drawing processing according to a drawing command from the CPU 22, and outputs data from the output unit 36 to the display device 16 and the like. The main memory 26 is composed of a RAM (Random Access Memory) and stores programs and data required for processing.

FIG. 4 shows the configuration of the functional blocks of the information processing device 10, the controller 14, and the attachment 18. Each functional block shown in the figure can be realized by a CPU, GPU, various memories, microprocessors, various elements, operating means, etc. in terms of hardware, and data loaded into memory from a recording medium, etc. in terms of software. It is realized by a program that exerts various functions such as input function, data retention function, image processing function, and communication function. Therefore, it is understood by those skilled in the art that these functional blocks can be realized in various forms by hardware only, software only, or a combination thereof, and is not limited to any of them.

As described above, the controller 14 and the attachment 18 include motion sensors 41 and 44, respectively. The controller 14 shown in the figure further includes an operation button 42 for the user to operate, and the attachment 18 includes a light emitting unit 46 that generates light in a predetermined wavelength band. However, if both can realize different functions as described above, a motion sensor Functional blocks other than are not particularly limited. The information processing device 10 includes an input information acquisition unit 50 that acquires data from the controller 14, the attachment 18, and the image pickup device 12, a combination determination unit 54 that determines the connection between the controller 14 and the attachment 18, and information according to a user operation. It includes an information processing unit 58 that performs processing, and an output unit 60 that outputs image and audio data.

The input information acquisition unit 50 includes a measurement value acquisition unit 52, and sequentially acquires measurement values measured at a predetermined rate from the controller 14 and the attachment 18 by the motion sensors 41 and 44 provided therein. The measurement value acquisition unit 52 continues to acquire the measurement values of the motion sensors 41 and 44 for a predetermined period of time, at least in the phase of determining the coupling between the controller 14 and the attachment 18. Here, the measured value is typically a numerical value representing at least one of three-axis acceleration and angular velocity.

However, the types are not limited as long as they have the same parameters representing the movements of the controller 14 and the attachment 18. During operation of a game or the like, the input information acquisition unit 50 acquires the measured values of the motion sensor 41 or the motion sensor 44 as needed, and sequentially acquires the images captured by the image pickup device 12 at a predetermined frame rate. , The content of the user operation for the operation button 42 is acquired from the controller 14 at any time.

The coupling determination unit 54 determines the coupling between the two by comparing the waveforms representing the time changes of the measured values of the motion sensors 41 and 44. Here, "determination of coupling" means determining whether or not a controller 14 and an attachment 18 are coupled, identifying an attachment 18 coupled to a controller 14, and a plurality of controllers 14 and attachments. At least one of the identification of the conjugate consisting of 18.

Further, the coupling determination unit 54 further compares the waveforms of the measured values of the motion sensors 41 and 44 in the background at all times or at a predetermined timing during the operation of a game or the like, so that the controller 14 and the attachment 18 that have been coupled can be combined. It may be detected that it has come off. Specifically, the combination determination unit 54 includes a user notification unit 56 and a waveform comparison unit 57. The user notification unit 56 requests the user to give the controller 14 a predetermined motion. The request is output from the output unit 60 to the display device 16 in at least one of the forms of character information, voice information, and an image.

For example, the user notification unit 56 requests the user to move the controller 14 so as to draw a circle in the air. When the user executes this, the controller 14 is given a rotational motion, so that the waveform indicated by the measured value of the motion sensor 41 is also unique to represent it. At this time, if the value measured by the motion sensor 44 of the attachment 18 has the same waveform, it can be concluded that the two are combined. By giving a predetermined motion to the controller 14, it is possible to clarify the distinction from the motion of the other controller 14 and the attachment 18, and to align the start points of the waveforms to be compared, so that the accuracy of the coupling determination can be improved.

On the other hand, when the other controller 14 or the attachment 18 does not exist, or when both can be synchronized with high accuracy, the user notification unit 56 may omit the request to the user. As described above, the waveform comparison unit 57 compares the waveforms indicated by the time changes of the measured values of the motion sensors 41 and 44, and determines the connection between the controller 14 and the attachment 18 when a predetermined condition that can be regarded as the movement of one rigid body is satisfied. To do.

As described above, the waveform comparison unit 57 may detect by waveform comparison that the connection between the controller 14 and the attachment 18 is loosened or disconnected as a preprocessing during operation of a game or the like, in addition to determining the connection between the controller 14 and the attachment 18. For example, once the connection is loosened or disconnected by the user swinging the controller 14 determined to be connected to the attachment 18, the position information of the attachment 18 deviates from the actual position of the controller 14, and the accuracy of information processing is high. It is possible to affect the.

Therefore, the waveform comparison unit 57 monitors whether or not the coupling is sustained by comparing the waveforms in the background at all times or at a predetermined timing. When the waveform comparison unit 57 determines that there is a difference in the waveforms and cannot be regarded as one rigid body, the user notification unit 56 notifies the user that the connection has been broken. The notification is also output from the output unit 60 to the display device 16 in at least one of the forms of text information, voice information, and an image. Then, the waveform comparison unit 57 determines the combination of the controller 14 and the attachment 18 by comparing the waveforms after the user has improved the connection state of the controller 14 and the attachment 18 again.

The information processing unit 58 performs information processing such as a game specified by the user. At this time, the information processing unit 58 acquires information related to the combination of the controller 14 and the attachment 18 for which the connection is determined from the combination determination unit 54, and combines the information obtained through the information processing unit 58 for information processing. For example, the information processing unit 58 acquires the identification information of the controller 14 whose combination is determined and the information of the emission color of the attachment 18 from the combination determination unit 54.

Then, the information processing unit 58 extracts the image of the emission color from the captured image to acquire the position information of the attachment 18, and also obtains the content of the user operation for the controller 14 having the corresponding identification information and the measured value of the motion sensor 41. get. Then, the information processing unit 58 combines the information and uses it as an input value for information processing.

As a result, the information processing unit 58 can perform more complicated information processing such as virtual reality by using the position information that is undefined only by the controller 14. Alternatively, when the attachment 18 is attached to the head-mounted display, the information processing unit 58 has a field of view determined by the orientation of the user's face obtained from the IMU sensor built in the head-mounted display and the position information of the attachment, as described above. , A display image may be generated.

The processing by the information processing unit 58 may be stopped while the coupling determination unit 54 does not determine the coupling between the controller 14 and the attachment 18, and while the coupling is once determined but resolved. The output unit 60 acquires the result of information processing such as a game executed by the information processing unit 58 as image or audio data, and outputs the result to the display device 16 at an appropriate timing. The output unit 60 also outputs to the display device 16 a request for giving a predetermined motion to the controller 14 and an output data indicating a notification that the coupling is broken, in accordance with the request from the user notification unit 56.

FIG. 5 is a diagram for explaining a process in which the waveform comparison unit 57 compares the waveforms of the measured values of the motion sensors 41 and 44. In the figure, the acceleration waveforms 64 and 66 of the controller 14 and the attachment 18 are shown as the measured values of the motion sensors 41 and 44. However, the actual measured value is a three-dimensional value, and is obtained at a sampling rate peculiar to the motion sensors 41 and 44. Further, since the measured value is generally output as data expressing the dynamic range with a predetermined number of gradations, the relationship between the numerical value output by the set dynamic range and the actual acceleration changes.

Further, the controller 14 and the attachment 18 operate with their own clocks. Therefore, for example, the waveform comparison unit 57 plots the output values individually transmitted from the controller 14 and the attachment 18 on a common time axis and acceleration axis and interpolates them appropriately, so that the waveform 64 can be compared as shown in the figure. , 66. For example, the waveform comparison unit 57 causes the controller 14 and the attachment 18 to transmit a reply to a signal transmitted from the information processing device 10 with a time stamp together with the time stamp when the signal is received.

As a result, the time difference between the clocks of the information processing device 10 and each device including the transfer time can be acquired, so that the time measured by the two motion sensors 41 and 44 is converted into the value of the clock inside the information processing device 10. Then, each measured value can be represented by a common time axis. Further, if the dynamine cleanliness of the motion sensors 41 and 44 is acquired in advance, the output value can be converted into acceleration and represented by a common axis. If the dynamic ranges of the two motion sensors 41 and 44 are significantly different, the output value of only one of them may be saturated when the acceleration is large, and accurate waveform comparison may not be possible.

Therefore, preferably, the dynamic range is prepared in advance. When it is known that the maximum acceleration is not so large, such as when the controller 14 is used as a head-mounted display, the possibility of saturation is low. Therefore, the acceleration resolution may be improved by narrowing the dynamic range. .. Further, as described above, if a predetermined motion is given to the controller 14 to obtain a waveform significantly different from the motion up to that point, the starting points 68 for comparison can be clearly aligned.

Then, the waveform comparison unit 57 determines, for example, the coupling between the controller 14 and the attachment 18 when the root mean square error of both is smaller than the threshold value. For example, with a motion sensor having a sampling rate of about 2000 Hz, a significant determination can be made in about 1 second. However, there are various algorithms for determining that the difference between the waveforms is small, and thus they are similar, and the waveform comparison unit 57 may make a determination based on any one of them or a combination thereof. For example, the waveform comparison unit 57 may obtain the mutual correlation coefficient or compare the frequency components. Alternatively, the waveform comparison unit 57 may make a determination using DP (Dynamic Programming) matching used in voice recognition or the like.

FIG. 6 shows an example of a screen displayed when the user notification unit 56 requests the user to give a predetermined motion to the controller. The request screen 70 includes a graphic 72 representing the requested movement and character information 74. In this example, assuming a request to move the controller 14 so as to draw a circle in the air, a circular arrow is shown as the figure 72, and the character "Please draw a circle with the controller" is shown as the character information 74. However, the motion to be given is not limited to a circle, and any waveform that can be distinguished from the start point of the waveform to be compared and the motion of the controller 14 or the attachment 18 may be obtained. Further, the information shown on the request screen 70 is not limited to the one shown in the figure.

Next, the operation of the information processing apparatus 10 that can be realized by the above configuration will be described. FIG. 7 is a flowchart showing a processing procedure in which the information processing device 10 performs information processing by combining information related to the controller 14 and the attachment 18 in the present embodiment. This flowchart is started, for example, when a user requests the start of processing via a controller 14 or the like in a state where communication is established between the information processing device 10 and other devices.

Then, the user notification unit 56 of the combination determination unit 54 requests the user to give a predetermined motion to the controller 14 by displaying the screen as shown in FIG. 6 on the display device 16 (S10). On the other hand, the measured value acquisition unit 52 of the input information acquisition unit 50 starts acquiring the measured values from the motion sensors 41 and 44 of the controller 14 and the attachment 18, and the waveform comparison unit 57 of the coupling determination unit 54 has been described with reference to FIG. As described above, the waveform is acquired by representing them on the same time axis and the axis of the measured value (S12).

When the information processing device 10 establishes communication with a plurality of controllers 14 and attachments 18, a waveform is naturally obtained for each of them. When the measurement value acquisition unit 52 obtains the identification information of the transmission source device together with the measurement value, the waveform comparison unit 57 can identify which controller 14 or attachment 18 each waveform belongs to. Then, when the difference between the waveforms of the measured values of the controller 14 and the attachment 18 or the index representing the difference is smaller than the threshold value, the waveform comparison unit 57 determines the combination of the two (Y, S16 in S14).

When a plurality of controllers 14 and attachments 18 are present, the waveform comparison unit 57 identifies the combined controller 14 and attachment 18 by detecting a pair whose waveform difference is smaller than the threshold value. In the determination process of S14, the parameters to be evaluated are not limited as described above as long as it can be confirmed that the difference between the waveforms is substantially smaller than the reference (high similarity). Further, the waveform acquisition of S12 and the determination process of S14 can be performed at the same time by statistical processing.

In any case, when the coupling determination unit 54 determines the coupling, the information processing unit 58 performs information processing by combining the information obtained via the controller 14 and the information obtained via the attachment 18 as an input value. Make progress (S18). For example, the information processing unit 58 combines the user operation with respect to the operation button 42 of the controller 14 and the position information based on the captured image of the light emitting unit 46 of the attachment 18 to be an input value for the game. Further, the information processing unit 58 may combine the measured values of the motion sensor 41 of the controller 14 as input values.

When the connection between the controller 14 and the attachment 18 is confirmed in the background at a predetermined timing, the information processing unit 58 continues the information processing according to the user operation, while the connection determination unit 54 waits for the confirmation until the timing ( N of S20, N of S26, S18). When the confirmation timing arrives, the waveform comparison unit 57 of the coupling determination unit 54 again acquires the waveforms of the measured values of the motion sensors 41 and 44 of the controller 14 and the attachment 18 (Y, S12 of S20).

In this case, since the attachment 18 to be coupled to the controller 14 is limited, the waveform to be compared is also limited. Therefore, the processing in the information processing unit 58 may be continued as it is, and the waveform comparison unit 57 may perform the waveform comparison without giving a predetermined movement to the controller 14. When the waveform comparison unit 57 determines that the coupling is maintained (Y, S16 in S14), the information processing unit 58 continues information processing as it is (S18). After that, the waveform comparison unit 57 confirms the combination of S12 to S16 at a predetermined timing (Y in S20).

If the difference between the waveforms is equal to or greater than the threshold value in the determination process of S14 before the start of information processing by the information processing unit 58 or during the information processing, the waveform comparison unit 57 has the attachment 18 coupled to the controller 14. It is determined that this is not the case (N in S14, S22). Then, the user notification unit 56 of the combination determination unit 54 displays character information such as "Please connect the attachment to the controller" or an image showing the connection method on the display device 16 to indicate that the combination is not performed. Notify the user (S24). If the connection is broken during the information processing by the information processing unit 58, the information processing may be interrupted.

In this case as well, the waveform comparison unit 57 of the coupling determination unit 54 again acquires the waveforms of the measured values of the motion sensors 41 and 44 of the controller 14 and the attachment 18 (S12), and determines the presence or absence of coupling (S14). The user who has improved the coupling state of the controller 14 and the attachment 18 may be made to input to that effect via the controller 14, and the waveform comparison unit 57 may perform the determination process from S12 as an opportunity. Immediately before that, the user notification unit 56 may perform the process of S10 to give the controller 14 a predetermined motion.

Then, when the combination is determined (Y, S16 in S14), the information processing unit 58 combines the information obtained via the controller 14 and the attachment 18 as an input value, and restarts the information processing (S18). If it is determined that they are not yet combined (N, S22 in S14), the user notification unit 56 notifies the user again to that effect (S24). While the process does not need to be terminated due to a user's end request or the like, the above process is continued (N in S26), and when it becomes necessary to terminate the process, all the processes are terminated (Y in S26).

FIG. 8 illustrates information that can be used as an input value by the information processing unit 58. The figure shows information obtained when three players play a game using three controllers a, b, and c and three attachments A, B, and C. Since the information processing apparatus 10 does not know which attachment 18 is coupled to which controller 14 in the initial state, the coupling determination unit 54 compares the waveforms as described above and acquires the coupling relationship.

When the controller 14 is given a predetermined motion to determine the coupling, the controller 14 may be moved one by one to determine the coupling in order, or different motions may be applied at the same time to determine the coupling in parallel. In the illustrated example, it is indicated by a white arrow that the connection between the controller A and the attachment b, the controller B and the attachment c, and the controller C and the attachment a is determined. The information processing unit 58 acquires the information from the combination determination unit 54 and holds it in the internal memory, so that the information related to each device is used in combination.

For example, as shown in the upper part of the figure, the information processing unit 58 extracts the images of the attachments a, b, and c from the image taken by the image pickup apparatus 12, and keeps tracking the respective position information based on the color and the like. However, in reality, the position information is obtained as the position coordinates in the three-dimensional space. On the other hand, the information processing unit 58 detects the timing at which each player presses the operation button 42 with respect to the controllers A, B, and C, and the type of the pressed button. At the bottom of the figure, each operation timing is indicated by a circle.

Then, as shown by the upward arrow, the information processing unit 58 specifies the position of the attachment b at the time t1 when the controller A is operated, and determines the operation content for the game by the combination of the position and the operated button. To do. Similarly, the position of the attachment c at the time t2 when the controller B is operated and the position of the attachment a at the time t3 when the controller C is operated are specified, and by combining them, the operation content for each player's game is determined. To do.

As a result, even if the controller 14 does not require the position information, the information processing that requires the position information can be performed only by connecting the attachment 18 by a simple means. As described above, the information to be combined by combining a plurality of devices is not limited to the one shown in the figure. Further, by combining three or more devices, three or more pieces of information may be combined and used as an input value.

According to the present embodiment described above, the coupling between the devices is determined by comparing the waveforms of the measured values of the motion sensors built in the individual devices. As a result, even if there is no electrical connection, information obtained individually via both can be combined and used as an input value for information processing. As a result, the function expansion of the input device can be realized at low cost. Further, if both can be fixed, the connecting means is not limited, so that it is not necessary to provide a mechanism for connecting, and there is no increase in restrictions on the design.

Since it is not necessary to change the attachment position and shape depending on the dominant hand or the like, the range of application is wide even for controllers and attachments having the same shape, and the manufacturing cost can be further reduced. Furthermore, if various attachments are prepared and replaced according to the information processing to be performed, the original controller only needs to have the minimum necessary functions, so it is necessary to design in anticipation of future expansion. Absent. Furthermore, since it can be flexibly customized according to each user's circumstances, the purchase cost can be suppressed, and the device can be made lighter and smaller.

The present invention has been described above based on the embodiments. The above-described embodiment is an example, and it is understood by those skilled in the art that various modifications are possible for each of these components and combinations of each processing process, and that such modifications are also within the scope of the present invention. is there.

8 Information processing system, 10 Information processing device, 12 Imaging device, 14 Controller, 16 Display device, 18 Attachment, 41 Motion sensor, 42 Operation button, 44 Motion sensor, 46 Light emitting unit, 50 Input information acquisition unit, 52 Measurement value acquisition Unit, 54 coupling judgment unit, 56 user notification unit, 57 waveform comparison unit, 58 information processing unit, 60 output unit.

Claims (11)

A measurement value acquisition unit that acquires the measurement values of motion sensors provided by multiple devices,
A coupling determination unit that determines that the plurality of devices are coupled by comparing waveforms representing changes in the measured values over time.
An information processing unit that performs information processing based on a combination of information obtained via each of a plurality of connected devices,
An information processing device characterized by being equipped with. The information processing apparatus according to claim 1, wherein the coupling determination unit determines that the plurality of devices are coupled when the index representing the difference between the waveforms is smaller than the threshold value. The information processing unit considers the position acquired by one of the plurality of coupled devices to be the position of the other device, so that at least the position information is combined with the other information to input the information processing. The information processing apparatus according to claim 1 or 2. The information processing device according to claim 3, wherein the other device is an input device, and the information processing unit combines the content of a user operation with respect to the input device and the position information into an input value. .. The other device is a head-mounted display, and the information processing unit determines the field of view of the displayed image based on the orientation of the user's face specified by the motion sensor included in the head-mounted display and the position information. The information processing apparatus according to claim 3, wherein the information processing device is characterized by the above. The coupling determination unit requests the user to give a predetermined motion to the device, and compares the waveforms during the period in which the predetermined motion occurs according to any one of claims 1 to 5. The information processing device described. From claim 1, the information processing unit identifies a plurality of coupled devices of the device based on the determination result by the combined determination unit, and combines the obtained information for each coupled body. The information processing apparatus according to any one of 6. The coupling determination unit detects that the plurality of devices are disconnected by comparing the waveforms in the background of information processing by the information processing unit, and notifies the user to that effect. The information processing apparatus according to any one of claims 1 to 7. With multiple devices with motion sensors
It is equipped with an information processing device that acquires an input value via a combination of the devices and performs information processing according to the input value.
The information processing device
A measurement value acquisition unit that acquires the measurement values of motion sensors provided by multiple devices,
A coupling determination unit that determines that the plurality of devices are coupled by comparing waveforms representing changes in the measured values over time.
An information processing unit that performs information processing based on a combination of information obtained via each of a plurality of connected devices,
An information processing system characterized by being equipped with. Steps to acquire the measured values of the motion sensors provided by multiple devices,
A step of determining that the plurality of devices are coupled by comparing waveforms representing the time change of the measured value, and a step of determining that the plurality of devices are coupled.
Steps to perform information processing based on a combination of information obtained through each of multiple connected devices,
An information processing method by an information processing device, which comprises. A function to acquire the measurement value of the motion sensor provided by multiple devices, and
A function for determining that the plurality of devices are connected by comparing waveforms representing changes in the measured values over time, and a function for determining that the plurality of devices are connected.
A function that performs information processing based on a combination of information obtained through each of multiple connected devices, and
A computer program characterized by realizing a computer.

Images