WO2022180894A1

WO2022180894A1 - Tactile-sensation-expansion information processing system, software, method, and storage medium

Info

Publication number: WO2022180894A1
Application number: PCT/JP2021/031776
Authority: WO
Inventors: 友三郎岡▲崎▼
Original assignee: 合同会社Vessk
Priority date: 2021-02-24
Filing date: 2021-08-30
Publication date: 2022-09-01
Also published as: JPWO2022180894A1

Abstract

Provided are a tactile-sensation-expansion information processing system, software, a method, and a storage medium capable of essentially providing a tactile sensation in a real space without using a special additional device connected to a computer. A tactile-sensation-expansion information processing system according to a first embodiment of the present invention comprises: a display unit which is worn by an operator located near a target object having a physical substance, or which is disposed near the operator; a sensor which can detect an action by the operator with respect to the target object, or an action associated with the target object; a display control unit that instructs or specifies content to be displayed to the display unit; and an associative memory formation promotion unit which, on the basis of the detection, as a detected action, of an action performed by the operator with respect to an object in a virtual space corresponding to the target object, calculates virtual position information in the virtual space corresponding to the action, acquires action specification information corresponding to the action on the basis of a predetermined correspondence relationship for the detected action and action specification information corresponding thereto, and causes the operator to form an associative memory between a memory relating to the action specification information and a memory relating to a physical reaction imparted from the target object in response to the action.

Description

Haptic augmented information processing system, software, method and recording medium

The present invention relates to, for example, a tactile augmented information processing system, software, method, and recording medium, and in particular, promotes memory fixation in a tactile simulating manner while eliminating the need to use an additional device connected to a computer that simulates tactile sensation. Haptic augmented information processing system, software, method and recording medium suitable for

Some application software (APP), including games, present images and sounds as presentation content, and add operations to those images and sounds. In conventional computer systems, real devices (operating panels, controllers, handles, wheels, keyboards, mice, tablets, trackpads, etc.) that allow users to actually perform such operations with tactile sensations were used a lot.

Here, the content includes, for example, contents displayed by various (video) games and computer APPs (for example, payroll, factory production system, medical diagnosis support system, musical instrument performance simulation system, driving simulation system, language learning system, etc.) , tourism experience system, etc.).

On the other hand, technologies that use virtual reality (VR) and augmented reality (AR) using head-mounted display units, so-called head-mounted displays (HMDs), have begun to be used in various fields. In this case, the user (wearer) operates the object displayed in the virtual space displayed on the HMD with a hand or finger existing in the real space, thereby allowing the user to view the content presented by the computer. (the wearer) applies an operation (for example, Patent Document 1).

However, in the operation in the HMD, although the correspondence relationship between the objects in the virtual space and the hands and fingers in the real space is built in the HMD, the hands and fingers in the real space move in the void. In addition, even if the object in the virtual space is manipulated, it is not possible to obtain the tactile sensation of manipulating it. had.

In this respect, Patent Document 1, for example, discloses a technical idea regarding a game device capable of appropriately suppressing vibrations such as camera shake that can occur in objects. However, in Patent Document 1, position information of a controller facing a display device is sequentially acquired with respect to arrangement information of an object to be controlled in a virtual space, and displacement amount of the controller is calculated based on the acquired position information. is disclosed, and the displacement amount correction unit performs correction based on this, but there is no focus on the tactile sensation of the operator when operating the object.

On the other hand, the technical idea that the operation part is provided in the virtual space and the operation means having the sense of touch is the keyboard of a musical instrument is disclosed in Patent Document 2, for example. However, Patent Document 2 does not delve into the fundamental idea of simulating the sense of touch, and does not disclose the idea of substantially providing the sense of touch in the real space. In other words, it does not disclose the technical idea of how to establish the technical significance of simulating the sense of touch and how to use the laws of nature regarding this significance to create what kind of value.

JP 2008-113845 A Japanese Patent No. 6419932

In view of the problems in the prior art as described above, the present invention provides a computer or game operation in a virtual space using, for example, an HMD, without using a special additional device connected to the computer. An object of the present invention is to provide a tactile augmented information processing system, software, method, and recording medium capable of substantially providing an internal tactile sensation.

The inventor first started by digging deep into the technical idea of eliminating the need to use an additional device connected to a computer that simulates the sense of touch. In other words, the starting point was to define the technical significance of simulating the sense of touch. As a result of repeated trial and error, it is a natural law that a computer system realizes strengthening the association formation (described later) between visual and/or auditory senses and tactile senses in the human brain, thereby promoting memory consolidation. I arrived at a technical idea using This point will be described in detail below.

First, the inventor focused on the importance of tactile sensation in operation. As a result of various considerations, I came up with the idea of applying the concept of associative memory to this field, especially to the information processing field that utilizes the sense of touch in operation. In other words, associative learning and memory formed by it (associative memory or associative memory) are important for organisms including humans. A simple example of associative learning is the famous Pavlovian dog. In humans, it is believed that associative memory is formed by constructing high-level interrelationships among a large number of memories formed by association (for example, Non-Patent Document 1).

"Association" is supposed to learn the relationship between two different stimuli when given. That is, as an example, when operating a real keyboard, associative learning is established between the content displayed on the display (visual sense) and the button pressing operation (tactile sense). From this point of view, if we define the problem of the above-mentioned conventional technology from the point of view of the present invention, when operating the virtual keyboard displayed on the HMD, it is not possible to obtain the tactile sensation from the button pressing operation, so the associated learning is difficult. The present inventor discovered that the essence of the problem is that it is difficult to establish

Through associative learning, it is possible to acquire advantages such as a stronger memory for two stimuli, a faster response speed, a more reliable response, and the ability to respond unconsciously. In particular, being able to improve the response ability (performance) when the same stimulus is given is an improvement in game skills (for example, the ability to quickly and accurately return a desired action to a predetermined target), computer operation, etc. improvement of typing skills in , or improvement of operation skills of machinery in general (including vehicles such as passenger cars and aircraft, machine tools, construction/tool machines, etc.), improvement of operator's ability to operate in response to changes in display display It is considered to be Developing from this, the present inventor experienced an information system that utilizes the natural law that memory for two stimuli becomes stronger through associative learning. In the information processing field where processing is progressing (especially in the above-mentioned game field, typing field, general manufacturing industry field including machine operation, etc.), the operator's operation ability is improved, or it is possible to contribute to the improvement of the ability. It was sublimated to the creation of the technical idea.

That is, in order to solve the above-described problems, the augmented haptic information processing system according to the first aspect of the present invention is worn by an operator located near an object having a physical substance, or is mounted near the operator. a display unit arranged in the display unit, a sensor capable of detecting an action of the operator on the object or an action associated with the object, and a display control unit that instructs or specifies content to be displayed on the display unit and determining virtual position information in the virtual space corresponding to the motion based on the motion performed by the operator with respect to the object in the virtual space corresponding to the target being detected as the detection motion, and detecting the motion. Acquiring action specifying information corresponding to the action based on a predetermined correspondence relationship between the movement specifying information and the action specifying information corresponding thereto, and storing the physical reaction force applied to the action by the object; and an associated memory formation promoting unit that causes the operator to form an associated memory with the memory related to the action specifying information.

By having the above configuration, when the operator determines an object having a specific physical entity as it and performs a certain action, the sensor detects the action, and then the operator can superimpose the object on the object. Actions (e.g. rightward or leftward arrows) on displayed objects in virtual space (e.g. game controllers, touch panels, discs, control panels, mice, keyboards, pianos, guitars, drums, wheels, handles, levers, sticks) Pressing motion, rotating fingertips, stretching fingers, sticking out fingers, pinching the tips of two or more fingers together, making shapes such as circles and polygons with fingers, gripping hands, hands When the sensor detects the motion, the associative memory formation promoting part detects the motion by opening the hand, stroking the palm, the back of the hand, or the pad of the finger, moving the hand up, down, left, or right, or rotating the hand. , the virtual position information in the virtual space corresponding to the motion is determined, the motion specifying information corresponding to the motion is obtained from a predetermined correspondence table, and the physical physical given to the motion from the object In order to form an associative memory between the reaction force and the acquired action-specifying information, the above action (for example, fingertip rotation action) and the resulting event in the virtual space (virtual space As a result of the formation of associative memory, the operator/user will be able to master certain actions (such as game operations) to objects in the virtual space. .

This mechanism will be further detailed in this application. In other words, from the operator's point of view, compared to the case of performing an input action on an object in a virtual space in the void as in the conventional technology, an object having a physical substance (for example, a desk, The reaction force from the wall, the other hand, etc.) is applied, and the reaction force is instantly transmitted to the brain of the operator. Therefore, the reaction force (strictly, feeling the reaction force) and the output as a result of information processing with the previous input motion as input (for example, a character in the virtual space moves sideways) An associative memory is formed between In this case, when inputting into the void, since there is no reaction force as described above, the one necessary for the formation of associative memory does not exist, and associative memory is not formed. Therefore, compared to the case of empty space, it contributes to steady memory formation. If this is applied to real life, it will lead to effects such as proficiency in game control technology and improvement in keyboard input speed and accuracy. In other words, when one performs a certain action, there is physical pressure and substance in the response to that action, and a memory with an associativity is formed between this substance and the information processing result. By doing so, a structure is created in which one can confirm by oneself that one has performed the action in question. It is something that will be invested.

As a second aspect of the present invention, in the first aspect, the associative memory formation promoting unit performs a specific action performed by the operator on the object or in relation to the object at a real position on the real space. a virtual operation target display for displaying the object defined for the specific action at a virtual position in the virtual space corresponding to the real position on the display unit when the sensor detects that the a control unit, based on the action given by the operator to the object in the real space corresponding to the action, and a virtual position in the virtual space corresponding to the real position information; an operating subject motion detection unit that obtains information and motion specifying information corresponding to the motion; It is good also as what is provided with a thing.

As a third aspect of the present invention, in the first or second aspect, the sensor may have an imaging function.

As a fourth aspect of the present invention, in the first aspect, an imaging unit having an imaging function is further provided, and the associative memory formation promotion unit causes the operator to associate the object with or with the object. When the imaging unit detects that a specific action is performed at a real position on the physical space, the object defined for the specific action is displayed on the virtual space corresponding to the real position on the display unit. a virtual operation target display control unit for displaying at a virtual position of , and real position information where the target object is located in the real space corresponding to the action from the action given to the object by the operator and the real position an operating subject motion detection unit that obtains virtual position information in the virtual space corresponding to the information and motion specifying information corresponding to the motion; and an object that exerts a physical reaction force on the operator.

As a fifth aspect of the present invention, in any one of the first to fourth aspects, the display section may have a see-through mechanism. Here, the see-through mechanism means that, in addition to the function of displaying image data generated by the program, the wearer can see through the physical entity facing the wearer through the display section by providing transparency. It means a thing equipped with a function that enables

As a sixth aspect of the present invention, in any one of the first to fifth aspects, the display unit may be anything as long as it can form a virtual space for the user to recognize, such as a head mount A display or AR goggles are preferable, but a device for displaying an image such as a screen or a display may be arranged between the object and the user.

As a seventh aspect of the present invention, in any one of the first to sixth aspects, the sensor has at least one of an optical sensor, a magnetic sensor, a contact sensor, and a distance image sensor. may be

As an eighth aspect of the present invention, in any one of the first to seventh aspects, the object has elasticity, and the memory related to the physical reaction force has the elasticity may be weighted.

As a ninth aspect of the present invention, in any one of the first to eighth aspects, the object has hardness, elasticity, temperature, texture, etc. that provide a tactile sensation sufficient for forming the associative memory. or a combination thereof. In addition, it may be not only a horizontal surface, but also curved, uneven, or non-horizontal.

As a tenth aspect of the present invention, in any one of the first to ninth aspects, the display further comprises an associative learning detection unit that detects progress of associative learning based on the detected hand or finger movement. The control unit may change the content, which is the operation target, according to the progress of the operation and/or the association learning on the virtual operation target detected by the association learning detection unit.

As an eleventh aspect of the present invention, in any one of the first to ninth aspects, an associative learning detection unit for measuring and estimating the progress of manipulation and/or associative learning with respect to the virtual operation target is further provided. In addition, the display control unit may change the content that is the operation target according to the progress of the operation and/or the association learning on the virtual operation target estimated by the association learning detection unit.

As a twelfth aspect of the present invention, in the eleventh aspect, even if the visual information and/or auditory information related to the content is changed according to the degree of progress of the federated learning estimated by the federated learning detection unit good.

As a thirteenth aspect of the present invention, in the eleventh or twelfth aspect, the estimation of the progress of the associated learning includes a learning curve, a typing error rate, a game winning/losing rate, a degree of progress of a predetermined series of actions, User's movement speed and reaction speed, correct answer rate, accuracy, perfection calculated by algorithm or variables to measure perfection (for example, player's openness in reversi), key press strength, keys At least one of finger shape change information and button pressing time information for determining pressing strength may be included.

As a fourteenth aspect of the present invention, in the third or fourth aspect, the display unit is a display installed in front of the operator's eyes, visual information related to the content is presented on the display, and the imaging is performed. Movement of the operator may be detected by the unit and/or the sensor.

Further, in order to solve the above problems, a method for augmented haptic information processing according to a fifteenth aspect of the present invention includes a display unit and a sensor capable of detecting the action of an operator located near an object having physical substance. and an information processing system comprising the display unit and the control unit for controlling the sensor, wherein the operator's action on the object or in association with the object is detected as a detection action. a first step, and a second step in which the control unit identifies virtual position information in a virtual space corresponding to real position information in the real space related to the detected motion and associated with the object; a third step of acquiring motion specifying information corresponding to said motion based on a predetermined correspondence relationship between a sensed motion and motion specifying information corresponding to said motion; and a fourth step of causing the operator to form an associative memory between the memory of the physical reaction force applied and the memory of the action specifying information.

As a sixteenth aspect of the present invention, in the fifteenth aspect, in the fourth step, the controller adjusts the size of the object in the virtual space corresponding to the specific action, and adjusts the size of the object after the adjustment. the rear object is displayed on the display unit at a position identified by the virtual position information in the virtual space and superimposed on the real position information in the real space associated with the object; is detected as a motion given by the operator to the object in the virtual space, and the type of the detected motion and the real position information on the physical space related to the motion is specified by the control unit, and the object gives a physical reaction force against the motion to the operator at a position in the real space corresponding to the object in the virtual space, and the detected motion The control unit finds out the real position information on the physical space and identifies it with the virtual position information on the virtual space. The motion specifying information and the virtual position information corresponding to the motion are specified, and the first memory in which the operator perceives the physical reaction force given by the object to the motion and the motion specifying information are stored as the The operator may form an associative memory with the second memory perceived by the operator.

Furthermore, in order to solve the above problems, a tactile augmented information processing method according to a seventeenth aspect of the present invention includes a display unit and a sensor capable of detecting the action of an operator located near an object having physical substance. and a control unit for controlling the display unit and the sensor, a first step of displaying content to be operated on the display unit; a second step of detecting spatial information about an entity with a certain physical entity; a third step of setting a virtual plane at the position detected in the second step; a fourth step of setting a physically operable range on the set virtual plane; and virtual buttons, game controllers, touch panels, discs, and operations within the operable range set in the fourth step. a fifth step of displaying a virtual operation target including at least one of a board, a mouse, a keyboard, a musical instrument, a wheel, a handle, a lever, and a stick in the virtual space; A sixth step of detecting the operation while applying a reaction force in real space to the operation of the virtual operation target by the 7, an eighth step of detecting progress of association learning based on the movement of the operation instructing subject detected in the seventh step, and an operation on the virtual operation target detected in the eighth step. and/or a ninth step of changing the content to be operated according to the progress of the federated learning.

Furthermore, in order to solve the above problems, a haptic augmented information processing program according to an eighteenth aspect of the present invention is capable of detecting a motion of an operator located near a display unit and an object having physical substance. In an information processing system comprising a sensor, and a control unit for controlling the display unit and the sensor, the sensor is used to notify the computer that the operator has performed an action on the object or in association with the object. a first means for causing the sensor to detect the motion as a detection motion; and the control unit transmitting virtual position information in the virtual space corresponding to the real position information in the real space associated with the object related to the detection motion detected by the sensor. a third means for acquiring action specifying information corresponding to the action based on a predetermined correspondence relationship between the detected action and the action specifying information corresponding thereto; and On the other hand, in order to cause the operator to form an associative memory between the memory related to the physical reaction force given by the object and the memory related to the action specifying information, substantially the same as when the physical reaction force is given. and fourth means for presenting to the operator the action specifying information or information obtained by processing the action specifying information at a given timing.

As a 19th aspect of the present invention, in the 18th aspect, the fourth means further instructs the computer, and further, the control unit to adjust the size of the object in the virtual space corresponding to the specific action. The adjusted object after adjustment is displayed on the display unit at a position identified by the virtual position information in the virtual space and superimposed on the real position information in the real space associated with the object. a motion given by the operator to the real object is detected as a motion given by the operator to the object in the virtual space, and a type of the detected motion and a reality related to the motion; the control unit identifies real position information in space, and the object gives a physical reaction force to the action to the operator at a position in the real space corresponding to the object in the virtual space; After the controller determines the real position information on the physical space related to the detected motion and identifies it with the virtual position information on the virtual space, the motion corresponding to the type of the detected motion is performed. a first memory in which the operator perceives the physical reaction force given by the object to the motion, and the motion specifying information related to the type of motion and the virtual position information corresponding to the motion as a first memory; the action specifying information at approximately the same timing as the physical reaction force is applied so as to cause the operator to form an associated memory between the action specifying information and a second memory perceived by the operator; Alternatively, the information obtained by processing the action specifying information may be presented to the operator.

A twentieth aspect of the present invention can be implemented as a recording medium storing the augmented haptic information processing program according to the eighteenth or nineteenth aspect.

In the above, the display unit refers to a unit that allows the wearer or a person in the vicinity to form a virtual space in terms of recognition by displaying content. It is preferable to have a see-through mechanism and be able to display the supplied image in front of both eyes. It may be a device that can be viewed by opening or being transparent, or a device that displays an image such as a screen or a display that is simply arranged between the object and the user. may be

A sensor that detects an object can detect the outline of a fixed object or the movement of a moving object. A unit (for example, a TV camera) is preferable, but other magnetic sensors, contact sensors, myoelectric sensors, acceleration sensors, or the like may be used, or any combination of the above sensors may be used. Further, the sensor may include a distance image sensor capable of obtaining a distance image by outputting the distance from the photographed image of the subject to the object in units of pixels.

In the above case, the sensor (for example, a TV camera, etc.) captures an image of the outside world, acquires or processes distance information, acceleration information, etc., and uses the image display control unit or directly, head-mounted type It is configured so that it can be displayed on the display unit and the captured image can be stored or transmitted. may be placed.

The display unit and sensor described above may be implemented as, for example, a head mounted display (HMD). In this case, the HMD preferably includes at least a display and a sensor (which may include an imaging unit (for example, a TV camera)), and more preferably includes a communication interface (I/F). Configured.

All visual information required by the operator is displayed on the display of the HMD, information outside the HMD to be sent to the operation control unit shown below is imaged by the TV camera and sensor, and the communication I / F Signals are exchanged between the HMD and the motion control section.

In terms of hardware, the display control unit includes a communication I/F for exchanging signals with the display unit (for example, the display unit of the HMD when implemented as the HMD described above), a CPU for performing calculations, and a numerical calculation unit. It is configured with a GPU that performs high-speed processing, a memory in which the program and data being executed are placed, and an external storage device that saves the program and data. It comprises a program for controlling at least one of the CPU, GPU, communication I/F, and external storage device so as to transmit and instruct content information to be processed.

On the other hand, there is a substantially flat surface outside the HMD, on which the operator operates the keyboard, buttons, musical instruments, operation panel, mouse, touch panel, levers, etc. The substantially flat portion is preferably a substantially horizontal surface of an object such as a desk, table, wooden box, cardboard box, flat plate, floor, sand, soil, water surface, cushion, chair, etc. Regardless of the type, it includes not only horizontal surfaces but also slightly inclined surfaces, surfaces with small steps and unevenness on the surface, and slightly curved surfaces.In addition to horizontal surfaces, walls and box-shaped surfaces It may be a side surface of an object, or a vertical or near-vertical surface such as one's own or another person's hand, leg, belly, or other person's back. Furthermore, an obstacle or the like may be installed within a range that does not interfere with the operation.

The subject of operation instruction refers to a part of the body such as the fingers of the person who uses this system, or the operation unit held by the user, and the place where they are placed is detected and recognized by the sensor and the image display control unit. It is also an object capable of causing a sensor to detect motions thereof, that is, motions in the direction perpendicular to the substantially flat portion, or motions that move the substantially flat portion in the plane direction.

In the above hardware configuration, a sensor (e.g., a TV camera) built in, attached to, or placed near the HMD recognizes or images a substantially flat surface (desk, etc.) in front of the operator, and the operator's fingers, hands, A virtual operation range is set by specifying a plurality of points on a substantially plane existing in the real space using a controller or an AR marker or the like placed in the real space.

On the screen of the HMD, the operator's fingers and hands are displayed in addition to the virtual keyboard, virtual buttons, game images to be operated, etc. within the virtual operation range set in this way. The operator operates the virtual keyboard and virtual buttons on the screen of this HMD, but since the reality is a substantially flat surface that actually exists in the real space, the tactile feedback from it can be obtained without wearing a special device. , can be received as a realistic feeling, and by this, associative learning between visual sense and tactile sense such as game images can be performed, and associative memory can be constructed.

According to each of the above aspects of the present invention, an existing keyboard (for forming an associative memory) becomes unnecessary, freeing space on the desk and allowing the keyboard to coexist with paper such as documents. the problem is resolved. In addition, it is possible to set a virtual keyboard in a place where papers exist, which significantly improves work efficiency and space efficiency.

Further, based on the technical idea of the present invention, it is possible to estimate the degree of progress of learning in the virtual buttons and the virtual keyboard. Not only does it promote the improvement of skills, but it also makes it possible to provide complex games that have not been possible before.

In the above case, virtual buttons and virtual keyboards can be placed anywhere within the virtual space's operating range, and even dynamically changed by the program. This makes it possible to implement even more complex games. For example, it is possible to change the controller depending on the operator's situation. This has been difficult to achieve with conventional game experience equipment with physical controllers. Furthermore, it is possible to change or adjust not only virtual buttons and virtual keyboards, but also all other virtual operation objects, and even the appearance and size of the user's hands (for example, animal hands, robot hands, etc.). , translucent hands, etc.).

In the above description, keyboards and buttons were mainly used as objects to be operated by the subject of operation instruction, but there are other objects such as sticks, levers, sliders, game controllers, touch panels, discs, wheels, handles, control panels, A mouse, a keyboard, a stringed instrument, a percussion instrument, or any other instrument that can be operated by an operator and whose movement can be detected by a TV camera or sensor can be used.

Furthermore, by creating a different virtual controller for each APP in software and changing the content, it is possible to operate content with a sense of touch without replacing different hardware for each APP, reducing costs and increasing convenience. performance improvement is realized.

In the haptic augmented information processing system, method, program, and storage medium according to aspects of the present invention, for example, when operating a computer or game in a virtual space using an HMD or the like, without using a special additional device connected to the computer, Or, while reducing the number of additional devices, substantially providing tactile sensations in the real space, thereby promoting associative learning of visual and/or auditory and tactile sensations, and forming strong associative memories. becomes possible. Furthermore, it is also possible to monitor the progress of associative learning by detecting movements of fingers and hands. That is, it is possible to provide tactile sensations generated by pressing keyboards and buttons in a virtual space including HMD, thereby promoting associative learning between visual information and/or auditory information and tactile sensations, and forming strong associative memory. In addition, by monitoring the progress of learning, it is possible to further promote associative learning and form strong associative memory.

Furthermore, there is no need for a separate additional device that was previously required to give the operator a sense of touch. Alternatively, the number of additional devices can be reduced or the additional devices can be changed to simple ones. As a result, compared to the conventional situation in which a different controller was prepared for each game, there is an extremely large effect of improving flexibility and convenience and reducing costs.

Furthermore, not only can the virtual buttons and virtual keyboards be freely arranged and sized within the virtual space, but they can also be dynamically changed by a program without the operator's permission. program can be provided.

In addition, by forming a virtual object on the flat surface of an object such as a desk that may become an obstacle in the virtual space, the user does not need to move much, and the risk of colliding with surrounding furniture is reduced. . Furthermore, by designating the operation range within the plane portion, the risk of hitting unintended obstacles (such as a coffee cup placed on the edge of the desk surface) is likewise reduced.

Furthermore, a virtual touch display can be used as a virtual controller. In this case, the user can use a large size touch display in the virtual space, which in reality is very expensive.

In addition, it is now possible to change the size of the virtual touch display, which was difficult with conventional smartphones and tablets. freely adjustable.

For example, it is possible to adjust the size of the virtual keyboard to an optimal size that exceeds the restrictions of the display size according to the size of the user's hand, etc., and it is also possible to arrange a large number of keys that exceed the physical restrictions. can.

The operability of the virtual touch panel and virtual keyboard, as well as the overall user interface in the virtual space, and the novelty and inventiveness of the present invention will be described in more detail below. A virtual keyboard in conventional virtual or mixed reality software is a character that selects keys floating in the virtual space one by one with a laser (virtual laser pointer) that extends into the virtual space from the controller held by the user. Input method was the mainstream. However, in reality, operations performed by tapping the screen or keyboard with 10 fingers are performed with only two controllers on both hands, so simultaneous pressing of 3 or more keys cannot be performed, and input speed is slow. was a problem.

In an attempt to solve this problem, there was an attempt to reproduce objects that mimic real keyboards and touch panels in virtual space. In addition to being difficult to use, there were frequent erroneous selections (misoperations) such as neighboring keys and other UI buttons reacting at the same time. In order to reduce this erroneous operation, the user interface in the virtual space accepts input only when the panel floating in the air is touched continuously for a certain period of time (or the user presses the button of the external controller while touching the panel). Although many are also taken, this ended up sacrificing the input speed.

According to the present invention, it is possible to give the user's "collision determination" to the virtual keyboard or virtual touch panel a "collision" involving the substance of a real, substantially flat surface. As a result, the above-mentioned poor operability due to the fact that it does not involve the entity in the real space, which has been a problem with virtual reality or mixed reality software that has been common so far, can be solved without special additional equipment. can be resolved to

In addition, operations such as pushing and stroking real entities reduce "blurring" compared to performing similar operations in the air with no physical resistance. This reduction in blurring not only reduces erroneous operations, but also enables detailed operation information such as "key pressing time" and "direction moved after touching a key" to be acquired in a state close to the user's intention. This makes it easier to create a keyboard that allows you to change the appearance and behavior of keys with long presses, which was difficult or expensive with conventional physical keyboards.

In addition, by developing methods such as "swipe operation", "flick input", "glide input", etc., which are known as (character) input methods on touch displays, complex and dynamic input methods that exceed the screen size restrictions of smartphones and tablets "Swipe operation", "flick input" and "glide input" that require operation are also possible in the present invention.

Furthermore, the advantage of the virtual touch display in the present invention is added. This is because the number of simultaneous touches that can be performed on a real touch display differs depending on the performance of the pressure-sensitive sensor and the capacitive sensor, and the price and operability may differ accordingly. Multiple simultaneous touch processing (multiple collision processing) is no longer related to such sensor restrictions, and is replaced by the number of collision determination (object distance calculation) processing on software, so the number of simultaneous touches can be said to be virtually unlimited. From this point of view, it can be said that it will be possible to provide more complex operations and deeper experiences than ever before.

Of course, multiple virtual touch displays can be arranged, and they can coexist with virtual buttons and virtual controllers.

1 is a configuration diagram mainly from the hardware side of a haptic augmented information processing system (for example, an image display system) according to an embodiment of the present invention; FIG. 1 is a functional configuration diagram from the software side that configures an image display system according to an embodiment of the present invention; FIG. FIG. 3 is a functional block diagram obtained by adding the configuration of an associated memory formation promoting unit 30 to the functional configuration diagram (functional block diagram) according to FIG. 2 ; FIG. 4 is a conceptual block diagram showing an enlarged associative memory formation promoting unit 30 in FIG. 3 and conceptually illustrating how an associative memory is formed; FIG. 4 is a conceptual block diagram showing an enlarged associative memory formation promoting unit 30 in FIG. 3 and conceptually illustrating how an associative memory is formed; 4 is a timing flowchart for explaining an operation defining a specific operation according to an embodiment of the present invention; FIG. 4 is a conceptual perspective view for explaining actual movements of an operator in specific actions according to one embodiment of the present invention; FIG. 4 is a conceptual perspective view for explaining actual movements of an operator in specific actions according to one embodiment of the present invention; FIG. 4 is a conceptual perspective view for explaining actual movements of an operator in specific actions according to one embodiment of the present invention; FIG. 4 is a conceptual perspective view for explaining actual movements of an operator in specific actions according to one embodiment of the present invention; FIG. 4 is a conceptual diagram for explaining a learning progress estimation method according to an embodiment of the present invention; 4 is a flowchart for explaining the operation of the image display system according to one embodiment of the present invention;

A haptic augmented information processing system (for example, an image display system) according to the first embodiment of the present invention will be described below with reference to the drawings. In the following, the range necessary for the description to achieve the object of the present invention is schematically shown, and the range necessary for the description of the relevant part of the present invention is mainly described. It shall be based on a well-known technique.

FIG. 1 is an overall hardware configuration diagram of a haptic augmented information processing system (for example, an image display system) 100 according to an embodiment of the present invention. It is a view in which a displayed image and a conceptual perspective view for explaining a range captured by a TV camera, which is one element of the same hardware, are arranged around, and in the same figure, the case where an HMD is used is exemplified. showing. As shown in the figure, the image display system 100 includes, in terms of hardware configuration, a display 6, a TV camera 7, a sensor 8, and an HMD 5 equipped with or incorporated with a communication I/F (interface) 9; It comprises an operation control unit 21 having a communication I/F 16, a CPU 17, a GPU 18, a memory 19, and an external storage device 20, which are connected to the HMD 5 via a communication I/F 9. An operation range on a substantially planar desk 1 (as an entity) is specified, and a virtual keyboard 2, a virtual button 3, and an operation instructing subject 4 (as an entity) (both hands are shown as an example) ) are shown. An operation instructing subject 4 (as an entity) can wear an HMD (head mounted display) 5 (as an entity). Needless to say, the virtual keyboard 2 and the virtual buttons 3 are virtual entities from the user's (wearer's) point of view. When the user (wearer) recognizes the electronic entity displayed as video data on the display, which is one piece of hardware, by operating another entity called wear, the entity appears as if it exists in the void. It is an object that can be recognized as if. Details of these technical features will be described later.

The situation in front of the operator 4 including the desk 1 is imaged by the TV camera 7 attached to the HMD 5 and displayed on the display 6 of the HMD 5. In this case, the display 6 preferably has a see-through function. When the display 6 has a see-through function, the display image displayed on the display 6 may be a see-through real image in which the front real space is seen through the see-through function, or may be an image captured by the TV camera 7. It may be a superimposed image obtained by superimposing a captured image displayed on the display 6 so as to be superimposed on the real space based on the obtained image data and the see-through real image. If the display 6 does not have a see-through function, the display image displayed on the display 6 may be image data captured by the TV camera 7 or processed data thereof. The substantially flat surface (which is the surface of the desk 1) is detected from the image captured by the TV camera 7, but alternatively may be detected by a sensor 8 with imaging capability. These signals, that is, information related to moving images and/or still images captured by the TV camera 7 (or the sensor 8) and detection information detected by the sensor 8 are controlled via the communication I/F 9. sent to section 21.

An example of the screen displayed on the display 6 of the HMD 5 is shown as screen 10 (in FIG. 1). A screen (strictly speaking, an image appearing in a virtual space) 11 of an APP being executed (here, for example, a game image is exemplified) is displayed, and an operator's operation generated from a substantially plane is displayed. A virtual plane 12 is also displayed. An operation range is designated on this virtual plane 12, and a virtual keyboard 13 and virtual buttons 14 are displayed within the range, and furthermore, an operation instruction subject (both hands are shown as an example) that overlaps the virtual keyboard 13 and the virtual button ) on the real space is displayed. The mode in which the image 15 of the main subject of the operation instruction is displayed may be a real image (the above-mentioned see-through real image) captured visually by the wearer by the see-through mechanism of the HMD 5, or (the see-through function is absent or turned off). Instead of a real image, for example, a virtualized image (the above-described captured image or Alternatively, it may be an image in which a real image perceived by the wearer's eyes and the virtualized image are superimposed (the superimposed image described above). may

Screen data related to the screen 10 displayed on the display 6 of the HMD 5 is generated by the operation control unit 21, sent to the communication I/F 9 of the HMD 5 through the communication I/F 16, and displayed. More specifically, for example, the CPU 17 or GPU 18 reads a certain program pre-stored in the external storage device 20 all at once or each time, stores it in the memory 19, and the CPU 17 or GPU 18 stores it in the memory 19. The screen data can be generated by reading the stored program and operating the corresponding hardware according to the program.

As described above, the operation control unit 21 includes the CPU 17, the GPU 18, the memory 19, and the external storage device 20, and preferably also includes the communication I/F 16, but may be configured with additional necessary devices and components. may be

FIG. 2 is a functional configuration diagram (functional block diagram) from the software aspect constituting the image display system according to one embodiment of the present invention. FIG. 10 is a functional configuration diagram showing a configuration of a part particularly related to motion of an operation instructing subject and progress estimation of association learning. In other words, functions constituting the software constituting the image display system may be other than those shown in FIG. , which may be omitted here. As shown in FIG. 2, as a hardware configuration, the motion control unit 21 (shown in FIG. 1) includes, in terms of functional blocks, a space detection unit 2122, an operation range setting unit 2123, an operation detection unit 2124, and an operation instruction subject. , a motion detection unit 2125, a virtual operation target display control unit 2126, a display control unit 2127, a content display control unit 2128, and an associated learning progress detection unit 2129. Note that the federated learning progress detector 2129 may be optional.

The image signal (signal relating to the moving image information and/or the still image information) sent from the TV camera 7 of the HMD 5 is preprocessed by an algorithm described later. It is detected by the detection unit 2122 . Alternatively, the object detected by the space detection unit 2122 may be a space (three-dimensional object) such as a three-dimensional object. In any case, the space detection unit 2122 can specify which part of the image signal captured by the TV camera 7 forms a substantially flat surface. The space detection unit 2122 operates the hardware of the TV camera 7 and/or the sensor 8 and the hardware to detect a plane by a known plane detection algorithm based on the information obtained from the TV camera 7 and/or the sensor 8. or a program (software) that causes computer resources to perform a function of detecting space by a known space detection algorithm. That is, the space detection unit 2122 performs, for example, a plane detection method (for example, Yamato Ideoka, Masuda Nitta, Kiyotaka Kato: "Face detection in range image using three-dimensional Hough transform" (Proceedings of the Japan Society for Precision Engineering Autumn Meeting Scientific Lecture Proceedings (2011)), but it is not limited to this.

The operation range is determined by the operation range designating unit 2123, whereby the subject of the operation instruction in the real space is associated with the virtual space, for example, as position information. This set of software components prepares the APP for operation. When the operation range specifying unit 2123 determines the operation range, it may be specified by the operator, or may be physically specified in advance in the real space (for example, a marker in advance in the real space (for example, an AR The operation range may be specified by placing a marker)). For example, in the case where the operation range is determined by being specified by the operator, the operation range specifying unit 2123 operates the (external) hardware such as the TV camera 7 and/or the sensor 8 to (Internal) computer resources (communication I/F 16, CPU 17, GPU 18) have a function of acquiring physical position information of a designated operation range and correlating (mapping processing) the acquired position information in the virtual space. , memory 19, and/or external storage device 20) and the above-mentioned hardware cooperate with each other.

After that, an operation is performed by the operation instruction subject according to the activated APP, and the operation is detected by the operation detection unit 2124 and passed to the APP. This gives the APP the same effect as operating a real keyboard or real button. The operation detection unit 2124 operates hardware such as the TV camera 7 and/or the sensor 8, and detects an action, that is, an operation by an operation instructing subject obtained from the TV camera 7 and/or the sensor 8, for example, based on position information ( The algorithm (software) that causes the computer resources to perform the function of detecting the operation by specifying the operation related to the operation through various information obtained as a result of the operation, including location information. It is realized by cooperating with hardware.

The motion of the operation instruction subject is detected by the motion detection unit 2125 of the operation subject. The objects to be detected here include reaction time, reaction speed, time interval between key touches, key pressing strength, right or left arrow pressing, finger rotation, finger extension, finger thrust, and two or more fingers. Pinch the tips of the fingers together, create shapes such as circles and polygons with your fingers, hold your hands, open your hands, close your fingers, open your fingers, and rub your palms, backs of your hands, and pad of your fingers. At least one of a motion, a motion of moving a hand up, down, left and right, and a motion of rotating a hand can be included, and the reaction time, reaction speed, and key touch detected in relation to the specific motion can be included. Time interval, key press intensity, learning curve, typing error rate, game win/loss rate, progress of a series of prescribed actions, user's action speed and reaction speed, correct answer rate, accuracy, completeness calculated by algorithm Alternatively, it includes at least one of specific operation information such as a variable for determining the degree of perfection (for example, the degree of openness of the user's stroke in reversi) and finger shape change information for determining the key pressing strength. Also, the detection can include detection for estimating the degree of progress of associated learning, but the detection function of the motion detection unit 2125 of the operating subject is not limited to these. The motion detection unit 2125 of the operating subject operates hardware such as the TV camera 7 and/or the sensor 8, and detects an action of an operation by the operating instruction subject obtained from the TV camera 7 and/or the sensor 8, for example, as an action related to the operation. It is detected as position information or the like, or as position information, time information, and pressure information obtained by adding pressing force information corresponding to the key pressing strength to these, and the detected information thus detected is used for each operation (for example, reaction, Action information for each key touch, key press, slider slide, touch panel flick, wheel rotation, action such as playing a musical instrument, etc. Algorithms (software) that cause computer resources to automatically identify as information such as movement proficiency, movement accuracy, movement perfection, degree of divergence from ideal movement, and the above hardware are realized by working together. In order to automatically identify the motion information based on the detected information in this case, for example, a certain threshold is set for each type of detected information for each motion corresponding to each motion information, and the actually detected detection information However, it is not limited to this, and "based on the detected information, automatically by sorting into predefined motion information for each detected value Any known algorithm that implements the function of identifying corresponding operational information may be used.

Next, the "specific action" according to one embodiment of the present invention will be described above. A specific action indicates a specific action predefined on the system. For example, let's take the case where the specific action is defined as the action of "touching and stroking an object to form a circle". FIG. 6 is a timing flowchart for explaining the operation that defines the specific operation according to one embodiment of the present invention, and FIGS. is a perspective view. As shown in FIG. 7, when an operator performs a specific action on a specific object (for example, the object "top surface of a desk" is substantially circled about a specific first position P1 as the center). (Step 301). Then, the motion detection unit 2125 of the operating subject, based on the detected detection information described above, uses, for example, the following algorithm to systematically determine that an action of "stroking to form a circle", that is, a specific action, has been performed. (Step 302), the center point of the circle related to this specific action is determined using a known algorithm, and the determined center point of the circle, i.e., the specific first position P1, of the virtual controller (in the virtual space ) identifies the controller first position Z1 (FIG. 8) and displays it (step 303). When this is repeated (steps 300B, 304) by, for example, a predefined number N (for example, "4" here) as the number of positions related to the specific motion, the actual specific first position P1 ~ Controller first position Z1 (in virtual space) of the virtual controller corresponding to a specific Nth position PN (N may be 4, for example) ~ Controller Nth position ZN (N may be 4, for example) ) is specified (identified), and controller first positions Z1 (in virtual space) of the virtual controller to ( For example, the controller fourth position Z4 and a virtual plane (virtual plane 12 in FIG. 1) defined by Z1 to Z4 are displayed in the virtual space by the virtual operation target display control unit 2126 (FIG. 9). .

The algorithm for determining whether a specific action has been performed from the detection information described above can be realized, for example, by the following, but is not limited to this. In other words, the virtual operation target display control unit 2126 compares the position information related to the action of “stroking to form a circle” with the captured image in the real world, so that the “stroking to form a circle” is performed. "Specify the real-world object (real-world object: for example, the desk surface in front of the wearer) that has been stroked (physically by the wearer in the real world, for example) to form a circle, that is, the real-world It is possible to identify the virtual position information of the object in the virtual world (associating the real object with the position information in the virtual world) (step 303).

Next, the explanation of the operation after step 304 will be continued. As shown in FIG. 9, the controller first position Z1 to controller Nth position ZN (N may be 4, for example) of the virtual controller (in the virtual space) are specified (identified) in the virtual space. , or after such display, the virtual operation target display control unit 2126 superimposes a specific function object having a specific function (for example, a game controller or a keyboard) on the real object. And the size is displayed as a virtual world (step 305). Specifically, for example, if the specific function object is a game controller, the controller first position Z1 to ( For example, the controller 4th position Z4 is identified, and a plane controller having the controller 1st position Z1 to the controller 4th position Z4 defined in this way, for example, as the four corners is identified and arranged in the virtual space, and displayed on the display 6. (Fig. 10). At this time, if the specific functional object is a game controller, for example, operation buttons B1 to B4 (corresponding to the virtual keyboard 13 and the virtual button 14 in FIG. 1) are defined, and each operation button is displayed in the virtual space. The position may be associated with the position on the physical space for identification. Needless to say, the above-mentioned specific function objects are not limited to game controllers, and examples include touch panels, track pads, touch displays, toggle switches, button switches, wheels, steering wheels, levers, Joysticks, discs, sliders, rotary bodies, control panels, work instruction devices, mice, keyboards, air hockey tables, whack-a-mole, musical instruments with keyboards such as pianos, stringed instruments such as guitars, musical instruments with striking surfaces such as drums, card games cards and play area, mahjong game tiles, board game pieces, etc., as long as it realizes the function of transmitting (inputting) the operator's intention or action to this system.

After that, each time the wearer gives an action at the position of this real object (for example, the wearer presses a specific button among the buttons B1 to B4), the position information of the real object is used as a base. can be detected as motion information for each motion (for example, a “reaction” motion, a “key touch” motion, a “key press” motion, etc.). It should be noted that the "specific action" is not limited to the action of "stroking to form a circle", and includes, for example, closing the finger, pressing and holding with the fingertip, tapping twice with the fingertip, pointing the finger in a specific direction, Any motion that can identify one or more points by a known algorithm, such as drawing a line with a fingertip or pointing with a controller, can be used. Furthermore, the operation to specify one or more points as described above is to specify a substantially plane and an operation target range or a safety confirmation range in it by continuing or combining (so-called "manual calibration", "clearance confirmation" ), but instead of that continuous action, for example, by tracing the periphery of the operation target range with a finger or stroking it with the palm, it is possible to specify an approximately plane and the operation target range at the same time with one action. good. This includes a step of processing the range detected as a result of stroking or tracing into a circular or polygonal range through a known algorithm, and a range adjustment of the operation unit to deal with detection accuracy (detection deviation or blurring). , the size, or the step of adjusting the starting position of the hit determination.

In addition, a kick is defined as a specific preliminary action (user hits the hand any number of times, draws an arbitrary figure such as a circle in the air, emits a specific sound, makes the hand a specific shape, etc.). , then target the indexed target (by the user) as a specific object, and perform a specific post-action (which may be the same as the above-mentioned pre-action, clap the hand a different number of times than the pre-action, or perform a pre-action such as ▽ in the air). Draw a figure different from the action, make a specific sound) is detected as a kick to complete the action of targeting a specific object, or while the previous action is continuing (for example, controller button, hand wave, index finger and thumb pinch), and the end of the pre-action (e.g., hand off button, hand still, hand open). ) or complete the object-viewing action upon detection of an action that triggers the completion of the pre-action (e.g., pressing another button, rotating the hand, pinching the middle finger and thumb). Furthermore, by combining or processing the contents of space detection and the (virtual) position information given in advance by hardware or software, the target object is first placed in the virtual space. After displaying, you can select and move the vertices of the object, move the object by pressing the top, bottom, left, and right with one hand or both hands, and adjust it after the fact by performing button operations corresponding to scaling and movement of the object. You can Furthermore, this adjusting action may include the action of designating one or more points, the action of targeting by the pre-action, the post-action, and the like. In addition, the above-mentioned actions performed with fingers and hands can be replaced by detecting actions of external devices (controllers, input devices based on them, or physical objects in a form that can detect actions, such as AR markers, etc.) and button input operations. can be done.

Further, when the virtual operation target display control unit 2126 detects such a series of actions, it is not limited to detection via the above-described still image. Alternatively, detection may be performed by measuring a physical quantity other than a still image, such as a moving image, sound, vibration, or the like.

Defining this from the wearer's point of view, an object in the real world (for example, a desk surface board, a cup/doll on a table, etc.) is regarded as a game controller, and a specific action (for example, "stroking to form a circle" action) is performed. I do. On the system side, by detecting that this specific action has been performed, the real world object is regarded as matching the specific function object (for example, a game controller) in terms of position information, or the real world object and the virtual A base (or platform) is generated for linking with a specific functional object corresponding to this in the world, that is, for forming an associative memory, which will be described later. After performing this specific action (for example, the action of "stroking to form a circle") (using the specific action as a kick), combined with the effect of forming associated memory, which will be described later, the wearer can use the game controller, touch panel, By realistically adding various operations to the above-mentioned objects in the real world, you can experience the feeling that your intention to operate is transmitted to the system side as if you were operating a work instruction device, a musical instrument, or a card or piece in a board game. It will be possible. In terms of the technical concept of the present invention, the associative memory formation promotion part (to be described later) can also exhibit the effect of improving the precision and finesse of the above operations. In terms of human psychology, it is as if an object in the real world can be used as a game controller, that is, a specific object with physical substance (which can be arbitrarily specified by the operator each time). For example, it can be regarded as an interface between the virtual world and the real world, such as a controller, and it is not the experience of operating in the void as before, but the reaction force of physical objects or the real experience accompanied by physical entities. will get In other words, it is possible to have an experience that brings a sense of touch into the virtual reality of the game. The associative memory formation promoting part will be described in detail later. Since it is possible to obtain physical feedback in the form of reaction force from the desk, the stimulation from this feedback is directly transmitted to the human nerves, increasing the sense of immersion. Dexterity, high speed, precision, etc.) in performing certain operations will also be obtained. Furthermore, in addition to the reaction force, elasticity, friction, temperature difference, temperature change, hammering sound from the object, and odor emitted by the action (for example, desk Feedback such as fragrance applied to the skin) can also be incorporated to enhance immersion and associative memory.

Therefore, as a wearer, a specific object in the virtual world, such as a controller, is allowed to "possess" on a physical object, and thereafter, a situation is first created as if the controller is possessed by such a physical object. , (for example) game experience, performance experience, and work experience accompanied by physical feedback (i.e., feeling a reaction force from a physical object, in other words, forming an associative memory, which will be described later) under such circumstances , learning and training experiences. For example, if we define this from the perspective of the wearer during the progress of the game, it will be more realistic, especially when it is operated with a momentary intense movement, as if it were a sport (so-called E-sports). You will get a real feeling of operation. Therefore, according to the present embodiment, the operator/user can be given a sense of realism and a sense of immersion that has never been experienced before, and a greater sense of satisfaction. In other words, for example, a user can use a desk or wall (furthermore, a cardboard box, a wooden box, a pillar, a musical instrument, a floor, a surface of water, a part of the body such as one's or another's belly, a doll, a fruit, etc.) as a "controller." ”, and you will have a different (for example) gaming experience than before. For this reason, objects (desks, tables, etc.) anywhere in the real world can be used (for example) as "controllers" for games. It is possible to provide a more powerful game experience or a game experience with a mysterious feeling. In addition, for example, in the case of domestic games, game content (for example, animal breeding games, farm management games, medieval immersive role-playing games, defense games, shooting games, music games, card games, puzzle games, mahjong games, board games, etc.) games, etc.) (for example, collars, farm equipment, facility models, swords and shields, triggers and firing buttons, musical instruments, cards and play mats, puzzle pieces, mahjong tiles, small items such as pieces, etc.) can be selected.

In the above game, for example, when a desk is a specific object, the wearer confirms that there are no obstacles on the desk (also referred to as "clearance confirmation" or "manual calibration"). You can intervene. In this case, for example, it is possible to proceed on the assumption that the desktop surface is relatively safe. In the case of applying this to a game, the action and operation center of the game play will be concentrated on the top surface of the desk, so even in a relatively narrow space, the above-mentioned This leads to the realization of a game experience with a high degree of sensibility. In addition, in order to further ensure safety by this clearance confirmation, an object indicating the clearance confirmation result range or the processed range (for example, an obstacle object such as a fence or a signboard, or a warning object) may be displayed. . In addition, the above object indicating the safe range is processed and displayed according to the detection result when the user is about to go out of the safe range by mistake or when an operation outside the safe range is detected. It can be anything.

On the other hand, the changed APP display screen detected by the operation detection unit 2124 and sent to the APP is sent to the content display control unit 2128 . At this time, when the display of the APP is changed to reflect the degree of progress of joint learning (according to another embodiment of the present invention, which will be described later), information from the joint learning progress detection unit 2129 is used. good. Here, the content display control unit 2128 controls the display 6 of the HMD 5, which is the hardware, to display the changed APP display screen. It is realized by cooperating with wear. The function of the federated learning progress detector 2129 according to another embodiment of the invention will be described later.

Information including an image in the real space captured by the TV camera 7 of the HMD 5 is sent to the virtual operation target display control unit 2126, processed there, and then merged with the image from the content display control unit 2128 in the display control unit 2127. and sent to the HMD. Thereby, the necessary information is comprehensively displayed on the screen 10 of the HMD. If we define this from the viewpoint of the operator (wearer), we can say that it is a semi-real/semi-virtual space in which virtual objects are mixed in the real space, or a mixed reality space in which the real space and the virtual space are aligned and superimposed. is appearing in front of you. The virtual operation target display control unit 2126 pre-edits (for example, scales) the information including the image in the real space captured by the hardware of the TV camera 7 so as to be suitable for the image merge operation (to be described later). It is realized by the cooperation of the above-mentioned hardware and the above-mentioned computer resources with an algorithm (software) that causes the computer resources to perform the function of controlling the above-mentioned hardware and computer resources. The display control unit 2127 merges (combines) the image information edited by the virtual operation target display control unit 2126 and the image information from the content display control unit 2128 into combined image information. This is achieved by cooperation between an algorithm (software) that causes computer resources to perform the function of controlling display of image information on hardware such as the display 6 of the HMD 5 and the hardware.

Here, referring back to FIG. 2, the details of formation of associative memory (also referred to as "associative learning") according to one embodiment of the present invention will be described. FIG. 3 is a functional block diagram obtained by adding the configuration of an associated memory formation promotion unit 30 to the functional configuration diagram (functional block diagram) according to FIG. 2, and FIGS. 30 is an enlarged view of the conceptual configuration block diagram conceptually depicting how an associative memory is formed. As shown in FIGS. 3 and 4, the associative memory formation promoting unit 30 is composed of an operating subject movement detecting unit 2125, a virtual operating target display control unit 2126, and a desk 1 (as an entity). (Strictly speaking, the reaction force given by the desk 1 to an action such as pushing down on the desk 1). In this case, the desk 1 is given as an example of a realistic (that is, having a physical substance) object, and such an object is not limited to a desk, but can be any other movable object. Even if it is (desktop light, document holder, chair, dining table, cardboard box, mahjong table, bed, sofa, flat plate, musical instrument, doll, vegetables, fruit, etc.), fixed objects (walls, handles, pillars, floors, etc.) or anything else (manipulator's hands, fingers, stomach, chest, other person's back, soil, water, etc.) that has physical substance, i.e., can give a reaction force. It doesn't matter which one.

As shown in FIG. 4, the operator 4 gives an action (stimulation action) S1 of pressing down on the desk 1 (strictly speaking, the operator 4, for example, the index finger 4a is in direct contact with the desk 1). ), the desk 1 instantaneously applies a reaction force S2 against S1 to the forefinger 4a. Then, the stimulation by this reaction force S2 is transmitted to the central nerves of the operator 4 via the peripheral nerve network of the operator 4 (event S3), resulting in a certain experience in the brain (for example, the action S1 is The experience of giving to things) S4 is formed as a memory. At this time, substantially concurrently, as shown in FIG. 5, outside the operator 4, as the information processing operation of this system, the above-described motion detection unit 2125 of the operating subject and the virtual operation target display control unit 2126 Due to this movement, the action S1 is translated/information-converted as a specific input action (for example, the action of instructing the "controller" to move rightward) in an information-processing manner on this system, and this specific input action is reflected. (That is, for example, the object 1601 in the virtual space is moved to the position of the object 1603 by proceeding rightward by the "controller"). 4 sees this, a specific success experience (for example, the experience that the object 1601 was able to move to the position of the object 1603) S5 is formed as a memory in the brain. Since the experience S4 and the successful experience S5 are generated substantially simultaneously as described above, an associativity S6 is formed between the memories of both S4 and S5 in the brain. As a result, an associated memory S6 of S4 and S5 is formed. In the above, when the specific success experience S5 is formed as a memory in the brain, the operator 4 visually recognizes that the result of translating/converting the action S1 as the specific input action is displayed on the display unit. Although an example in which S5 is formed as a memory is not limited to visual recognition, for example, the operator 4 confirms by auditory sense, olfactory confirmation, A specific success experience S5 is triggered by any of the five human senses other than vision, such as the result of being confirmed by touch (the application of a specific stimulus), the result of being confirmed by taste, or any combination of these. It may be formed as a memory.

The above aspects will be explained from the operator's point of view. The input action S1 is performed on the desk 1, which is a real object in which the object in the virtual space is embodied after recognizing that the input action S1 has been performed. At this time, a reaction force S2 from an object having a physical substance (for example, a desk, a wall, one's other hand, desk 1, etc.) is instantaneously applied to (hand or) finger 4a, so that The experience of sensing the reaction force S2 is transmitted from the peripheral nerve of the finger 4a to the operator's brain (S3), thereby forming a certain memory S4. On the other hand, the reaction force (strictly, feeling the reaction force) S2 and the output as a result of information processing with the previous input motion S1 as input (for example, a character moving sideways in the virtual space etc.) is formed by viewing the image 10 displayed on the display unit (for example, the display 6). Since the memory S4 and the memory S5 are generated substantially at the same time, a causal relationship between the memory S4 and the memory S5 is recognized in the brain. Associative memory is formed. In other words, by superimposing the virtual space and the real space through the physical object, an associative memory unique to the present application is formed. When an input action S1 is performed on an object in the virtual space in the void as in the prior art, there is no reaction force as described above. . Therefore, according to the above aspect of the present application, it contributes to steady memory formation compared to the case of empty space. If this is applied to real life, effects such as proficiency in game control technology, improvement in keyboard input speed and accuracy, improvement in driving technique, acquisition and improvement of machine operation technique, acquisition and improvement of musical instrument playing technique, etc. It leads to playing. In other words, when you perform a certain action, there is an entity that produces a response to that action and physical pressure, and the memory and information processing results related to such physical pressure are perceived (visually recognized, By forming an associative memory with associative memories (auditory, olfactory, tactile, taste, etc.), a structure is created in which one can confirm that one has performed the action. By actually using these technical ideas, it will contribute to the improvement of certain human actions (skills and techniques). From a different point of view, this means that associative learning in virtual reality has become possible.

In the above, objects with elasticity, texture, temperature, and smell may be adopted. By using objects with appropriate elasticity, texture, temperature, and smell as objects, associative memory is formed with the addition of elasticity, texture, temperature, and smell, resulting in deeper and diverse associative memory, That is, there is a possibility that an associative memory in which memories related to elasticity, texture, temperature, and smell are added to the above-described associative memory.

Next, the function of the federated learning progress detector 2129 according to another embodiment of the invention will be described in detail. In general, it is possible to estimate the degree of progress using a learning curve (however, estimation of the degree of progress is not limited to this). FIG. 11 is a conceptual diagram for explaining a learning progress estimation method according to an embodiment of the present invention. More specifically, the correspondence between learning progress and reaction time (reciprocal of proficiency). FIG. 4 is a diagram showing relationships; An example of a method for estimating the progress of federated learning will be described using a learning curve with reference to FIG. Actions such as button pressing are measured as reaction time from the presentation of a stimulus to the button being pressed. Generally, as shown in FIG. (For example, Non-Patent Document 2), by measuring the reaction time from the presentation of a certain stimulus to the pressing of a button, the operator's proficiency for actions related to such a stimulus can be estimated. . For example, by measuring the reaction time of pressing a button in a game, it is possible to determine whether the operator is at stage A (elementary stage of learning progress) or stage B (intermediate stage of learning progress). , or C stage (a stage where the learning progress is advanced). In other words, by focusing on the fact that it is possible to estimate proficiency by measuring the reaction time from the presentation of a stimulus to the pressing of a button, it is possible to estimate the progress of learning. It is possible to select the display method of corresponding contents. In addition to this, the estimation of progress and proficiency includes typing error rate, game win/loss rate, progress of pre-instructed (a series of) actions, user's action speed and reaction speed, quiz correct answer rate, action accuracy, degree of completion or degree of completion of actions calculated by algorithms (for example, the number of chains in puzzle games, the degree of openness in reversi, the number of listeners in mahjong games, etc.), using the above variables It is also possible to use other methods, such as the required score, key press intensity, and finger shape change information for finding it.

In the case of using a learning curve, for example, in a game, it is possible to estimate whether the operator is in stage A, stage B, or stage C of learning progress by measuring the button press time. It becomes possible. It is possible to control the progress speed of the game according to the proficiency level.

Based on the progress estimated in this way, it is possible to give an appropriate load to the operator. That is, for example, in the case of an operator estimated to be in stage A of learning progress, a task that gives a lighter load is presented to the operator estimated to be in stage C of learning progress. On the other hand, by presenting a task that gives a fairly heavy load, setting (designing) learning and memory for stimulus-related actions according to the progress of learning, and incorporating this into the system It is possible to realize a tactile-simulated memory consolidation promoting system that utilizes the strength of associative memory. In this way, the load design based on the estimated degree of progress promotes consolidation of associative memory (dotted line in FIG. 11) and strong associative memory (in this case, shorter reaction time, i.e., less likely to forget, and faster response speed). It is possible to realize the formation of (broken line in FIG. 11). In other words, if this is used in the game industry, for example, it is possible to measure the degree of proficiency in game operation (based on the estimation of learning progress) and use it for game content, degree, environment setting, etc. according to the level of proficiency. becomes possible.

FIG. 11 has been explained using a learning curve, but in addition to this, there is a typing error rate, a game winning/losing rate, a progress of a pre-instructed (a series of) actions, a user's action speed and reaction speed, a quiz correct answer rate, Accuracy of movement, degree of completion of movement calculated by algorithm or variables for obtaining degree of completion (for example, number of chains in puzzle games, degree of openness in reversi, number of listeners in mahjong games, etc.), using the above variables It is possible to use an index that is considered appropriate for APP, including the score obtained by pressing the key, information on changes in finger shape for obtaining it, or any combination thereof.

Also, in the above description, a currently widely used display may be used instead of the HMD 5. In this case, it is possible to eliminate the need for an existing keyboard by arranging a TV camera and sensors externally. , and the operator can perform key operation or button press operation according to the operator's operation instruction subject.

Next, the operation and action of the image display system according to another embodiment of the invention configured as described above will be described. FIG. 12 is an example of a flowchart for explaining the operation of the image display system according to one embodiment of the present invention.

First, a space including a substantially plane is displayed on the display screen including the HMD5 screen (step 601). Next, an object having a substantially flat portion is detected by the space detection unit 2122 from the image captured by the TV camera 7 of the HMD 5, and a virtual plane is set at a predetermined position on the display screen 10 and displayed (step 603). ).

Next, the motion detection unit 2125 of the operating instruction subject detects the motion of the operating instruction subject specifying the operation range, and the operation range is determined on a substantially plane (step 605). Further, the virtual operation object display control unit 2126 displays a virtual operation object including a virtual keyboard and virtual buttons within the operation range (step 607).

Here, the content display control unit 2128 displays a screen (content) generated by application software including games and office software in the background of the virtual operation target (step 609). Through such a process, the operation of the operation instruction subject is given to the content, whereby the operation of the operation instruction subject is detected by the operation detection unit 2124 (step 611), and the operation of the operation instruction subject is transferred to the application software. As a result, the display content is changed by the content display control unit 2128 (step 613).

Next, the movement detection unit 2125 of the subject of the operation instruction detects the movement of the subject of the operation instruction with respect to the content (step 615). This motion detection is for obtaining the information necessary for estimating the progress of associative learning. may include, but are not limited to, As an example of how to measure the key pressing strength for objects in virtual space, the difference in fingertip shape (for example, degree of curvature, angle between the fingertip and the first and second joints) when the key is pressed lightly and when the key is pressed strongly, Variations can be used. Next, the motion detection unit 2125 of the operation instruction subject detects the movement of the operation instruction subject, the association learning progress detection unit 2129 detects the progress of the association learning (step 617), and the detected virtual operation target is operated and detected. / Or, according to the progress of the associated learning, a change occurs in the content to be operated (step 619), prompting the operator to perform the next operation.

As described above, according to the image display system according to one embodiment and/or another embodiment of the present invention, when operating a computer or a game in a virtual space using, for example, an HMD, a special It is possible to provide a tactile sensation in the real space without using any additional equipment, thereby promoting associative learning of visual and/or auditory and tactile sensations and forming a strong associative memory. Become. From the point of view of the operator/user, these have the unique effect of being able to obtain a tactile sensation accompanied by a tangible sensation even when operating an object in the virtual space. . For this reason, it is possible to provide a virtual or augmented world operation experience with more realism and impact (for example, game worlds, typing assignments, live houses, tourist facilities, factory training facilities, etc.).

In addition, according to the above aspect, it is also possible to monitor the progress of associative learning by detecting (operator's) movements of fingers, hands, and the like. That is, it is possible to give a tactile sensation generated by a certain action, gesture, or operation such as pressing a keyboard or a button in a virtual space including an HMD, thereby providing visual information (alternatively, or combined with this, auditory information). , olfactory information, gustatory information, tactile information, etc.) and the tactile sensations generated by the above-mentioned certain actions, gestures, and operations, and form a stronger associative memory. Become. Furthermore, in this case, by monitoring the progress of learning, it becomes possible to further promote associative learning and form strong associative memory.

In addition, it eliminates the need for an additional device that was previously required to give the operator a sense of touch, and as a result eliminates the need for a different controller for each game. It has a very large effect on the top and resources.

Furthermore, according to the above aspect, not only can the virtual buttons and the virtual keyboard be arranged and sized freely in the virtual space, but the program can also be used without the operator's permission (or without the operator's awareness). Since it can be changed dynamically, it becomes possible to provide programs such as complicated games that could not be realized so far.

In addition, by forming a virtual object on the plane of an object such as a desk that may become an obstacle in the virtual space, the user's motion target range will be within the range of the formed object. . This eliminates the need for large movements by the user and reduces the risk of colliding with surrounding furniture. In addition, when specifying an object and its range, inserting a safety range check ("clearance check" or "manual calibration") prevents unintended situations within the real range when the user operates in the virtual space. (e.g. knocking over a coffee cup placed on the edge of the desk) can be reduced. This effect can be further enhanced by combining means for displaying the confirmed safety range according to the user's operation (display of boundary lines, fences, etc.).
The appearance (shape, layout, etc.) of the displayed operation target can also be changed according to the size of the operation target range. As a result, unintended changes in operability and visual discomfort due to differences in the size of the operation range specified by the user due to differences in the size of the real space, etc., can be further reduced, and content with a high degree of immersion can be created. can be provided.

In the above description, the description has been given mainly from the viewpoint of a tactile augmented information processing system as a memory fixation promotion system that does not require an additional device that simulates the tactile sense. or as a method for producing a specific effect by the functions detailed above, or as a recording medium on which the software is recorded.

As described above, according to the tactile augmented information processing system according to one aspect of the present invention, which can be said to be a memory consolidation promoting system that does not require an additional device that simulates a tactile sense, there is no need to use an additional device that simulates a tactile sense. , using a flat surface in front of the operator to give tactile sensations such as key presses. The feeling is improved. In addition, associative learning between visual, auditory and tactile senses is formed. Therefore, the present invention is useful in the information industry, the game industry, the music industry, the tourism industry, the construction industry, the equipment maintenance industry, the manufacturing industry, and the education and training industry, even from the perspective of being able to provide improved APP usage literacy. It has great industrial applicability and convenience in industry, medical industry, medical-related industry, and the like.

1: Substantially flat surface 2: Virtual keyboard 3: Virtual button 4: Operation instruction subject in real space of operator (exemplified by hand)
5: HMDs
6: Display 7: TV camera 8: Sensor 9: Communication I/F
10: Display screen of HMD 11: Display screen of APP 12: Virtual plane 13: Keyboard 14: Button 15: Display in virtual space of an operation instruction subject (example hand) in real space 16: Communication I/F
17: CPU
18: GPUs
19: Memory 20: External storage device 21: Operation control unit 601: Display of space including approximately plane on display screen including HMD screen Step 603: An object having approximately plane portion is detected from the photographed image and displayed on display screen A virtual plane is set and displayed at a predetermined position. Step 605: The operation range is determined on a substantially plane by the operation instruction subject's specifying operation of the operation range. Step 607: A virtual keyboard or a virtual Step 609 for displaying a virtual operation target including target buttons: A screen (content) generated by an APP including a game or office software is displayed around the virtual operation target, or at the position of the virtual operation target. Step 611 displayed accordingly: The operation of the operation subject is detected by the operation instruction subject giving an operation to the content, or the operation instruction subject gives an operation to the content with a sense of touch, and the operation is detected. An operation is detected, Step 613: The operation of the operation instructing subject is transferred to the application software, and as a result, a change is given to the displayed content. Step 615: Movement of the operation instructing subject with respect to the content is detected. Step 617: Operation. Step 619: Motion of the instruction subject is detected, and progress of associative learning is detected: In accordance with the operation on the detected virtual operation target and/or the progress of associative learning, the content that is the operation target is changed to the next step. Step 2122 where the operator is prompted to perform an operation: space detection unit 2123: operation range setting unit 2124: operation detection unit 2125: motion detection unit 2126: virtual operation target display control unit 2127: display control unit 2128: content display Control unit 2129: Associative learning progress detection unit S1: (Stimulation) action S2: Reaction force from desk 1 to S1 S3: Stimulation by reaction force S2 is transmitted to the central nervous system via peripheral nerve network of operator 4 Event S4: Experiential memory of giving action S1 to a physical object S5: Experiential memory of being able to move object 601 to the position of object 603 in virtual space S6: Formed associativity, associative memory

Claims

a display unit worn by an operator located in the vicinity of an object having physical substance or arranged in the vicinity of said operator;
a sensor capable of detecting motion of the operator with respect to or associated with the object;
a display control unit that instructs or specifies content to be displayed on the display unit;
virtual position information in the virtual space corresponding to the motion is determined based on the motion performed by the operator with respect to the object in the virtual space corresponding to the target object being detected as the detection motion; Acquiring action specifying information corresponding to the motion based on a predetermined correspondence relationship with the action specifying information corresponding to the motion specifying information, and storing the physical reaction force given by the object to the motion and the motion A tactile augmented information processing system, comprising: an associated memory formation promoting unit that causes the operator to form an associated memory with a memory related to specific information.
The associative memory formation promoting unit
When the sensor detects that the operator has performed a specific action on the object or in relation to the object at a real position on the physical space, the defined action for the specific action is detected. a virtual operation target display control unit for displaying an object at a virtual position in the virtual space corresponding to the real position on the display unit;
From the action given to the object by the operator, real position information where the object is located in the real space corresponding to the action, virtual position information in the virtual space corresponding to the real position information, and the action an operating subject motion detection unit that obtains motion specifying information corresponding to the
2. The tactile augmented information processing system according to claim 1, further comprising: an object having the physical substance, the object giving the operator a physical reaction force against the action given by the operator. .
The tactile augmented information processing system according to claim 1 or 2, wherein the sensor has an imaging function.
further comprising an imaging unit having an imaging function,
The associative memory formation promoting unit
defined for the specific action when the imaging unit detects that the operator performs a specific action on the object or in relation to the object at a real position on the physical space a virtual operation target display control unit for displaying the object at a virtual position in the virtual space corresponding to the real position on the display unit;
From the action given to the object by the operator, real position information where the object is located in the real space corresponding to the action, virtual position information in the virtual space corresponding to the real position information, and the action an operating subject motion detection unit that obtains motion specifying information corresponding to the
2. The tactile augmented information processing system according to claim 1, further comprising: an object having the physical substance, the object giving the operator a physical reaction force against the action given by the operator. .
The tactile augmented information processing system according to any one of claims 1 to 4, wherein the display unit has a see-through mechanism.
The tactile augmented information processing system according to any one of claims 1 to 5, characterized in that the display unit can form a virtual space for the user to perceive.
The tactile augmented information processing system according to any one of claims 1 to 6, wherein the sensor has at least one of an optical sensor, a magnetic sensor, a contact sensor, and a distance image sensor.
the object is elastic,
The tactile augmented information processing system according to any one of claims 1 to 7, wherein the memory related to the physical reaction force is weighted by the memory related to the elasticity.
The object has any one or a combination of hardness, elasticity, temperature, and texture that provides a tactile sensation sufficient for forming the associative memory, and is at least one of curved, uneven, and non-horizontal. Item 9. The haptic augmented information processing system according to any one of items 1 to 8.
It further comprises an associative learning detection unit that detects the progress of associative learning based on detected hand and finger movements,
10. The display control unit changes the content that is the operation target according to the operation of the virtual operation target detected by the associated learning detection unit and/or the progress of the associated learning. The haptic augmented information processing system according to any one of the above.
further comprising a federated learning detection unit for measuring and estimating the progress of manipulation and/or federated learning on the virtual operation target;
10. The display control unit changes the content that is the operation target according to the progress of the operation of the virtual operation target estimated by the associated learning detection unit and/or the associated learning. The haptic augmented information processing system according to any one of the above.
12. The augmented haptic information processing system according to claim 11, wherein the visual information and/or auditory information related to the content is changed according to the degree of progress of the joint learning estimated by the joint learning detection unit.
The estimation of the progress of the federated learning includes the learning curve, typing error rate, game win/loss rate, progress of a series of predetermined actions, user's action speed and reaction speed, correct answer rate, accuracy, calculated by algorithm at least one of a degree of completion or a variable for measuring the degree of completion, key pressing strength, finger shape change information for obtaining key pressing strength, and button pressing time information. 13. The haptic augmented information processing system according to 11 or 12.
The display unit is a display installed in front of the operator,
visual information relating to the content is presented on the display;
5. The tactile augmented information processing system according to claim 3, wherein said imaging unit and/or said sensor can detect movement of said operator.
An information processing system comprising a display unit, a sensor capable of detecting actions of an operator located near a physical object, and a control unit controlling the display unit and the sensor,
a first step in which a motion of the operator with respect to the object or in association with the object is detected as a detected motion;
a second step of identifying, by the control unit, virtual position information in a virtual space corresponding to real position information in a real space associated with the object in relation to the detected sensing action;
a third step of acquiring motion specifying information corresponding to the motion based on a predetermined correspondence relationship between the sensed motion and motion specifying information corresponding thereto;
and a fourth step of causing the operator to form an associative memory between a memory related to the physical reaction force given by the object to the motion and a memory related to the motion specifying information. haptic augmented information processing method.
The fourth step is
The control unit adjusts the size of the object in the virtual space corresponding to the specific action, and the adjusted object is at the position identified by the virtual position information in the virtual space of the display unit. to display at a position superimposed on the real position information on the real space associated with the object,
A motion given by the operator to the real object is detected as a motion given by the operator to the object in the virtual space, and a type of the detected motion and a real space related to the motion The control unit specifies the above real position information, and the object gives a physical reaction force to the action to the operator at a position in the real space corresponding to the object in the virtual space,
After the controller determines the real position information on the physical space related to the detected motion and identifies it with the virtual position information on the virtual space, the motion corresponding to the type of the detected motion is performed. to identify the motion identification information related to the type of motion and the virtual position information corresponding to the motion,
said associative memory between a first memory in which said operator perceives a physical reaction force given by said object to said action and a second memory in which said operator perceives said action specifying information; 16. The tactile augmented information processing method according to claim 15, wherein the haptic augmented information processing method is formed by an operator.
An information processing system comprising a display unit, a sensor capable of detecting actions of an operator located near a physical object, and a control unit controlling the display unit and the sensor,
a first step of displaying content to be operated on the display unit;
a second step of detecting spatial information relating to entities with physical entities in real space in front of the operator;
a third step of setting a virtual plane at the position detected in the second step;
a fourth step of setting a physically operable range on the virtual plane set in the third step;
A virtual including at least one of virtual buttons, game controllers, touch panels, discs, operation panels, mice, keyboards, musical instruments, wheels, handles, levers, and sticks within the operable range set in the fourth step a fifth step of displaying a target for manipulation in the virtual space;
a sixth step of detecting the operation while applying a reaction force in real space to the operation of the virtual operation target by the operator in response to a change in content;
a seventh step of detecting a movement of an operation instructing subject in the operation on the virtual operation target;
an eighth step of detecting the progress of associative learning based on the movement of the operation instructing subject detected in the seventh step;
and a ninth step of changing content as an operation target in accordance with the operation of the virtual operation target detected in the eighth step and/or the progress of associated learning. Processing method.
An information processing system comprising a display unit, a sensor capable of detecting actions of an operator located near a physical object, and a control unit controlling the display unit and the sensor,
to the computer,
a first means for causing the sensor to detect, as a detection action, that the operator has performed an action with respect to the object or in association with the object;
a second means for causing the control unit to identify virtual position information in the virtual space corresponding to the real position information in the real space associated with the object related to the sensing action detected by the sensor;
a third means for acquiring action specifying information corresponding to the action based on a predetermined correspondence relationship between the detected action and the action specifying information corresponding thereto;
The physical reaction force is applied to cause the operator to form an associative memory between the memory of the physical reaction force applied by the object to the action and the memory of the action specifying information. and a fourth means for presenting the action specifying information or information obtained by processing the action specifying information to the operator at substantially the same timing as the augmented haptic information processing program.
The fourth means further comprises:
The control unit adjusts the size of the object in the virtual space corresponding to the specific action, and the adjusted object is at the position identified by the virtual position information in the virtual space of the display unit. to display at a position superimposed on the real position information on the real space associated with the object,
A motion given by the operator to the real object is detected as a motion given by the operator to the object in the virtual space, and a type of the detected motion and a real space related to the motion The control unit specifies the above real position information, and the object gives a physical reaction force to the action to the operator at a position in the real space corresponding to the object in the virtual space,
After the controller determines the real position information on the physical space related to the detected motion and identifies it with the virtual position information on the virtual space, the motion corresponding to the type of the detected motion is performed. to identify the motion identification information related to the type of motion and the virtual position information corresponding to the motion,
said associative memory between a first memory in which said operator perceives a physical reaction force given by said object to said action and a second memory in which said operator perceives said action specifying information; The operator is caused to present the action specifying information or information obtained by processing the action specifying information at substantially the same timing as the physical reaction force is applied. 19. The haptic augmented information processing program according to claim 18, wherein the haptic augmented information processing program is caused to function.
A recording medium storing the extended haptic information processing program according to claim 18 or 19.