JP2019200491A - Information processing apparatus, information processing method and program - Google Patents

Abstract

To enable a user to intuitively recognize information in a virtual space displayed on a head mounted display without forcing the user to perform complicated operation.SOLUTION: An information processing apparatus 10 comprises: an object management unit 11 which manages a plurality of objects holding information in association with each other on the basis of the information held in the plurality of objects respectively; a user movement detection unit 12 which detects movement within a virtual space of a user 30 wearing a head mounted display 20 displaying the virtual space; a user position control unit 13 which moves the user 30 within the virtual space into an object within the virtual space in response to detecting contact with this object by the user 30; and an object arrangement unit 14 which, after the movement of the user 30 into the object, specifies another object associated with the object and arranges another specified object within the object.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an information processing apparatus and an information processing method for providing a user interface in a virtual reality that a user experiences through a head-mounted display, and further relates to a program for realizing them.

  In recent years, since the price of a head mounted display (HMD) has been reduced, practical use of a system using HMD and virtual reality (VR) has been advanced. In particular, according to the non-transparent HMD, it is possible to provide an immersive feeling to the user in the VR, and the user can feel the VR as an actual reality.

  In addition, the use of such a system using HMD and VR is not limited to games, and is diverse, including simulation, medical care, entertainment, automobiles, advertising, and education.

  By the way, in such a system using HMD and VR, the user cannot see the hand and cannot use existing input devices such as a mouse, a touch panel, and a keyboard. For this reason, various user interfaces have been developed for HMD (see, for example, Patent Document 1).

  Specifically, Patent Document 1 discloses an HMD having a function of detecting a focus on a screen corresponding to a user's line of sight. In the HMD disclosed in Patent Document 1, the user adjusts the position of the focus to a specific object (operation button or the like) on the screen by the line of sight, and holds the state for a specific period, thereby operating the specific object. It can be performed.

JP 2017-510877 A

  However, in the HMD disclosed in Patent Document 1, the focal point functions in the same way as a pointer on a two-dimensional screen, and the position of the focal point is used when necessary information is extracted from hierarchically structured information. It is necessary to perform the operation of holding for a certain period of time many times. For this reason, in the HMD disclosed in Patent Document 1, the operation by the user becomes more complicated as the amount of data to be operated increases.

  An example of the object of the present invention is to solve the above-mentioned problem, and in a virtual space displayed on a head-mounted display, the user can intuitively grasp information without forcing the user to perform complicated operations. An information processing apparatus, an information processing method, and a program are provided.

In order to achieve the above object, an information processing apparatus according to one aspect of the present invention provides:
An object management unit that manages a plurality of objects that hold information in association with each other based on the information that each holds;
A user motion detector that detects a motion of the user in the virtual space, with a head-mounted display that displays the virtual space;
When the user motion detection unit detects the user's touch on the object arranged in the virtual space as the user motion, the user touches the user in the virtual space. A user position controller that moves the object within the object,
When the user moves into the object, an object placement unit that identifies another object associated with the object and places the identified other object in the object;
It is characterized by having.

In order to achieve the above object, an information processing method according to one aspect of the present invention includes:
(A) managing a plurality of objects holding information in association with each other based on the information held by each of the objects;
(B) detecting a user's operation in the virtual space, which is equipped with a head mounted display that displays the virtual space;
(C) In the step (b), when the user's action is detected by the user to contact the object placed in the virtual space, the user in the virtual space is Moving the object into contact with the object;
(D) when the user moves into the object, identifying another object associated with the object, and placing the identified other object in the object;
It is characterized by having.

Furthermore, in order to achieve the above object, a program according to one aspect of the present invention is provided.
On the computer,
(A) managing a plurality of objects holding information in association with each other based on the information held by each of the objects;
(B) detecting a user's movement in the virtual space, which is equipped with a head mounted display that displays the virtual space;
(C) In the step (b), when the user's action is detected by the user to contact the object placed in the virtual space, the user in the virtual space is Moving the object into contact with the object;
(D) when the user moves into the object, identifying another object associated with the object, and placing the identified other object in the object;
Is executed.

  As described above, according to the present invention, in the virtual space displayed on the head mounted display, the user can intuitively grasp information without being forced to perform complicated operations.

FIG. 1 is a block diagram showing a schematic configuration of an information processing apparatus according to an embodiment of the present invention. FIG. 2 is a block diagram specifically showing the configuration of the information processing apparatus according to the embodiment of the present invention. FIG. 3 is a diagram showing a bird's-eye view of the movement of the user performed in the virtual space in the embodiment of the present invention. FIG. 4 is a diagram illustrating a virtual space displayed on the head mounted display when the movement of the user illustrated in FIG. 3 is performed. FIG. 5 is a diagram illustrating an example of another object developed when the user touches a specific object, and FIGS. 5A and 5B illustrate different examples. FIG. 6 is a flowchart showing the operation of the information processing apparatus according to the embodiment of the present invention. FIG. 7 is a block diagram illustrating an example of a computer that implements the information processing apparatus according to the embodiment of the present invention.

(Embodiment)
Hereinafter, an information processing apparatus, an information processing method, and a program according to an embodiment of the present invention will be described with reference to FIGS.

[Device configuration]
First, a schematic configuration of the information processing apparatus according to the embodiment of the present invention will be described with reference to FIG. FIG. 1 is a block diagram showing a schematic configuration of an information processing apparatus according to an embodiment of the present invention.

  The information processing apparatus 10 according to the present embodiment illustrated in FIG. 1 is an apparatus that provides a user interface in the virtual reality that the user 30 is experiencing via the head mounted display 20. As illustrated in FIG. 1, the information processing apparatus 10 includes an object management unit 11, a user motion detection unit 12, a user position control unit 13, and an object placement unit 14.

  The object management unit 11 manages each of a plurality of objects holding information in association with each other based on the information held by each. Hereinafter, the above-mentioned “object holding information” will be referred to as “information object”. The user motion detection unit 12 detects a motion in the virtual space of the user 30 wearing the head mounted display 20 displaying the virtual space.

  When the user motion detection unit 12 detects that the user 30 has touched the information object arranged in the virtual space as the operation of the user 30, the user position control unit 13 displays the user 30 in the virtual space as the user 30. Move into the information object that is touching.

  When the user 30 moves into the information object, the object placement unit 14 identifies another information object associated with the information object, and places the identified other information object in the information object.

  Thus, in this embodiment, when a user touches an information object that holds information, the user moves into the information object, where the information object associated with the information object can be confirmed. . Therefore, according to the present embodiment, in the virtual space displayed on the head mounted display, the user can intuitively grasp information without being forced to perform complicated operations.

Next, in addition to FIG. 1, the configuration of the information processing apparatus 10 in the present embodiment will be described more specifically with reference to FIGS. 2 to 5. FIG. 2 is a block diagram specifically showing the configuration of the information processing apparatus according to the embodiment of the present invention. FIG. 3 is a diagram showing a bird's-eye view of the movement of the user performed in the virtual space in the embodiment of the present invention. FIG. 4 is a diagram illustrating a virtual space displayed on the head mounted display when the movement of the user illustrated in FIG. 3 is performed.
FIG. 5 is a diagram illustrating an example of another object developed when the user touches a specific object, and FIGS. 5A and 5B illustrate different examples.

  As shown in FIG. 2, in the present embodiment, the information processing apparatus 10 includes a virtual space in addition to the object management unit 11, the user motion detection unit 12, the user position control unit 13, and the object placement unit 14 described above. A construction unit 15 is provided.

  As shown in FIG. 2, in the present embodiment, a head mounted display 20 and a camera 21 for detecting the position of the user 39 are connected to the information processing apparatus 10.

  Although not shown, the head mounted display 20 includes a motion sensor that detects its own movement and an infrared sensor that detects the movement of the hand of the user wearing the head mounted display 20.

  The motion sensor includes an acceleration sensor and an angular velocity sensor, and outputs a signal specifying the movement of the head mounted display 20 itself. The infrared sensor includes a light source that outputs infrared light and an image sensor that receives the infrared light reflected by the wearer's hand, and outputs an infrared image as a signal.

  Moreover, in this Embodiment, the head mounted display 20 and the sensor used for it are not specifically limited. In the present embodiment, various head mounted displays and sensors that will be developed in the future can be used.

  The object management unit 11 includes an object database 16 in the present embodiment. The object database 16 registers all objects arranged in the virtual space including the information objects described above. Further, as described above, the object database 16 registers each information object in association with each other based on the information held by each information object.

  As illustrated in FIG. 3, the virtual space construction unit 15 uses the objects registered in the object database 16 and uses a virtual space 40 within a memory section of a memory (shown in FIG. 2) included in the information processing apparatus 10. Build up. As shown in FIG. 3, the virtual space construction unit 15 also arranges an object 31 (hereinafter referred to as “user object”) 31 representing the user 30 in the constructed virtual space, and displays it on the screen of the head mounted display 20. The video in the virtual space with the user object 31 as a viewpoint is displayed.

  In the present embodiment, the user motion detection unit 12 is based on the signal from the motion sensor mounted on the head mounted display 20 and the signal from the infrared sensor, and the posture and hand of the user 30 in the real space. Detect motion and. Further, the user motion detection unit 12 detects the position of the user 30 in the real space based on the video data from the camera 21.

  In addition, the user motion detection unit 12 reflects the detected posture, hand movement, and position of the user 30 in the real space on the user object 31 and detects the motion of the user 30 in the virtual space. For example, in the example of FIG. 4, contact with the file box 41 by the user object 31 is detected as an operation of the user 30 in the virtual space.

  In the present embodiment, the user position control unit 13 is configured to detect a user object in the virtual space 40 when the user motion detection unit 12 detects that the user 30 has touched the information object arranged in the virtual space. 31 is moved into this information object.

  In the present embodiment, when the user object 31 moves into the information object, the object placement unit 14 specifies another information object associated with the information object, for example, an information object that holds information on a lower concept. Then, the object placement unit 14 places the identified other information object in the information object in which the user object 31 exists.

  Here, specific examples of the functions of the user motion detection unit 12, the user position control unit 13, and the object placement unit 14 will be described with reference to FIGS.

  As shown in the upper part of FIG. 3, the user object 31 is arranged in the virtual space 40. In this case, as shown in the upper part of FIG. 4, an image in the virtual space 40 with the user object 31 as a viewpoint is displayed on the screen of the head mounted display 20.

  Then, it is assumed that the user 30 touches the file box 41 disposed in the virtual space 40 while looking at the screen of the head mounted display 20. In this case, the user action detection unit 12 reflects this action on the user object 31 and detects the contact of the user object 31 with the file box 41 as the action of the user 30 in the virtual space.

  If the file box 41 is an information object holding information, the user position control unit 13 moves the user object 31 in the virtual space 40 into the file box 41 as shown in the middle part of FIG. Move.

  Assuming that the files A to C included in the file box 41 are associated in advance, the object placement unit 14 stores the file box 41 in the file box 41 as shown in the lower part of FIG. Identify the associated files A-C. Then, the object placement unit 14 places an object 42 that holds each of the identified files A to C in the file box 41.

  Moreover, this Embodiment can be set as the aspect from which the other information object specified changes according to where the user 30 contacted the information object. That is, in this aspect, the user motion detection unit 12 also detects a contact position of the information object with the user 30 when detecting the contact of the user 30 with the information object arranged in the virtual space. The object placement unit 14 specifies another object associated with the information object according to the detected contact position.

  Specifically, it is assumed that the object management unit 11 manages, for example, each information object in the object database 16 based on the vertical relationship and parallel relationship of the information held by each.

  In the example of FIG. 5A, the object management unit 11 manages the file D and its child files d1 to d3 based on the vertical relationship. In the example of FIG. 5B, the object management unit 11 manages the file D and the related files E to G based on the parallel relationship. In the examples of FIGS. 5A and 5B, it is assumed that the contact with the top surface of the information object is tied in a vertical relationship, and the contact with the side surface of the information object is tied in a parallel relationship.

In such a case, as shown in FIG. 5A, when the user motion detection unit 12 detects the contact of the information object holding the parent file D with the upper surface, the object management unit 11 detects the child files d1-d3. Identify each information object that holds and place them. on the other hand,
As shown in FIG. 5B, when the user action detection unit 12 detects contact with the side surface of the information object holding the parent file D, the object management unit 11 sets each piece of information holding the related files EG. Identify objects and place them.

[Description of operation]
Next, the operation of the information processing apparatus 10 in the present embodiment will be described with reference to FIG. FIG. 6 is a flowchart showing the operation of the information processing apparatus according to the embodiment of the present invention. In the following description, FIGS. Moreover, in this Embodiment, the information processing method is implemented by operating the information processing apparatus 10. Therefore, the description of the information processing method in the present embodiment is replaced with the following description of the operation of the information processing apparatus 10.

  First, as a premise, it is assumed that the virtual space construction unit 15 constructs the virtual space 40 in the memory section of the memory included in the information processing apparatus 10 using the objects registered in the object database 16. The virtual space construction unit 15 also arranges the user object 31 in the virtual space, displays the video in the virtual space with the user object 31 as a viewpoint on the screen of the head mounted display 20, and the user 30 receives the virtual reality. Suppose you are experiencing.

  Under the above assumption, as shown in FIG. 6, the user motion detection unit 12 is based on the signals from the motion sensor and infrared sensor of the head mounted display 20 and the video data from the camera 21, and the posture of the user in the real space. The movement and position of the hand are detected (step A1).

  Next, the user motion detection unit 12 reflects the detected posture, hand movement, and position of the user 30 in the real space on the user object 31 (step A2). Furthermore, the user action detection unit 12 detects the action of the user object 31, that is, the action of the user 30 in the virtual space, from the reflection result. (Step A3).

  Next, the user position control unit 13 determines whether or not the action of the user 30 detected in step A3 is a contact with a specific part (for example, an upper surface, a side surface, etc.) of the information object (step A4). .

  As a result of the determination in step A4, when the action of the user 30 detected in step A3 is not a contact with a specific part (for example, an upper surface, a side surface, etc.) of the information object, the user position control unit 13 again determines The operation detection unit 12 is caused to execute Step A1 again.

  As a result of the determination in step A4, when the action of the user 30 detected in step A3 is a contact with a specific part (for example, an upper surface, a side surface, etc.) of the information object, the user position control unit 13 The user object 31 within 40 is moved into the information object that the user 30 has contacted (step A5).

  Next, the object placement unit 14 identifies another information object associated with the information object contacted by the user 30 according to the contact site, and the user object 31 is the identified other information object. It arrange | positions in an information object (step A6).

  Specifically, as illustrated in FIGS. 5A and 5B, the object placement unit 14 has an object having information that is vertically related to the information object contacted by the user 30 according to the contact part, or Identify objects with information in parallel relationship.

  By executing step A6, the user can quickly grasp necessary information. Further, the information processing apparatus 10 repeatedly executes Steps A1 to A6 while the user 30 is experiencing virtual reality.

[Effects of the embodiment]
As described above, in the present embodiment, a user interface is provided that can provide necessary information to a user wearing a head mounted display only by a simple operation of touching an object. For this reason, the user can grasp information intuitively without being forced to perform complicated operations in the virtual space.

  In addition, the information processing apparatus 10 according to the present embodiment can be used, for example, when performing a factory work simulation using virtual reality, or when performing a machine maintenance simulation. In these cases, the user can easily obtain a necessary tool by, for example, contacting an object containing various tools.

  Furthermore, the information processing apparatus 10 according to the present embodiment can also be used in the case of providing different feedback for each option in educational content in which a scene in virtual reality progresses when a user selects an option. In such a case, the information processing apparatus 10 can visualize the options and provide an intuitive interface for the selection operation to the user.

[program]
The program in the present embodiment may be a program that causes a computer to execute steps A1 to A6 shown in FIG. The information processing apparatus 10 and the information processing method in the present embodiment can be realized by installing and executing this program on a computer. In this case, the processor of the computer functions as the object management unit 11, the user motion detection unit 12, the user position control unit 13, the object placement unit 14, and the virtual space construction unit 15, and performs processing.

  The program in the present embodiment may be executed by a computer system constructed by a plurality of computers. In this case, for example, each computer may function as any one of the object management unit 11, the user motion detection unit 12, the user position control unit 13, the object placement unit 14, and the virtual space construction unit 15.

  Here, a computer that realizes the information processing apparatus 10 by executing the program according to the present embodiment will be described with reference to FIG. FIG. 7 is a block diagram illustrating an example of a computer that implements the information processing apparatus according to the embodiment of the present invention.

  As shown in FIG. 7, a computer 110 includes a CPU (Central Processing Unit) 111, a main memory 112, a storage device 113, an input interface 114, a display controller 115, a data reader / writer 116, and a communication interface 117. With. These units are connected to each other via a bus 121 so that data communication is possible. The computer 110 may include a graphics processing unit (GPU) or a field-programmable gate array (FPGA) in addition to the CPU 111 or instead of the CPU 111.

The CPU 111 performs various calculations by developing the program (code) in the present embodiment stored in the storage device 113 in the main memory 112 and executing them in a predetermined order. The main memory 112 is typically a volatile storage device such as a DRAM (Dynamic Random Access Memory). Further, the program in the present embodiment is provided in a state of being stored in a computer-readable recording medium 120. Note that the program in the present embodiment may be distributed on the Internet connected via the communication interface 117.

  Specific examples of the storage device 113 include a hard disk drive and a semiconductor storage device such as a flash memory. The input interface 114 mediates data transmission between the CPU 111 and an input device 118 such as a keyboard and a mouse. The display controller 115 is connected to the display device 119 and controls display on the display device 119.

  The data reader / writer 116 mediates data transmission between the CPU 111 and the recording medium 120, and reads a program from the recording medium 120 and writes a processing result in the computer 110 to the recording medium 120. The communication interface 117 mediates data transmission between the CPU 111 and another computer.

  Specific examples of the recording medium 120 include general-purpose semiconductor storage devices such as CF (Compact Flash (registered trademark)) and SD (Secure Digital), magnetic recording media such as a flexible disk, or CD- An optical recording medium such as ROM (Compact Disk Read Only Memory) can be used.

  Note that the information processing apparatus 10 according to the present embodiment can be realized by using hardware corresponding to each unit, instead of a computer in which a program is installed. Furthermore, a part of the information processing apparatus 10 may be realized by a program, and the remaining part may be realized by hardware.

  Part or all of the above-described embodiments can be expressed by (Appendix 1) to (Appendix 9) described below, but is not limited to the following description.

(Appendix 1)
An object management unit that manages a plurality of objects that hold information in association with each other based on the information that each holds;
A user motion detector that detects a motion of the user in the virtual space, with a head-mounted display that displays the virtual space;
When the user motion detection unit detects the user's touch on the object arranged in the virtual space as the user motion, the user touches the user in the virtual space. A user position controller that moves the object within the object,
When the user moves into the object, an object placement unit that identifies another object associated with the object and places the identified other object in the object;
An information processing apparatus comprising:

(Appendix 2)
An information processing apparatus according to attachment 1, wherein
When the user motion detection unit detects contact by the user to the object arranged in the virtual space, the contact position of the object with the user is detected,
The object placement unit identifies another object associated with the object according to the detected contact position;
An information processing apparatus characterized by that.

(Appendix 3)
An information processing apparatus according to appendix 2, wherein
The object management unit manages each of the plurality of objects based on the vertical relationship and parallel relationship of information held by each of the plurality of objects,
The object placement unit identifies another object that is in a vertical relationship with the object, or another object that is in a parallel relationship with the object, according to the detected contact position.
An information processing apparatus characterized by that.

(Appendix 4)
(A) managing a plurality of objects holding information in association with each other based on the information held by each of the objects;
(B) detecting a user's movement in the virtual space, which is equipped with a head mounted display that displays the virtual space;
(C) In the step (b), when the user's action is detected by the user to contact the object placed in the virtual space, the user in the virtual space is Moving the object into contact with the object;
(D) when the user moves into the object, identifying another object associated with the object, and placing the identified other object in the object;
An information processing method characterized by comprising:

(Appendix 5)
An information processing method according to attachment 4, wherein
In the step (b), when detecting contact by the user to the object arranged in the virtual space, a contact position of the object with the user is detected,
In the step (d), according to the detected contact position, another object associated with the object is specified.
An information processing method characterized by the above.

(Appendix 6)
An information processing method according to appendix 5,
In the step (a), each of the plurality of objects is managed based on the hierarchical relationship and parallel relationship of information held by each of the plurality of objects,
In the step (d), according to the detected contact position, another object that is vertically related to the object or another object that is parallel to the object is specified.
An information processing method characterized by the above.

(Appendix 7)
On the computer,
(A) managing a plurality of objects holding information in association with each other based on the information held by each of the objects;
(B) detecting a user's movement in the virtual space, which is equipped with a head mounted display that displays the virtual space;
(C) In the step (b), when the user's action is detected by the user to contact the object placed in the virtual space, the user in the virtual space is Moving the object into contact with the object;
(D) when the user moves into the object, identifying another object associated with the object, and placing the identified other object in the object;
A program that executes

(Appendix 8)
The program according to appendix 7,
In the step (b), when detecting contact by the user to the object arranged in the virtual space, a contact position of the object with the user is detected,
In the step (d), according to the detected contact position, another object associated with the object is specified.
A program characterized by that.

(Appendix 9)
The program according to attachment 8, wherein
In the step (a), each of the plurality of objects is managed based on the hierarchical relationship and parallel relationship of information held by each of the plurality of objects,
In the step (d), according to the detected contact position, another object that is vertically related to the object or another object that is parallel to the object is specified.
A program characterized by that.

  As described above, according to the present invention, in the virtual space displayed on the head mounted display, the user can intuitively grasp information without being forced to perform complicated operations. The present invention is useful for various systems in which virtual reality is used.

DESCRIPTION OF SYMBOLS 10 Information processing apparatus 11 Object management part 12 User motion detection part 13 User position control part 14 Object arrangement | positioning part 15 Virtual space construction part 16 Object database 20 Head mounted display 21 Camera 30 User 40 Virtual space 41 File box (information object)
42 Objects that hold files (information objects)
31 User object 110 Computer 111 CPU
112 Main Memory 113 Storage Device 114 Input Interface 115 Display Controller 116 Data Reader / Writer 117 Communication Interface 118 Input Device 119 Display Device 120 Recording Medium 121 Bus

Claims (9)

An object management unit that manages a plurality of objects that hold information in association with each other based on the information that each holds;
A user motion detector that detects a motion of the user in the virtual space, with a head-mounted display that displays the virtual space;
When the user motion detection unit detects the user's touch on the object arranged in the virtual space as the user motion, the user touches the user in the virtual space. A user position controller that moves the object within the object,
When the user moves into the object, an object placement unit that identifies another object associated with the object and places the identified other object in the object;
An information processing apparatus comprising: The information processing apparatus according to claim 1,
When the user motion detection unit detects contact by the user to the object arranged in the virtual space, the contact position of the object with the user is detected,
The object placement unit identifies another object associated with the object according to the detected contact position;
An information processing apparatus characterized by that. An information processing apparatus according to claim 2,
The object management unit manages each of the plurality of objects based on the vertical relationship and parallel relationship of information held by each of the plurality of objects,
The object placement unit identifies another object that is in a vertical relationship with the object, or another object that is in a parallel relationship with the object, according to the detected contact position.
An information processing apparatus characterized by that. (A) managing a plurality of objects holding information in association with each other based on the information held by each of the objects;
(B) detecting a user's movement in the virtual space, which is equipped with a head mounted display that displays the virtual space;
(C) In the step (b), when the user's action is detected by the user to contact the object placed in the virtual space, the user in the virtual space is Moving the object into contact with the object;
(D) when the user moves into the object, identifying another object associated with the object, and placing the identified other object in the object;
An information processing method characterized by comprising: An information processing method according to claim 4,
In the step (b), when detecting contact by the user to the object arranged in the virtual space, a contact position of the object with the user is detected,
In the step (d), according to the detected contact position, another object associated with the object is specified.
An information processing method characterized by the above. An information processing method according to claim 5,
In the step (a), each of the plurality of objects is managed based on the hierarchical relationship and parallel relationship of information held by each of the plurality of objects,
In the step (d), according to the detected contact position, another object that is vertically related to the object or another object that is parallel to the object is specified.
An information processing method characterized by the above. On the computer,
(A) managing a plurality of objects holding information in association with each other based on the information held by each of the objects;
(B) detecting a user's movement in the virtual space, which is equipped with a head mounted display that displays the virtual space;
(C) In the step (b), when the user's action is detected by the user to contact the object placed in the virtual space, the user in the virtual space is Moving the object into contact with the object;
(D) when the user moves into the object, identifying another object associated with the object, and placing the identified other object in the object;
A program that executes The program according to claim 7,
In the step (b), when detecting contact by the user to the object arranged in the virtual space, a contact position of the object with the user is detected,
In the step (d), according to the detected contact position, another object associated with the object is specified.
A program characterized by that. The program according to claim 8, wherein
In the step (a), each of the plurality of objects is managed based on the hierarchical relationship and parallel relationship of information held by each of the plurality of objects,
In the step (d), according to the detected contact position, another object that is vertically related to the object or another object that is parallel to the object is specified.
A program characterized by that.

Images