JP4546125B2 - Interface presenting method and interface presenting system - Google Patents

Description

The present invention relates to an interface presentation method and an interface presentation system .

  Information transmission methods via computers include electronic mail, chat, video conferencing, remote control of machines, and the like. In addition, as subjects that input and output information, virtual world and real world objects (humans, computers, robots, electronic devices, characters in a screen, etc.) can be considered.

An interface for inputting and outputting information is provided between the main body and the computer. The interface refers to a hardware or software contact for the above-mentioned main body and a computer to interact. For example, Patent Document 1 describes the technical contents of a method and an interface in which a person and a person communicate with each other via a computer.
JP 2003-84876 A

  2. Description of the Related Art Conventionally, an interface provided in a computer has adopted a function of adapting to each user based on an operation history of a user who operates the computer, a customization request by a user, and the like.

However, the functions for adapting to each user that are adopted in the conventional interface are generality of the information transmission subject, complexity of customization request, systematic adaptation of information presented to the subject, and presenting to the subject There were problems in any of the above points, such as systematic adaptation of the interface, information presented to the subject, and systematic explanation of the interface.
The present invention has been made in view of the above, the virtual world and various entities in the real world can be targeted to, systematically presented interface adaptation is achieved to the principal readily It is an object to provide an interface presentation method and an interface presentation system that can be realized.

In order to solve the above-described problems, the present invention provides a computer that allows a user to operate or browse an interface that allows the user to operate or browse data adapted to the user from data groups representing one or more features of virtual and real world objects and processes. The interface presenting method is a first method in which the computer stores unique information including at least one of profile information, operation information, location information, time information, reason information, and means / method information about the user. The interface comprising: a first step of calling the specific information from a storage means; a frame representing the interface; a scene in which the frame is grouped by at least one of meaning and function; and a film in which the scenes are collected. it is a set of interferon Second storage means over the scan group is stored, based on the specific information, the film suitable for the user, a second step of selecting the scene and the frame, the frame selected in the second step On the other hand, based on the specific information, a third step of performing an adaptive process for the user for determining at least one of the number of objects and processes to be presented at one time, arrangement, presentation order, and presentation time, and the third step The frame selected in the second step is presented regardless of the input information input from the user, or the frame is changed according to the input information and presented in accordance with the result of the adaptive processing in 4 steps are executed .

The present invention also characterized in that an interface presentation system to perform the interface presenting the above-described method.

In the present invention, the first storage means stores specific information about the user, and the second storage means stores an interface group consisting of at least one of the virtual world and real world objects, and is suitable for the user based on the specific information about the user. By performing interface determination and adaptation processing, an interface systematically adapted to the user is presented to the user.

The present invention, a human or machine can be targeted, systematic presentation adaptation of interfaces that achieved in the human or machine can be easily realized as various entities of the virtual world and the real world. Further, the present invention can eliminate a complicated customization request. Furthermore, the present invention can easily provide a systematic human or interface adaptation is achieved in the machine description.

As a result, it can exchange information with a human or machine and the computer using an interface adaptation is achieved to systematically principal.

According to the present invention, the virtual world and the real various entities in the world can be targeted, systematic presentation adaptation of interfaces that achieved in the principal can be easily realized.

The following describes engagement Louis centers face presentation method in an embodiment of the present invention. First, an outline of a system for presenting data and an interface used for implementing this method will be described with reference to FIG.
(System configuration)
The elements constituting this system will be described. This system includes a unique information DB 1, a data presentation function block 2, an interface presentation function block 3, and a bus connecting each functional element.

  The singular information DB 1 is a part that stores singular information regarding the person or the machine 8. Specific information regarding the human or the machine 8 includes specific information regarding the user 14. Singular information is information about a person or a machine itself and information about the environment surrounding them.

  The data presentation function block 2 is a part for presenting data adapted to a human or a machine 8 and includes a data group DB 4, a data filtering device 5, a data organization device 6, and a computer 7. The data group DB 4 stores a data group representing one or more characteristics of virtual world and real world objects and processes (hereinafter referred to as cyber objects and cyber processes). Based on the specific information stored in the specific information DB1, the data filtering device 5 calls the data necessary for the human or machine 8 that is the subject from the data group DB4. The data organization device 6 organizes the data called by filtering based on the specific information about the subject and determines the order of presentation to the subject. The computer 7 presents the data organized and determined in order to the subject, processes information given from the subject regarding the presentation, and feeds back to the unique information DB 1.

  The interface presentation function block 3 is a part that presents an interface adapted to the user 14 in order to operate and browse the presented data. The interface group DB 9, the interface filtering device 10, the interface organization device 11, and the computer 12 The input / output device 13 is configured. The interface group DB 9 stores data related to an interface group including at least one cyber object. The interface filtering device 10 calls data related to an interface suitable for the user 14 from the interface group DB 9 based on the unique information stored in the unique information DB 1. The interface organization device 11 adapts the user 14 to the data called by filtering based on the specific information about the user 14 and organizes an interface suitable for the user 14. The computer 12 supplies control information to the input / output device 13 to provide the organized interface to the user 14, processes information provided from the input / output device 13, and feeds it back to the singular information DB 1. The input / output device 13 presents an interface to the user 14 based on the given control information, and exchanges information between the computer 12 and the user 14.

  These components may exist at positions separated from each other. In that case, these components are connected by a network.

Hereinafter, the unique information stored in the unique information DB1, the data presentation method using the data presentation function block 2, and the interface presentation method using the interface presentation function block 3 will be described in detail.
(Unique information)
Specific information relating to humans or machines includes at least one of profile information, operation information, location information, time information, reason information, and means / method information.

  Profile information represents information unique to humans and machines, and is composed of data items such as name, affiliation, and cognitive ability. The operation information represents information related to the activities of humans and machines, and is composed of data items such as the type of activity currently being performed and the history of the activity. The location information represents geographical information of humans and machines, and is composed of data items such as latitude, longitude, altitude, indoor / outdoor. The time information represents information related to the date and time of humans and machines, and is composed of data items such as the current time in the real world and the virtual world. The reason information represents information related to the motives of human or machine activities, and is composed of data items such as reason, purpose, condition, and situation. Means / method information represents information necessary for humans and machines to perform activities that are not included in profile information, operation information, location information, time information, and reason information. I / O devices, sensors, and networks used It consists of data items such as language.

  These data items are determined by input / output devices operated by humans or other devices such as data input by a robot, GPS, sensors, etc., data collected by a computer such as history and statistics. For example, the name, affiliation, and language used are directly input by humans, the latitude and longitude of the current location are input by GPS, and the activity history is provided by automatic collection by a computer.

  Note that the data items included in each information such as profile information and operation information constituting the unique information, the input method, the update frequency, and the like may be changed according to the application instructed to be executed by the user.

The interface presentation method of the present invention, a human or profile information specific information about the machine, such as operation information, managed by the framework, at any time, refer or by updating, Lee adaptation to human or machine has been achieved Interface can be easily realized.

For example, by referring to the recognition ability of profile information, it is possible to present data and an interface mainly for auditory information to visually impaired people and visual information to visually impaired people. In addition, when referring to the sensor or network used in the method / method information to give a command to check the temperature to the machine, if the temperature sensor is connected, the information can be acquired from the sensor and the network can be used. For example, it is possible to easily obtain instructions suitable for each machine and the environment surrounding them, such as determining the temperature by obtaining from the information on the Internet and comparing both information if both are available.
(Data presentation method)
Hereinafter, will be explained with reference to FIG data presentation method. First, the data group DB4 will be described.

  A cyber process that represents a cyber object or its operation procedure can be represented by one or more features depending on how it is viewed, captured, or expressed. Such characteristics of cyber objects or cyber processes are expressed as individual data. This data is human or machine data.

  That is, one object or process is expressed by a data group that is a collection of human or machine data representing the characteristics of the object or process, and is stored in the data group DB 4.

  This data group includes perception information by the five senses, information on internal structures and materials, information on external structures or parts thereof, usage methods, behavior / actions, control, history information, semantic, spatial, time It is characterized by comprising at least one of information related to cyber objects and cyber processes that are strongly related to each other, and explanatory information depending on culture and language.

  The perception information based on the five senses is, for example, information on food images, sounds, touch, smell, and taste. The information on the internal structure and the material is information that, for example, an automobile is composed of an engine and a chassis. The information on the external structure or a part thereof is, for example, information on how the engine is a part of an automobile and how it is connected to other parts. The information on the method of use, the state of operation / action, control, and history is, for example, information on how to drive the car and how the engine moves. Information on cyber objects and cyber processes that are strongly related to semantic, spatial, and temporal information includes, for example, knives and spoons for forks that are eating tools, or Yamagata, Niigata, and Buri for Fukushima. On the other hand, it is information such as white-eye and hamachi, which are growing processes. The information depending on the culture and language is, for example, information that is a synonym word that sweeps when you vomit but “hak”.

Next, a method of adapting and presenting this data group to a human being or machine as a subject will be described. Hereinafter, processing performed by the data filtering device 5, the data organization device 6, and the computer 7 will be described in order.
(Step 1 data filtering)
The data filtering device 5 accesses the singular information DB 1 and calls the singular information related to the human or the machine 8 that is the target of data presentation. Next, with reference to the called unique information, data related to the cyber object or the cyber process is called from the data group DB 4. At this time, data selection is performed based on the specific information of the person or machine 8 to be presented. The data selected here is output to the data organization device 6.
(Step 2 Data organization)
The data organization device 6 accesses the singular information DB 1 and calls the singular information regarding the person or machine 8 that is the target of data presentation. Next, the presentation order of the data input by the data filtering device 5 is determined with reference to the called unique information. At this time, at least one of the data amount, the presentation order, and the presentation time to be presented at a time is determined based on the specific information of the person or machine 8 to be presented, and the determined content is stored as organization data. . The stored organization data is added to the data input by the data filtering device 5 and is output to the computer 7. Here, the presentation method defined by the organization data includes a static presentation method that ensures presentation according to the organization data regardless of the reaction of the human or the machine 8, a dynamic presentation method that changes the behavior according to the reaction of the human or the machine 8, And any combination thereof.
(Step 3 Data presentation)
The computer 7 presents data with knitting data input by the data knitting device 6 to a human or machine 8. Here, the data presentation method follows the designation of the organization data. When the organization data includes a dynamic presentation method, the computer 7 receives a response from the human or the machine 8 and interactively presents data according to the response. For example, it is possible to fast forward, rewind, and stop data to be presented in response to a human request. It is also possible to wait until a data reception completion signal is transmitted from the machine and transmit the next data. Furthermore, when a human or machine 8 requests more data, it can be reflected in the specific information related to the human or machine 8, and the data filtering device 5 and the data organization device 6 can be requested to present additional data. Data presentation information managed by the computer 7 such as the reaction from the human or machine 8 and the end of data presentation is output to the specific information DB 1 and reflected in the specific information regarding the human or machine 8.

  Here, consider a case where an instruction “lift a pencil” is given. Show the sentence with instructions according to the other party. Various methods can be considered, such as commanding by voice, showing a video, and running a control program. Information about the subject to whom an instruction is given, that is, a subject such as a person or machine 8, is stored in the singular information DB1, and a data group such as a text sentence, a voice, a video, and a program expressing the process of “lifting a pencil” is a data group. Stored in DB4.

  For example, when the subject is a hearing-impaired person, first, the data filtering device 5 refers to the specific information DB 1 and selects a text sentence and a video because information on sounds and programs is unnecessary. Next, the data organization device 6 similarly refers to the specific information DB 1 and determines whether to present the video and the text sentence at the same time, or sequentially present the text sentence and the video in this order. In accordance with the determination, the computer 7 presents a text sentence and a video as a subject. If the subject is a robot, a control program is similarly presented. Since the unique information DB 1 includes usable sensor information, it is possible to give a more accurate instruction by presenting not only the program but also the image data.

As a result of the instruction, the computer 7 can reflect the feedback information from the subject in the singular information DB 1 and perform processing such as sending the next instruction to the subject and retransmitting a more detailed instruction.
(Interface presentation method)
Hereinafter, the interface presentation method of the present invention will be described with reference to FIG. First, the interface group DB 9 will be described.

  A computer interface for browsing, manipulating, and retrieving data groups is composed of at least one cyber object and its operation. Here, the cyber object is a virtual world object that is presented by a virtual reality device such as a monitor, a speaker, a head mounting display, a tactile device with force feedback, or the input / output device 13 included in the input / output device 13. Real world objects such as robots and humans. All the cyber objects that constitute the interface have a unique ID, classification information that indicates an association with each interface, and pointers to data groups that represent the cyber objects stored in the data group DB 4 and the characteristics of their operations.

  That is, the interface information of an arbitrary system is a collection of the above-described information related to cyber objects. One system has at least one interface information, and uses it according to the specific information of the user 14 to be presented. A set of interface information related to one system is called an interface group and is stored in the interface group DB 9.

  In the presented interface, the user interacts with the cyber object via the input / output device 13. The types of processing performed by interaction are classified into three interaction modes. In the “operation mode”, search, browsing, and operation of cyber objects are performed. In the “description mode”, data is called which is an explanation of the cyber object or the interface itself. In the “control mode”, the interface presentation order is controlled.

  At least one of the cyber objects that make up the interface is a screen pet. The screen pet has a function of switching the interaction mode and a function of giving an operation instruction to the user.

Next, an example of an interface that is implemented by the interface presentation method of the present invention. Figure 2 is an image view of an embodiment implemented by the interface presentation method of the present invention. The screen 20 is displayed by a virtual reality device such as a monitor or a head mounting display included in the input / output device 13 (shown in FIG. 1). The screen 20 is, for example, a plane, a three-dimensional space, and a screen combining them, and may be displayed by a plurality of devices included in the input / output device 13.

  The screen 20 includes virtual world objects 21a to 21e. The virtual world object 21e is a screen pet 21e. Further, the sensor 21f, the tactile device 21g, the speaker 21h, the robot 21i, the teacher 21j, and the like are real-world objects and are part of the input / output device 13. That is, the interface of FIG. 2 is configured by virtual world objects 21a to 21e, real world objects 21f to 21j, and their operations.

Figure 3 is a concrete conceptual diagram of the interface implemented by the interface presentation method of the present invention. The screen 30 is displayed by a monitor or virtual reality device included in the input / output device 13 (shown in FIG. 1). A selection field 31 as a virtual world object, a multimedia hieroglyph 32, and a screen pet 33 are displayed in the screen 30. A dog robot 34 as a real-world object is a part of the input / output device 13. That is, the interface of FIG. 3 includes a selection field 31 as a virtual world object, a multimedia hieroglyph 32, a screen pet 33, and a real world object 34 and their operations. Although details of the multimedia hieroglyph 32 are described in Japanese Patent Laid-Open No. 2003-84876, they are omitted, but the contents of the operations executed by the dog robot 34 are expressed. Thus, in order to associate and present the virtual world object and the real world object (shown in FIG. 1), the user 14 makes the presented data and interface and its meaning more intuitive than the conventional interface method. Understandable.

  For example, the interface shown in FIG. 3 allows the user to select “O” or “X” indicating whether the action performed by the dog robot 34 is what the user 14 expects from the selection field 31 using a pointing device or the like. If the action performed by the dog robot 34 is expected, the user 14 selects “◯” in the selection field 31. When “O” is selected, the screen pet 33 performs an operation to congratulate the user 14 on the success. The dog robot 34 executes an action according to an instruction from the data presentation function block 2 as described above. On the other hand, if the action performed by the dog robot 34 is not expected, the user 14 selects “X” in the selection field 31. When “X” is selected, the screen pet 33 performs an operation that regrets the failure of the user 14.

  The user 14 can realize the interface shown in FIG. 4 related to the above-described “operation mode” by using the interaction mode switching function of the screen pet 33. FIG. 4 is an image diagram of an embodiment of an interface according to the operation mode.

  On the screen 41, virtual world objects 42a to 42c representing candidate mail recipients are displayed. The screen 41 (shown in FIG. 1) represents an example in which a joystick is used as the input / output device 13. (See FIG. 1) The user 14 operates the input / output device 13 to search, browse, and operate a cyber object. For example, the user 14 selects any one of the objects 42a to 42c in the virtual world by tilting the joystick up, down, or left. Any one of the selected virtual world objects 42a to 42c is set in the display field 44 as a mail recipient. In the display field 43, a virtual world object representing the sender of the mail is set. The screen 45 is an example in which the selected virtual world object 42 a is set in the display field 46.

  Further, the user 14 can realize the interface shown in FIG. 5 related to the above-described “explanation mode” by using the interaction mode switching function of the screen pet 33. FIG. 5 is an image diagram of an embodiment of the interface according to the explanation mode. For example, the user 14 (shown in FIG. 1) operates the input / output device 13 (shown in FIG. 1) to select the cyber object “drawing a picture” on the screen 56.

  When the cyber object “drawing a picture” on the screen 56 is selected, the interface presentation function block 3 acquires data describing the cyber object from the data presentation block. The screen group 57 is a series of screens that present the data, and is displayed on the input / output device 13. The screen group 57 is selected according to the specific information of the user 14.

  Further, the user 14 can realize the following interface related to the “control mode” by using the interaction mode switching function of the screen pet 33.

  For example, the user 14 can operate the input / output device 13 to cancel the operation performed (Undo), redo the operation (Redo), stop the operation of the cyber object, or the like.

Next, a method of adapting and presenting the interface group to the main user 14 will be described with reference to FIG. Hereinafter, processing performed by the interface filtering device 10, the interface organization device 11, the computer 12, and the input / output device 13 will be described in order.
(Step 1 Interface filtering)
The interface filtering device 10 accesses the singular information DB 1 and calls the singular information related to the user 14 that is the target of the interface presentation. Next, the called unique information is referred to, an interface group is determined based on the operation information and the reason information, and one interface is selected from the interface group based on other information. And the interface information regarding the selected interface is called from the interface group DB 9. At this time, the cyber object included in the interface is selected based on the specific information of the user 14 that is the target of presentation. Here, the interface information that has been selected is output to the interface organization device 11.
(Step 2 interface organization)
The interface organization device 11 accesses the singular information DB 1 and calls the singular information related to the user 14 who is the target of the interface presentation. Next, the presentation order of the interfaces input by the interface filtering device 10 is determined with reference to the called unique information. At this time, at least one of the number of cyber objects to be presented at one time, the order of presentation, and the presentation time is determined based on the specific information of the user 14 that is the target of presentation, and the determined content is stored as organization data. . The stored organization data is added to the interface information input by the interface filtering device 10 and output to the computer 12. Here, the presentation method defined by the organization data is a static presentation method that guarantees the presentation according to the organization data regardless of the reaction of the user 14, a dynamic presentation method that changes the behavior according to the user's reaction, and combinations thereof. Either.
(Step 3 interface presentation)
The computer 12 provides interface information with knitting data input by the interface knitting device 11 to the input / output device 13. At this time, the cyber object to be presented and the interface information representing the operation are distributed to the input / output device 13 in charge of the presentation. For example, among the cyber objects, the virtual object and its operation are displayed on a monitor included in the input / output device 13, or are displayed by being synthesized with a real-world video by a virtual reality device such as a head mounting display. Here, the instruction of the operation of the cyber object is performed based on the data presentation method.

When the organization data includes a dynamic presentation method, the computer 12 receives a response from the user 14 via the input / output device 13 and interactively presents an interface according to the response. Input information sent from the input / output device 13 as a user response includes, for example, numeric information using a numeric keyboard, direction information obtained by defeating a joystick control rod, RF-ID tag information provided by a reader, image information provided by a camera, microphone There is sound information by. The computer 12 has a conversion function for processing such input information. For example, when the interface of FIG. 4 is presented, the conversion function sets the input information “depressed joystick” to “select cyber object 42c” and “depressed joystick left” to “select cyber object 42b”. ”Is processed. The conversion function is stored in the computer 12 and in an external device connected via a network, and a different conversion function can be used for each input device or a combination thereof. Furthermore, the conversion function may be selected in accordance with the interface to be presented and the specific information of the user. Interface presentation information managed by the computer 12 such as the reaction of the user 14 processed by the conversion function and the end of presentation of the interface is output to the singular information DB 1 and reflected in the singular information regarding the user 14.
(Cyber film)
Cyber film will be described as an embodiment representing a data group and an interface group. FIG. 7 is an image diagram for explaining the concept of cyber film.

  Processes representing virtual world and real world objects and procedures representing objects and the like can be represented by one or more features depending on how they are viewed, captured or represented. Such object or process characteristics can be represented by human or machine data. Data representing this feature is called a view 60. That is, an object or process can be represented by a multiview 61 that is a collection of at least one view 60.

  For example, the view 60 in FIG. 7 is tactile information, text, robot control information, sound, still image, and moving image data. 7 includes at least one of tactile information, text, robot control information, sound, still image, and moving image data.

  The multi-view 61 represents the characteristics of an object or process with at least one view 60, but includes a similar view 60. Therefore, the similar view 60 is deleted from the multiview 61. The self-explanatory multi-view 62 is a set of views 60 from which redundancy is deleted by deleting similar views 60 from the multi-view 61.

  The cyber film 64 provides a view 60 in units of cyber frames 63 by giving a partial presentation order to a set of views 60 included in the self-explanatory multi-view 62. Cyber film 64 can represent more features of objects and processes by increasing the number of views 60. The cyber frame 64 includes at least one view 60, and a partial presentation order can be provided between the views 60. The cyber frames 64 are grouped according to meaning and function. This group is called the cyber scene.

  Examples of cyber films include a cyber communication film (hereinafter referred to as CCF) and a cyber interface film (hereinafter referred to as CIF).

  CCF expresses the user's intention (including instructions) from at least one of the following points: “subject”, “predicate”, “place”, “time”, “reason / condition / purpose”, and “means / method” Thus, a means for easily transmitting a user's will to a human or a machine via a computer is provided.

  The CIF provides an interface for the user 14 to easily search, view, or operate the multiview 62 via various input / output devices 13 (shown in FIG. 1).

  Furthermore, cyber film will be described in detail. FIG. 8 is an image diagram of an example of a cyber film.

  The cyber film group 65 is a set of cyber films 64 representing programs (hereinafter referred to as subsystems) such as an OS and applications that the user has instructed to execute. For example, the cyber film group 65 has subsystems F1 to F6. The cyber film 64 is an expansion of the subsystem F2.

The cyber film 64 is composed of a series of cyber frames 83 composed of at least one. The series of cyber frames 83 is divided into cyber scenes 82a to 82e representing functions and meanings of the subsystem. The cyber scenes 82a to 82e are associated with cyber frame groups 83a to 83e including at least one frame 83. For example, a series of cyber frames 83 are divided into cyber scenes 82a to 82e representing functions and meanings such as login and mail browsing.

  As the data group DB or the interface group DB, various types of databases and film systems such as a relational database, a hierarchical database, a network database, and a distributed database can be used. The film DB as one embodiment manages the cyber film 64, the cyber scenes 82a to 82e, and the cyber frame 83 as shown in FIG. 8 by using, for example, a film table, a scene table, and a frame table. FIG. 9 is a configuration diagram of an example of a film table. FIG. 10 is a configuration diagram of an example of a scene table. FIG. 11 is a configuration diagram of an example of a frame table.

  The film table of FIG. 9 has a film ID, a scene ID, and a scene number as data items. The film ID represents an identifier of the cyber film. The scene ID represents a cyber scene identifier. The scene number represents the number of the cyber scene in the cyber film represented by the data item “scene ID”. In other words, the film table represents the relationship between the cyber film and the cyber scene.

  The scene table in FIG. 10 has a scene ID, a frame ID, and a frame number as data items. The scene ID represents a cyber scene identifier. The frame ID represents an identifier of the cyber frame. The frame number represents the number of the cyber frame in the cyber scene that is represented by the data item “frame ID”. In other words, the scene table represents the relationship between the cyber scene and the cyber frame.

  The frame table of FIG. 11 has a frame ID, a cyber object ID, position information, action information, and user input correspondence processing information as data items. The frame ID represents an identifier of the cyber frame. The cyber object ID represents an identifier of the cyber object. The position information represents the position of the cyber object represented by the data item “cyber object ID”.

  The action information represents the action of the cyber object represented by the data item “cyber object ID”. Further, the user input handling process information represents a handling process for an input from the user in each of the three interaction modes.

  In other words, the frame table represents the relationship between the cyber frame, the cyber object, and the corresponding processing for the input from the user. Cyber film is a representation format of an interface for performing processing such as search, browsing, and operation on cyber objects.

  Cyberframe has several cyber objects. In each cyber frame, the objects in the virtual world among the cyber objects are arranged in a three-dimensional space or a pseudo three-dimensional space in which planes are combined in multiple layers. Real-world objects are placed in the real world as input / output devices. In other words, the cyber frame is a framework in which the screen having the virtual world object (shown in FIG. 2) and the real world input / output device are combined. These cyber objects accept information input from the user by a mouse, touch panel, joystick, tactile device, or the like included in the input / output device, and output information to the user according to the operation or state of the cyber object.

  FIG. 12 is a diagram for explaining an example of cyber film adaptation processing. The film DB manages cyber frames for each subsystem (F1 to F6) and cyber scenes (S1 to S5).

  In step S1, a cyber film (F2) is selected in accordance with a program such as an OS or application that the user has instructed to execute, and a series of cyber scenes S1 to S3 and The scene ID of S5 is acquired.

  In step S2, the cyber scenes S2, S3 and S5 are selected from the cyber scenes S1 to S3 and S5 included in the cyber film (F2) based on the user environment model as one embodiment of the unique information DB1, and the cyber scenes S2, S2 and S5 are selected. A series of frame IDs are obtained from the scene table using the scene IDs of S3 and S5.

  In step S3, a cyber frame group is selected from the frame table as shown in FIG. 11 using the frame ID acquired in step S2. In the cyber frame groups respectively associated with the cyber scenes S2, S3, and S5, necessary cyber frames are selected based on the user environment model, and become cyber scenes S2 ', S3', and S5 '.

  Next, a series of cyber object IDs associated with the frame ID of the selected cyber frame is acquired for each cyber frame from the frame table. The acquired cyber object ID is selected for each cyber frame based on the user environment model. For example, when two cyber object IDs are selected based on the user environment model, the process proceeds to step S4, and the cyber frame 91 that can display five cyber objects is changed to the cyber frame 92 that can display two cyber objects.

  Next, the cyber objects associated with each cyber frame are adjusted. For example, the arrangement of the cyber objects displayed on the cyber frames 91 and 92 changes according to the input / output device 13 that can be used by the computer 12.

  FIG. 13 is an image diagram of an example of a cyber frame in which the arrangement of cyber objects to be presented on the cyber frame is changed according to the input / output device. For example, when a mouse is used as the input / output device 13 (shown in FIG. 1), cyber objects are arranged as shown in FIG. Thereby, the cyber object can be selected by instructing the cyber object with the mouse pointer.

  When a numeric keypad is used as the input / output device 13, cyber objects are arranged along the numeric keypad layout as shown in FIG. 13B, and the corresponding key numbers are displayed on the cyber objects. Thereby, the cyber object can be selected by inputting a number with the numeric keypad.

  When a joystick is used as the input / output device 13, cyber objects are arranged on the circumference as shown in FIG. 13C, and an arrow indicating the direction from the center of the circle to the cyber object is placed on each cyber object. Deploy. Thereby, it is possible to select a cyber object by tilting the joystick in the direction of the arrow. The association between the input data input to the input / output device 13 and the cyber object is also automatically changed based on the data of the cyber frame adapted for the user environment model.

  Through the above processing, it is possible to adapt and provide the user with a cyber frame that is compatible with various input / output devices 13 and that can perform operations such as selection intuitively. As a result, the interface adapted to the user enables the user to operate the computer while exchanging information with the program.

In addition, when data and an interface are expressed using a cyber film, it has the following advantages.
(Simplified interface)
By using CIF, it is possible to reduce the number of objects constituting the functions and interfaces presented at one time.

In a conventional system such as an OS or various applications, the number of objects such as buttons and menus displayed on the screen at a time has dramatically increased as the functions become higher. CIF incorporates the concept of time and divides the operation into steps for each cyberframe or cyberscene. Since only the necessary cyber objects are presented to the user in each step, the complexity of the interface can be suppressed even if the system is highly functional. It is also possible to design an interface that looks like a natural image as a whole, rather than a set of buttons and menus that are mechanically designed. Therefore, it is possible to easily use a presentation method used in a movie or a television drama, and to provide an interface that allows a person unfamiliar with a computer to operate without resistance. Furthermore, it is possible to provide an intuitive interface by effectively combining real world and virtual world objects and processes.
(Control of interface presentation)
The interface shown in FIG. 6 related to the “control mode” described above can be expressed by CIF. FIG. 6 is an image diagram of an embodiment of an interface related to the control mode.

  For example, the user 14 (shown in FIG. 1) operates the input / output device 13 (shown in FIG. 1) to control interface presentation such as fast-forward, rewind, skip, play, and stop for each cyberframe or each cyberscene. It can be carried out. For example, in FIG. 6, it is possible to freely move between a series of cyber frames and cyber scenes as indicated by arrows in FIG. 6 by presentation control such as fast forward, rewind, skip, playback, and stop.

For this reason, a complicated keyboard is not necessary, and the computer can be operated with a simple input device having the same function as a controller of a portable audio device having fast-forward, rewind, play, and stop buttons. In addition, by using unique information, it is possible to adapt the interface and realize a simpler input device, for example, an operation using only one button.
(Cyber object search)
With CIF, it is possible to search for all cyber objects constituting the subsystem.

For example, by referring to the film table, scene table, and frame table, a list of cyber objects used in an arbitrary scene and frame can be searched. As a result, it is possible to search for a cyber object outside the currently presented frame and present an explanation.
(Adaptation to presentation target)
By using the CIF, it is possible to realize fine adaptation to the presentation target when presenting data.

First, the CIF performs interface adaptation processing according to the presentation target, and presents an operation environment suitable for the user and the usage environment thereof, for example, the type of the input / output device. Furthermore, since the data provided by the interface is provided in a cyber film format, the data itself can be subjected to adaptive processing. This allows adaptation by both the data and the interface that operates.
(Efficient communication)
By using CCF, it is possible to increase the efficiency of communication. In conventional natural languages and programming languages, the semantic separation of “subject”, “predicate”, “location”, etc. was performed by a human or machine that is the main subject of understanding. However, in the CCF, it is not necessary to perform a semantic separation process by specifying semantic separation at the time of expression. In addition, partial information that is semantically separated, for example, information that represents a “predicate” is expressed in a cyber film, and appropriate data is presented according to the subject that understands. Furthermore, if more information is needed for understanding and execution of instructions, additional information can be extracted from the film DB. As a result, the CCF can simplify the process of understanding the intention (including instructions) expressed by a human or a machine, and reduce the burden.

  In the past, it was necessary to learn and use different languages and grammars such as English, Japanese, or programming languages according to the object to be communicated. However, in CCF, by expressing information from the above viewpoints such as “subject”, “predicate”, and “location”, it is possible to convey intentions (including instructions) not only to people but also to various objects such as robots and computers. It is. As a result, the intention can be expressed in the same format regardless of the transmission target, and the CCF can reduce the burden of the expression.

In order to realize these functions, specific information regarding the transmission target is essential. This unique information can be changed and added as necessary, and the use range of the CCF can be easily expanded or limited.
(Effective knowledge acquisition)
All cyber objects and cyber processes used in CIF and CCF can have a cyber film description. In addition, cyber objects and cyber processes used in cyber films can be explained by other cyber films. At this time, the cyber film is adapted according to the specific information of the user, and further provides a description that combines real-world objects and virtual-world objects and their processes. The interface for browsing, searching, and manipulating the cyber film is expressed in CIF, and an interface suitable for each user and the usage environment is provided. Therefore, the user can effectively acquire knowledge such as acquisition of an operation method.

  In addition, this invention is not limited to above-described embodiment, In the range which does not deviate from the summary of this invention, a various change can be added.

  For example, although data is cyber film, a conventional GUI may be used as an interface for browsing / searching / manipulating the data, or CIF may be used as an interface for browsing / searching / manipulating data other than cyber film. As a database for storing unique information, data, and interfaces, an associative memory device, a neural network device, or the like may be used. By applying the present invention, application software such as document creation software, web browser, mail, chat, instant message, image processing software, program development software, network service, operating system, robot or other hardware, etc. In addition, it is possible to have a highly adaptive function for various humans and machines and their environments.

  In addition, hardware, software, a network, a combination of these, and any other means can be used as specific means of the constituent elements (including functional blocks) for realizing the above-described embodiment. Furthermore, the functional blocks may be combined and integrated into one functional block or apparatus. Further, the function of one functional block may be realized by cooperation of a plurality of functional blocks or devices. Furthermore, the functional blocks may be connected to each other via a network and can communicate with each other.

Hereinafter, an embodiment for implementing the interface presentation method of the present invention are described with reference to figures based on the following embodiments which utilize the CIF or CCF.

(System configuration)
FIG. 14 is a system configuration diagram of an embodiment of the interface presentation system according to the present invention. In the interface presentation system 100 in FIG. 14, a client 101, a film DB server 102, an authentication server 103, and a content server 104 are connected via a network 108 such as a LAN (Local Area Network) or the Internet.

  The client 101 is a personal computer or the like operated by the user, and holds a user environment model DB 105. The client 101 may acquire information stored in the user environment model DB 105 from the film DB server 102, the authentication server 103, or the like. The client 101 includes an input / output device that exchanges information with a human or a machine. The client 101 uses the CIF to control the input / output device, thereby realizing various interfaces or data that are adapted to the user or machine.

  The film DB server 102 manages a film DB in which CIF data is stored. Further, the film DB server 102 transmits CIF data to the client 101 in response to a request from the client 101 or the like. The authentication server 103 is used for user authentication, for example. The content server 104 manages content data such as images, sounds, and tactile data used in the CIF.

Note that the system configuration of the interface presentation system 100 is an example, and a system configuration in which the user environment model DB 105 is held other than the client 101, a system configuration in which the film DB is held in the client 101, a system configuration in which the authentication server 103 is omitted, and a film Various forms such as a system configuration in which the DB server 102, the authentication server 103, and the content server 104 are realized by a WWW server are conceivable.
(Hardware configuration)
FIG. 15 is a hardware configuration diagram of an embodiment of the client. The client 101 includes devices such as an input device 111, an output device 112, a drive device 113, an auxiliary storage device 115, a memory device 116, an arithmetic processing device 117, and a network connection device 118 that are connected to each other via a bus B. It is configured. Note that the various devices constituting the client 101 of FIG. 15 may be accommodated in one casing or may be distributed and accommodated in a plurality of casings.

  The input device 111 includes a mouse, a touch panel, a joystick, a camera, an RF-ID tag reading device, and the like, and is used to input various types of information. The output device 112 includes a computer-controllable device such as a monitor, a speaker, and a robot, and is used for outputting various types of information. For example, the output device 112 is used to display a frame including a cyber object or the like. The network connection device 118 includes a modem, a terminal adapter, and the like, and is used for connecting to the network 108.

  The interface presentation program related to the processing of the client 101 is provided by distributing the recording medium 114, downloading from the network 108, or the like. The recording medium 114 is a recording medium that records information optically, electrically, or magnetically, such as a CD-ROM, a flexible disk, a magneto-optical disk (MO), or the like, a ROM, a flash memory, or the like. Various types such as an electrically recording semiconductor memory can be used.

  When the recording medium 114 in which the interface presentation program is recorded is set in the drive device 113, the interface presentation program is installed in the auxiliary storage device 115 from the recording medium 114 through the drive device 113. The interface presentation program downloaded from the network 108 is installed in the auxiliary storage device 115 via the network connection device 118.

  The auxiliary storage device 115 stores the installed interface presentation program and also stores necessary files, data, and the like. The memory device 116 reads the interface presentation program from the auxiliary storage device 115 and stores it when the computer is started up. The arithmetic processing unit 117 implements various functions according to the interface presentation method of the present invention in accordance with the interface presentation program stored in the memory device 116.

As in the case of data presentation program, the processing unit 117 according to the data presented program stored in the memory device 116 to implement various functions of the data presentation method.
(Functional configuration)
FIG. 16 is a functional configuration diagram of an embodiment of the client. In addition to the interface presentation program according to the present invention, the client 101 is configured to include a communication control unit 131, a CIF processing unit 132, a computer 134, an input / output device 135, and a user environment model DB 105 realized by an OS or an application.

  The communication control unit 131 controls communication with other network devices connected via the network 108, such as the film DB server 102, the authentication server 103, and the content server 104. The CIF processing unit 132 receives CIF data from the film DB server 102 or the content server 104 via the communication control unit 131 in response to a request from the computer 134. The CIF processing unit 132 adapts CIF data according to the subsystem and the user environment model DB 105. The CIF processing unit 132 corresponds to the data filtering device 5, the data organization device 6, the interface filtering device 10, and the interface organization device 11.

  The user environment model DB 105 is updated in response to a request from the computer 134. For example, a user or an administrator can manually update the user environment model DB 105 by operating the input / output device 135 and requesting the computer 134 to set or update the user environment model DB 105.

  Further, the computer 134 determines the user environment according to the user operation history based on the data item “activity history” of the user environment model DB 105 as described later, the system state based on the data item “platform”, or information from various sensors. The model 104 may be updated automatically.

The computer 134 is a subsystem program that the user has instructed to execute. The computer 134 receives a frame selected by the CIF processing unit 132 and realizes an interface based on the frame. Through the interface, the computer 134 exchanges information with the user or machine via the input / output device 135 and performs necessary processing.
(Client processing)
Hereinafter, the processing of the client 101 illustrated in FIG. 16 will be described with reference to a flowchart. FIG. 17 is a flowchart of an example of a client process using the CIF according to the present invention.

  When the computer 134 receives a login request from the user via the input / output device 135, the computer 134 proceeds to step S <b> 10 and performs login processing to the authentication server 103. The user ID and password used for the login process are acquired via the input / output device 135, for example. The input / output device 135 mentioned here includes a floppy (registered trademark) disk drive, hardware storing data such as an RF-ID tag, a device for reading biometrics, and the like. The computer 134 logs in the authentication server 103 by transmitting the user ID and password to the authentication server 103 via the communication control unit 131.

  After logging in, the computer 134 proceeds to step S11 and updates data in the user environment model DB 105. If the user environment model DB 105 is not held, the computer 134 acquires information stored in the user environment model DB 105 from the film DB server 102, the authentication server 103, and the like.

  FIG. 18 is a configuration diagram of an example of the user environment model DB. The user environment model DB 105 classifies information about users from the viewpoint of so-called 5W1H classification (WHO, WHAT, WHERE, WHEN, WHY, HOW).

  The 5W1H classification (WHO) represents profile information and includes data items related to the profile information. The 5W1H classification (WHAT) represents operation information and includes data items related to the operation information. The 5W1H classification (WHERE) represents location information and includes data items related to the location information.

  The 5W1H classification (WHEN) represents time information and includes data items related to the time information. The 5W1H classification (WHY) represents reason information and includes data items related to the reason information. The 5W1H classification (HOW) represents means / method information and includes data items related to the means / method.

  Note that the data items constituting the user environment model DB 105 may be changed according to the subsystem that the user has instructed to execute. A specific example of the user environment model DB 105 will be described with a flowchart described later.

  In step S11, data that needs to be updated is updated at each login, such as the address of the data item included in the location information and the time of the data item included in the time information, among the data in the user environment model DB 105. When the process of step S11 ends, the computer 134 proceeds to step S12, and if the next process to be performed is not a logout process, the process of loop 1 is started. In loop 1 processing, the computer 134 proceeds to step S13, and requests the CIF processing unit 132 to perform CIF adaptation processing.

  The CIF processing unit 132 receives CIF data from the film DB server 102 and the content server 104 via the communication control unit 131 in response to a request from the computer 134. The CIF processing unit 132 performs CIF adaptation processing according to the subsystem that the user has instructed to execute and the user environment model DB 105. That is, in step S13, the CIF data is filtered and organized for adaptation to the user. Details of the adaptation process in step S13 will be described later.

  In step S14, the CIF processing unit 132 renders the frame selected in the adaptive processing in step S13 with view data. The CIF processing unit 132 transmits the rendered frame to the computer 134.

  In step S 15, the computer 134 presents the rendered frame received from the CIF processing unit 132 to the user via the input / output device 135. Next, when the frame received from the CIF processing unit 132 has a cyber object, the computer 134 determines whether or not there is an action of the cyber object.

  If it is determined that there is a cyber object action (Yes in S15), the computer 134 proceeds to step S16 and executes the action of the cyber object. In step S16, by executing the action of the cyber object, information can be presented to the user by the action or state of the object in the virtual world or the real world.

  After executing the action of the cyber object, the computer 134 proceeds to step S17. If it is determined that there is no cyber object action (no in S15), the computer 134 proceeds to step S17. In step S17, if all the actions are not completed, the processing of loop 2 is started.

  In loop 2, the computer 134 determines whether or not there is an input from the user. If it is determined that there is an input from the user (Yes in S18), the computer 134 proceeds to Step S19 and performs a process corresponding to the input from the user. If it is determined that there is no input from the user (none in S18), the computer 134 proceeds to step S20, returns to step S17 if all actions are not completed, and returns to step S21 if all actions are completed. Proceed to Even when the process of step S19 ends, the computer 134 proceeds to step S21.

  In step S21, the computer 134 updates the data in the user environment model DB 105 in accordance with the processing corresponding to the input from the user performed in step S19. The user environment model DB 105 is updated with data. In step S21, among the data stored in the user environment model DB 105, data that needs to be updated is updated every time processing corresponding to the input from the user is performed, such as the activity history of the data item included in the operation information. .

  When the process of step S21 ends, the computer 134 proceeds to step S22, returns to step S12 if the next process to be performed is not a logout process, and proceeds to step S23 if the next process to be performed is a logout process. In step S <b> 23, the computer 134 performs logout processing on the authentication server 103 and then ends the processing.

  Next, as an example of the user environment model DB 105, the CIF adaptation processing in step S13 will be described in detail using the user environment model DB 105 used in a communication support system (hereinafter simply referred to as FCS).

  FIG. 19 is a flowchart of an example of CIF adaptation processing. First, the CIF processing unit 132 receives CIF data from the film DB server 102 and the content server 104 via the communication control unit 131. In step S31, the CIF processing unit 132 uniquely determines the film ID by the subsystem that the user has instructed to execute, and acquires a series of scene IDs from the film table as shown in FIG. 9 using the film ID.

  In step S32, the CIF processing unit 132 selects a scene from the scene table as shown in FIG. 10 using the scene ID acquired in step S31. Further, the CIF processing unit 132 selects a scene according to the user environment model DB 105, and acquires a series of frame IDs from the scene table as shown in FIG. 10 using the scene ID of the scene.

  Hereinafter, a cyber film communication system (hereinafter referred to as FCS) will be described as an example. The FCS is a system that uses the CCF as an intention transmission medium and uses the CIF as an interface for browsing, searching, and operating data. FCS is composed of a series of subsystems that support the lives of so-called information-impaired persons such as disabled persons and elderly persons by using computers. The FCS is represented by a cyber film group 65 (shown in FIG. 8). The FCS subsystem includes, for example, a communication subsystem, a conversation assistance subsystem, a multimedia word dictionary subsystem, and a hieroglyph learning subsystem. Each subsystem is represented by a cyber film 64 (shown in FIG. 8).

  FIG. 20 is a configuration diagram of an example of a user environment model DB used in FCS. The user environment model DB 105 used in the FCS is configured with the 5W1H classification (WHO, WHAT, WHERE, WHEN, WHY, HOW) as described above.

  The 5W1H classification (WHO) represents profile information, such as name, affiliation, date of birth, gender, role in the system, type of disability, developmental stage, language proficiency, family, friends, teachers and hobbies / preferences. Consists of data items. The 5W1H classification (WHAT) represents operation information, and is composed of data items such as types of activities performed using the system and history of activities. The 5W1H classification (WHERE) represents location information and includes data items such as an address, a use area, a use place, and a use facility. The 5W1H classification (WHEN) represents time information, and is composed of data items such as time, day of the week, date, weekday / holiday, schedule (timetable), and virtual time in the system.

  The 5W1H classification (WHY) represents reason information and is composed of data items such as a reason for performing an operation and weather. The 5W1H classification (HOW) represents means / method information and includes data items such as a platform, a network, an interaction mode, a used input / output device, a used sensor device, and a used language.

  Progressing to step S33 following step S32, the CIF processing unit 132 selects a frame from the frame table as shown in FIG. 11 using the frame ID acquired in step S32. The CIF processing unit 132 selects a frame according to the user environment model DB 105, and acquires a series of cyber object IDs associated with the frame ID of the frame from the frame table. In step S34, the CIF processing unit 132 selects a series of cyber object IDs corresponding to the user environment model DB 105 from the series of cyber object IDs acquired in step S33.

  In step S35, the CIF processing unit 132 acquires position information, action information, and user input corresponding processing information associated with the selected series of cyber object IDs, and acquires the position information, action information, and user input corresponding processing information. Use to adjust parameters such as the speed of animation and the presence or absence of sound effects.

  In the CIF adaptation processing shown in the flowchart of FIG. 19, the CIF and cyber objects associated with the CIF can be selected and adjusted based on the user environment model DB 105. The selection of the scene ID, frame ID, and cyber object ID corresponding to the user environment model DB 105 performed in steps S32, S33, and S34 is related to the data items of the user environment model DB 105 as shown in FIG. This is done according to the table shown.

  For example, in the table of FIG. 21, data items of the user environment model DB 105 referred to when performing CIF adaptation processing are represented by “◯”. In the selection of the scene ID according to the user environment model DB 105 performed in step S32, the affiliation, the role in the system, the type of failure, the stage of development, the type of activity performed using the system, the use place, and the use in the user environment model DB 105. Data items such as facilities, schedule (timetable), virtual time in the system, reason for operation, network and input / output devices used are referenced.

  That is, the selection of the scene ID, frame ID, and cyber object ID is uniquely performed according to the data item of the user environment model DB 105 that is referred to when performing the CIF adaptation process.

  In this way, by referring to the user environment model DB 105 that manages profile information, operation history, various sensor information, platforms, and the like in the framework of the 5W1H classification, the CIF processing unit 132 selects the CIF adapted for each user. It can be easily realized.

  Next, CIF adaptation processing in a program (hereinafter referred to as F-Mail) that transmits and receives a message consisting of a single sentence (hereinafter referred to as single text mail) will be described as an example of the communication subsystem constituting the FCS.

  For example, in the case where the data item “type of activity performed using the system” in the user environment model DB 105 is “communication” and the data item “reason for performing operation” is “to create a simple text mail”, F-Mail The CIF adaptation process will be described with reference to FIG.

  The film DB server 102 manages frames for each subsystem (F1 to F6) and scene (S1 to S5). In step S 1, the CIF processing unit 132 selects CIF (F 2) according to the data item “type of activity performed using the system” and the data item “reason for performing operation” in the user environment model DB 105. The CIF processing unit 132 acquires a scene ID of a series of scenes S1 to S3 and S5 from the film table using the film ID of CIF (F2). This CIF (F2) corresponds to the subsystem that the user has instructed to execute. The process of step S1 is a process corresponding to step S31 of FIG.

  In step S2, the CIF processing unit 132 selects scenes S1 to S3 and S5 using the scene ID acquired in step S1. Then, the CIF processing unit 132 converts the scenes S1 to S3 and S5 to the scenes S2, S2 according to the data items “role in the system”, “type of failure”, “development stage”, and “use input / output device” of the user environment model DB 105. Select S3 and S5. The CIF processing unit 132 acquires a series of frame IDs from the scene table using the scene IDs of the scenes S2, S3, and S5. The process of step S2 is a process corresponding to step S32 of FIG.

  In step S3, the CIF processing unit 132 selects a frame from the frame table using the frame ID acquired in step S2. The CIF processing unit 132 selects a necessary frame from the frame group associated with the scenes S2, S3, and S5 according to the data items “type of failure”, “development stage”, etc. in the user environment model DB 105, and Let S2 ', S3' and S5 '.

  For example, in the scene S3, it is assumed that the usage cells of the hieroglyph are “when”, “where”, “who”, “what are you doing” from the data item “development stage” of the user environment model DB 105 In this case, four panel frames are selected from the frame group associated with the scene S3 to become the scene S3 ′. The CIF processing unit 132 acquires a series of cyber object IDs associated with the frame ID of the selected frame for each frame from the frame table. The process of step S3 is a process corresponding to step S33 of FIG.

  Next, the CIF processing unit 132 determines the necessary cyber object from the cyber object ID acquired in step S3 according to the data item “type of failure”, “development stage”, “used input / output device”, etc. in the user environment model DB 105. ID is selected for each frame. For example, when the user can cope with two things at a time from the “development stage” of the user environment model DB 105, the process proceeds to step S 4, and a frame that can display two cyber objects from a frame 91 that can display five cyber objects as standard. Change to 92. The process of step S4 is a process corresponding to step S34 of FIG.

  Next, the CIF processing unit 132 adjusts the cyber object associated with each frame. The CIF processing unit 132 changes the background of the frame according to the data items “weather” and “time” of the user environment model DB 105, and frames 91 and 92 according to the data items “hobby / preference” and “use place”. When the cyber object to be displayed is changed, the range of kanji used is changed according to the data item “language proficiency”, or the user's pulse is fast according to the data item “used sensor information”, animation Etc. can be changed to a slow speed, or the arrangement of cyber objects to be displayed on the frames 91, 92, etc. can be changed according to the data item “input / output device used”.

  Here, the processing of the client 101 shown in the flowcharts of FIGS. 17 and 19 will be described more specifically. In step S10, a login process is performed. In step S11, the user environment model DB 105 is updated. A part of the user environment model DB 105 updated in step S11 is configured as shown in FIG. 22, for example.

  In step S13, CIF adaptation processing is performed. The film table is configured as shown in FIG. 23, for example. In step S31 of FIG. 19, since the data item “type of activity performed using the system” in the user environment model DB 105 is “mail transmission / reception”, the CIF with the film ID “1” is selected.

  This CIF includes seven or more scenes, but since the data item “role in system” of the user environment model DB 105 is “user”, the scenes of scene numbers “2” to “7” are filtered. Access is allowed. When the data item “role in the system” is “superuser”, access is permitted to more scenes in addition to the scene numbers “1” to “7”.

  Further, since there is no “printer” in the data item “used input / output device” of the user environment model DB 105, further filtering is performed and access to the scene numbers “2”-“5”, “7” is finally permitted. The In step S32, since the data item “reason for operation” in the user environment model DB 105 is “to create a mail”, the scene with the scene ID “100003” corresponding to the scene number “3” for creating the mail is selected. Is done.

  The scene table is configured as shown in FIG. 24, for example. Since the data item “development stage” of the user environment model DB 105 is “Understanding the concept of subject, predicate, place, and time, and the number of options is up to nine”, the frames of frame numbers “1” to “6” are filtered. Access to is allowed. Further, filtering is further performed according to the data item “virtual time in the system” of the user environment model DB 105 to select a frame.

  In step S34, the cyber object associated with the frame ID of the frame selected in step S33 is selected according to the user environment model DB 105. In step S35, after adjusting the parameters associated with the cyber object ID of the cyber object selected in step S34, the frame rendered in step S14 of FIG. 17 is displayed.

  For example, when the frame having the frame number “3” is selected in step S33, the cyber object associated with the frame ID of the frame having the frame number “3” is selected from the frame table in step S34, and step S14 in FIG. The frame of FIG. 25 is displayed.

  The frame in FIG. 25 is configured to include a cyber object representing a screen pet 251, a sender 252, a destination 253, a hieroglyph 254, a background 255, an answer candidate launch object 256, and an answer candidate 257. The cyber object representing the answer candidate 257 included in the frame of FIG. 25 includes a question for selecting a subject, a cyber object already included in the hieroglyph 254 included in the data item “activity history”, and frequency information. Based on the above, a large number of cyber objects are selected by filtering and organization.

  In step S35, the following parameters are adjusted for the cyber object selected in step S34. For example, in step S35, different screen pets can be selected based on the data item “gender” in the user environment model DB 105. Further, the language of the speech can be selected based on the data item “language used” in the user environment model DB 105.

  In addition, a name and a photo to be displayed can be selected based on the data item “name” in the user environment model DB 105. In addition, based on the data item “development stage” of the user environment model DB 105, the unusable cell can be displayed in gray, and the currently selected cell can be highlighted. Further, the background can be selected based on the data items “weather” and “time” in the user environment model DB 105.

  Different cyber objects can be selected based on the data item “gender” in the user environment model DB 105. Further, the number of cyber objects to be displayed at a time can be selected based on the data items “development stage” and “used input / output device” of the user environment model DB 105. Further, the launch speed of answer candidates from the answer candidate launch object 256 can be selected based on the pulse of the data item “use sensor information” in the user environment model DB 105 and the reaction speed of “activity history”.

  For example, when the response speed is slow, the launch speed of answer candidates from the answer candidate launch object 256 is decreased. Even if the response speed is high, the response candidate launch rate from the answer candidate launch object 256 is reduced when the pulse increases. Also, a cyber object can be selected based on the data item “name” in the user environment model DB 105.

  Further, for all cyber objects having a text display portion, whether to display Japanese or English based on the data items “language of use” and “language proficiency” of the user environment model DB 105 is displayed. You can choose how many years of kanji you want to use.

  As a result of the above CIF adaptation processing, frame rendering processing is performed in step S14 of FIG. At this time, the position where the cyber object is displayed follows the position information of the frame table. The position information may be directly specified by coordinates, or may be dynamically determined according to the display order of cyber objects determined based on the user environment model DB 105, the number of cyber objects displayed at a time, and the like. .

  In the case of the frame in FIG. 25, the position information of the cyber object representing the screen pet 251, the sender 252, the destination 253, the hieroglyph 254, the background 255, the launch object 256 of the answer candidate is directly specified by coordinates. The position information of the cyber object representing the answer candidate 257 is dynamically determined according to the display order of the cyber objects, the number of cyber objects to be displayed at one time, and the like.

  In addition, since processing for selecting a cyber object differs depending on the input / output device 135, a marker (for example, an arrow or a number) indicating a method for selecting a cyber object is added when performing rendering processing.

  FIG. 26 is an image diagram of an example of a frame representing an arrangement of cyber objects when the data item “used input / output device” is a one-button switch as shown in FIG. In FIG. 26, cyber objects representing answer candidates are displayed one by one in order, and the cyber object displayed in the frame is selected when the one button switch is pressed.

  A frame representing the arrangement of cyber objects when the data item “used input / output device” is a mouse or a touch panel is as shown in FIG. Further, the frame representing the arrangement of the cyber objects when the data item “used input / output device” is the numeric keypad is as shown in FIG. Further, the frame representing the placement of the cyber object when the data item “used input / output device” is a joystick is as shown in FIG. Details of the above-described hieroglyphs, multimedia words, and screen pets are described in Japanese Patent Application Laid-Open No. 2003-84876, and thus description thereof is omitted.

  Next, CIF adaptation processing in the multimedia word dictionary subsystem constituting the FCS will be described. The Multimedia Word Dictionary Subsystem is a computerized picture dictionary for young children, elementary school students, disabled children, and people with aphasia. The multimedia word dictionary subsystem aims to increase the user's vocabulary by making the meaning intuitively and deeply understood by multi-view explanation using cyber objects.

  For example, when the data item “type of activity performed using the system” in the user environment model DB 105 is “vocabulary learning”, the CIF adaptation processing of the multimedia word dictionary subsystem is performed as shown in FIG. 5 described above. In each frame, the position of the cyber object is determined according to the frame ID and the cyber object ID of the frame table, and rendering is performed. The processing so far corresponds to, for example, steps S10 to S14 in FIG.

  Also, the presence / absence of a cyber object action is determined according to the action information in the frame table, and if there is an action, the cyber object action according to the action information is executed. The processing so far corresponds to, for example, steps S15 and S16 in FIG.

  In the frame 53 of FIG. 5, an action in which a girl draws a picture and an action in which a mother makes a telephone call are executed. Each cyber object refers to user input response processing information in the frame table in response to a click from the user, and performs different response processing depending on the interaction mode. The processing so far corresponds to, for example, the processing in steps S17 to S20 in FIG.

  In addition, in the frame 51 of FIG. 5, when the house is clicked in the operation mode, frames 53 to 55 representing the inside of the house are displayed. However, when the house is clicked in the explanation mode, the explanation about the house by the multi-view is displayed. A frame 52 to be presented is displayed.

  The explanation by multiview includes explanation by a plurality of hieroglyphs described in JP-A-2003-84876. For example, when a girl is clicked on the frame 53, the hieroglyph “Girl is drawing a picture with crayons” is called as an explanation. From this hieroglyph, it is the frame 56 that discriminates hieroglyph cells that can be used according to the data item “development stage” of the user environment model DB 105 and presents a cyber object.

  In the frame 56, among “Girl (Who)”, “Drawing a picture (What)”, and “Crayon (How)”, “Girl (Who)” and “Drawing (What)” Presents. Further, in the frame 56, a panel frame group 57 representing a detailed description of each cyber object can be called as necessary.

  According to the first embodiment, all cyber objects of the interface presentation system can be described by multiview. Further, by performing CIF filtering according to the user environment model DB 105, an adapted interface can be presented to the user or machine. Also, by organizing the CIF according to the user environment model DB 105, an interface incorporating the concept of time can be presented to the user or machine. Furthermore, the interface presented to the user or machine can be simplified using the user environment model DB 105.

(Multimedia hieroglyph)
First, a multimedia hieroglyph as an example of CCF will be described. The multimedia hieroglyph used in the present invention is for expressing a sentence in a so-called 5W1H data format. When (time), where (location), who (subject), what (predicate), why (reason) ), Composed of six hieroglyph cells representing how (means).

  A multimedia word as an example of a cyber object will be described. A multimedia word is a generic term for views consisting of human or machine data such as tactile information, text, robot control information, sound, still images, and moving images, and is multifaceted with various forms of things and movement processes. It represents. The multimedia word is composed of, for example, a multi-view in which at least one view is combined to intuitively understand a thing or a process of movement.

  The details of the multimedia hieroglyph and the multimedia word are described in Japanese Patent Application Laid-Open No. 2003-84876, and thus the description thereof is omitted. FIG. 28 is an image diagram of an example of a multimedia hieroglyph in which multimedia words are arranged. In the multimedia hieroglyph of FIG. 28, multimedia words representing when (time), where (location), who (subject), what to do (predicate), why (reason), and how (means) are arranged. That makes sense.

  The multimedia hieroglyph as shown in FIG. 28 represents a sentence according to a user's intention to transmit in a so-called 5W1H data format. Therefore, by making the user express the multimedia hieroglyph as shown in FIG. 28 and using the multi-view that composes the multimedia words arranged in the multimedia hieroglyph, the user's intention is easily made to cyber objects such as users and machines. Can be communicated to.

  When the cyber object is a human being, it is possible to easily convey the user's will by providing multi-views such as still images, moving images, and sounds as well as text. If the cyber object is a machine, more information can be presented by providing multi-views such as tactile information, text, robot control information, sound, still images, and videos, and the user's intention can be effectively executed. Is possible.

  For example, in the case of a multimedia hieroglyph representing the subject “robot”, the predicate “moving”, and the place “person A's place”, information about the person A is given to the robot in multiple views such as photographs, voiceprints, and dressed appearances. Thus, it is possible to improve the accuracy of specifying the person A by using various sensing technologies of the robot and to surely execute the user's intention.

  The learning system of the present invention creates a sentence using a multimedia hieroglyph and performs an action corresponding to the sentence by an object (for example, a toy robot, a teacher or a friend) existing in the real world or an object (for example, a virtual world) 2D or 3D character on the screen), the virtual world and the real world are associated with each other to allow the user to learn.

  In particular, users with intellectual disabilities can understand the meaning of sentences intuitively and have the ability to communicate, because cyber objects that exist in the real world or virtual world operate according to sentences created using multimedia hieroglyphs. It is possible to improve effectively.

  Hereinafter, a learning system using multimedia hieroglyphs will be described as an example of an interface presentation system according to the present invention. The system configuration of the learning system using multimedia hieroglyphs is the same as the system configuration of FIG.

The client 101 implements various interfaces that are systematically adapted to the user or machine by controlling the input / output device 13 using the CCF. The film DB server 102 manages a film DB in which CCF data is stored. In addition, the form DB server 102 transmits CCF data to the client 101 in response to a request from the client 101 or the like. The authentication server 103 is used for user authentication. The content server 104 manages content data such as images, sounds, and tactile data used in the CCF.
(Functional configuration)
FIG. 29 is a functional block diagram of an embodiment of a client. In addition to the interface presentation program according to the present invention, the client 101 is configured to include a communication control unit 131, a computer 134, an input / output device 135, a CCF processing unit 351, and a user environment model DB 105 realized by an OS or an application. In the functional configuration of the client 101 in FIG. 29, the same parts as those in the functional configuration diagram in FIG. The CCF processing unit 351 corresponds to the data filtering device 5, the data organization device 6, the interface filtering device 10, and the interface organization device 11.

  The CCF processing unit 351 includes a frame processing unit 352, a multimedia hieroglyph processing unit 353, and a multimedia hieroglyph presentation unit 354. The frame processing unit 352 receives CCF data from the film DB server 102 and the content server 104 via the communication control unit 131 in response to a request from the computer 134. The frame processing unit 352 adapts the CCF data according to the subsystem that the user has instructed to execute and the user environment model DB 105.

  The multimedia hieroglyph processing unit 353 performs processing related to creation of the multimedia hieroglyph when the multimedia hieroglyph is included in the frame adapted by the frame processing unit 352. The multimedia hieroglyph presentation unit 354 performs processing related to the presentation of the multimedia hieroglyph when the created multimedia hieroglyph is included in the frame. Details of processing related to creation or presentation of the multimedia hieroglyph will be described later.

The computer 134 is a subsystem program that the user instructs to execute. The computer 134 receives a frame from the CCF processing unit 351 and realizes an interface based on the frame. Through the interface, the computer 134 exchanges information with the user or machine via the input / output device 135 and performs necessary processing.
(Client processing)
Hereinafter, the processing of the client 101 illustrated in FIG. 29 will be described with reference to a flowchart. FIG. 30 is a flowchart of an example of a client process that implements a learning method using multimedia hieroglyphs. Note that the flowchart of FIG. 30 is the same as the flowchart of FIG.

  The processing in steps S100 and S101 is the same as that in steps S10 and S11 in FIG. The computer 134 proceeds to step S102, and starts loop 1 processing if the next processing to be performed is not logout processing. In loop 1 processing, the computer 134 proceeds to step S103, and requests the frame processing unit 352 of the CCF processing unit 351 to perform adaptive processing of CCF.

  The frame processing unit 352 receives CCF data from the film DB server 102 and the content server 104 via the communication control unit 131 in response to a request from the computer 134. Next, the frame processing unit 352 performs CCF adaptation processing according to the subsystem that the user has instructed to execute and the user environment model DB 105. That is, in step S103, the CCF data is filtered and organized for adaptation to the user. The processing in steps S104 to S106 is the same as that in steps S14 to S16 in FIG.

  In step S107, if all the actions have not been completed, the process proceeds to step S108, and the processing of loop 2 is started. In step S108, the computer 134 determines whether or not there is an input from the user.

  If it is determined that there is an input from the user (Yes in S108), the computer 134 proceeds to Step S109 and performs a response process for the input from the user. For example, if the input from the user relates to creation of a multimedia hieroglyph, the multimedia hieroglyph processing unit 353 performs processing related to creation of the multimedia hieroglyph. If the input from the user is related to the presentation of the multimedia hieroglyph, the multimedia hieroglyph presentation unit 354 performs processing related to the presentation of the multimedia hieroglyph.

  If it is determined that there is no input from the user (no in S108), the computer 134 proceeds to step S110, and determines whether all actions have been completed. If all actions have not been completed, the computer 134 returns to step S107, and if all actions have been completed, the process proceeds to step S111. Also, when the process of step S109 ends, the computer 134 proceeds to step S111. In addition, since the process of step S111-S113 is the same as that of step S21-S23 of FIG. 17, description is abbreviate | omitted.

  By performing the processing of the flowchart of FIG. 30, the hieroglyph creation processing for creating the multimedia hieroglyph as shown in FIG. 28 and the hieroglyph presentation processing for presenting the multimedia hieroglyph can be performed as follows.

  In the process of steps S100 to S106, a frame including a multimedia hieroglyph without a multimedia word is presented to the user. In the processing of steps S107 to S110, multimedia words are arranged in the multimedia hieroglyphs according to the input from the user. In the process of step S111, the data in the user environment model DB 105 is updated according to the arrangement of the multimedia words with respect to the hieroglyph cell.

  For example, in the process of step S111, the response to the input from the user performed in step S109 such as the data item “activity history” included in the operation information and the “virtual time in the system” of the data item included in the time information. Data that needs to be updated is updated according to the processing.

  The client 101 creates a multimedia hieroglyph in which multimedia words are arranged by repeatedly performing the above steps S102 to S112. It is assumed that after the multimedia hieroglyph having the multimedia words arranged therein is created, the start of the hieroglyph presentation process is instructed by the user in the process of step S109.

  In the process of step S111, the data in the user environment model DB 105 is updated according to the hieroglyph presentation process instructed in step S109. In the processing of the next steps S102 to S106, for example, a frame including the multimedia hieroglyph is presented to the user according to the subsystem that the user has instructed to execute and the user environment model DB 105.

  In the processing of steps S105 and S106, the action of the cyber object is executed as described later according to the contents of the created multimedia hieroglyph. In the processes of steps S107 to S110, the user inputs whether or not the action of the cyber object is an action expected by the user. In the process of step S111, the data in the user environment model DB 105 is updated according to the input content.

  Thereafter, the hieroglyph presentation process may be repeated according to the content input in step S109, the next hieroglyph creation process may be started, or the user may be logged out.

  Next, the procedure of the learning method using multimedia hieroglyphs realized by the processing of the flowchart of FIG. 30 will be described. FIG. 31 is a flowchart of an embodiment of a learning method using multimedia hieroglyphs.

  In step S131, a hieroglyph creation process for describing the operation of the cyber object in a multimedia hieroglyph is performed according to an instruction from the user. In step S132, the hieroglyph presentation process is performed in which the multimedia hieroglyph described in step S131 is presented to the user, the contents of the multimedia hieroglyph are read out by voice, and the action of the cyber object is executed according to the contents of the multimedia hieroglyph. Done.

  In step S133, an evaluation process is performed to confirm to the user whether or not the action executed by the cyber object in step S132 is what the user expected. In the evaluation process of step S133, it is acquired via the input / output device 135, for example, whether or not the action executed by the cyber object was expected.

  In step S134, it is determined whether or not to perform the hieroglyph presentation process in step S132 again depending on whether or not the action executed by the cyber object is what the user expected. If it is determined that the hieroglyph presentation process is to be performed again (YES in S134), the process proceeds to step S132. On the other hand, when it is determined that the hieroglyph presentation process is not performed again (NO in S134), the process proceeds to step S135.

  In step S135, the user is confirmed whether to end or continue the process, and if there is an instruction to end the process (YES in S135), the process ends. If there is an instruction to continue the process (NO in S135), the process returns to step S131 to continue the process.

  Hereinafter, the processing of steps S131 to S133 in the flowchart of FIG. 31 will be described in detail with reference to the drawings. In the hieroglyph creation process in step S131, a multimedia hieroglyph is described in which the action of the cyber object is described by one or more multimedia words. Each cyber object has its own behavior.

  Therefore, in order to create a multimedia hieroglyph that can be executed by a cyber object, an association between the cyber object and a multimedia word that can be executed by the cyber object is required in advance. FIG. 32 is a configuration diagram of an example of a word list table in which cyber objects and multimedia words are associated with each other.

  The word list table in FIG. 32 is configured to include data items such as a cyber object ID, a word ID, and 5W1H information. The cyber object ID is an identifier of the subject cyber object. The word ID is an identifier of a multimedia word that can be used in an operation command for a cyber object. The 5W1H information is information indicating which hieroglyph cell of the multimedia hieroglyph can the multimedia word be used.

  The hieroglyph creation processing in step S131 is performed according to the procedure shown in the flowchart of FIG. 33 using, for example, the word list table of FIG. FIG. 33 is a flowchart of an example of hieroglyph creation processing.

  In step S141, the multimedia hieroglyph processing unit 353 of the CCF processing unit 351 acquires a cyber object ID of a cyber object that can be used according to the user environment model DB 105 and the frame being displayed.

  In step S141, the multimedia hieroglyph processing unit 353 acquires cyber object IDs of usable cyber objects according to the data item “used input / output device” included in the means / method information. In step S142, the multimedia hieroglyph processing unit 353 analyzes the multimedia words used in the multimedia hieroglyph, and obtains the word ID of the multimedia word and the position information of the hieroglyph cell in which the multimedia word is arranged. get.

  The multimedia hieroglyph processing unit 353 searches the word list table using the cyber object ID acquired in step S141, the word ID acquired in step S142, and the position information of the hieroglyph cell, and can be executed with the multimedia hieroglyph being created. Get the cyber object ID of the correct cyber object.

  In step S143, the multimedia hieroglyph processing unit 353 uses the multimedia that can be used from the word list table based on the cyber object ID acquired in step S142 and the position information of the hieroglyph cell specified by the user via the input / output device 135. The word ID of the word is acquired, and the multimedia word of the word ID is presented to the user via the input / output device 135.

  In step S144, the user operates the input / output device 135 to select the multimedia word presented in step S143. The user selects the presented multimedia word by at least one method of clicking with a pointing device, presenting an actual object, miniature or card (reading an RF-ID tag, photographing with a camera, etc.) and voice input. . The multimedia hieroglyph processing unit 353 acquires the word ID of the multimedia word selected by the user via the input / output device 135.

  In step S145, the multimedia hieroglyph processing unit 353 places the multimedia word selected by the user in the hieroglyph cell designated by the user. Proceeding to step S146, the multimedia hieroglyph processing unit 353 determines whether or not the multimedia hieroglyph has been completed. If it is determined that the multimedia hieroglyph has been completed (YES in S146), the multimedia hieroglyph processing unit 353 ends the hieroglyph creation process. On the other hand, when it is determined that the multimedia hieroglyph is not completed (NO in S146), the multimedia hieroglyph processing unit 353 returns to step S142 and continues the hieroglyph creation process.

  The hieroglyph creation process of FIG. 33 will be described with a specific example. For example, it is assumed that cyber object IDs representing “toy dog”, “screen pet penguin”, and “teacher” are registered in the cyber object ID of the word list table. That is, the multimedia hieroglyph subject can be any of the multimedia words representing “toy dog”, “screen pet penguin”, and “teacher”.

  Also, in the word ID of the word list table, multimedia words representing “running”, “barking”, and “handing” corresponding to the toy dog are registered, and “word” is associated with the screen pet penguin “ Multimedia words representing “run”, “jump”, and “turn somersault” are registered, and multimedia words representing “run”, “jump”, and “sing” are registered in correspondence with the teacher.

  Hieroglyph creation processing for creating a multimedia hieroglyph composed of a subject and a predicate using such a word list table is as follows. In step S141, the multimedia hieroglyph processing unit 353 acquires toy dogs, screen pet penguins, and teacher cyber object IDs as usable cyber objects.

  Proceeding to step S142, the multimedia hieroglyph processing unit 353 proceeds to step S143 because the hieroglyph cell of the multimedia hieroglyph is blank. In step S143, the multimedia hieroglyph processing unit 353 uses the multimedia object ID acquired in step S141 as a usable multimedia word to run, bark, do, jump, flip over, sing a word The ID is acquired from the word list table, and the multimedia word of the word ID is presented to the user via the input / output device 135.

  In step S144, it is assumed that the user operates the input / output device 135 and selects “jump” from the multimedia words presented in step S143. The multimedia hieroglyph processing unit 353 acquires the word ID of the multimedia word representing “jump” selected by the user via the input / output device 135.

  In step S145, the multimedia hieroglyph processing unit 353 places the multimedia word representing “jump” selected by the user in the hieroglyph cell designated by the user. Proceeding to step S146, the multimedia hieroglyph processing unit 353 determines that the multimedia hieroglyph is not completed (NO in S146), and returns to step S142.

  In step S142, since the multimedia word representing “jump” is arranged as the multimedia hieroglyph predicate, the multimedia hieroglyph processing unit 353 represents the usable cyber object ID and “jump” as the predicate. The word list table is searched on the condition of the cyber object ID that can take the multimedia word, and the cyber object ID of the screen pet penguin and the teacher is acquired.

  In step S143, the multimedia hieroglyph processing unit 353 presents multimedia words representing penguins and teachers of screen pets as subject candidates to the user via the input / output device 135 based on the cyber object ID acquired in step S142. For example, when a multimedia word representing “singing” is arranged as a predicate of the multimedia hieroglyph, the multimedia word representing the teacher is presented to the user via the input / output device 135 in step S143.

  Proceeding to step S144, it is assumed that the user has selected a screen pet penguin from the multimedia words presented at step S143. The multimedia hieroglyph processing unit 353 acquires the word ID of the multimedia word representing the penguin of the screen pet selected by the user via the input / output device 135.

  In step S145, the multimedia hieroglyph processing unit 353 places a multimedia word representing the penguin of the screen pet selected by the user in the hieroglyph cell specified by the user. Proceeding to step S146, the multimedia hieroglyph processing unit 353 determines that the multimedia hieroglyph has been completed (YES in S146), and ends the hieroglyph creation process.

  As described above, according to the hieroglyph creation process of FIG. 33, a multimedia hieroglyph “a penguin jumps” can be created. The multimedia hieroglyph can describe the operation of the cyber object in detail by arranging the multimedia word in the hieroglyph cell other than the predicate.

  Next, the hieroglyph presentation process in step S132 will be described. In the hieroglyph presentation process in step S132, the multimedia hieroglyph described in the hieroglyph creation process in step S131 is displayed to the user, and the contents of the multimedia hieroglyph are read out by voice. Let the action be executed.

  The hieroglyph presentation process in step S132 is performed, for example, according to the procedure shown in the flowchart of FIG. FIG. 34 is a flowchart of an example of hieroglyph presentation processing. In step S151, the multimedia hieroglyph presentation unit 354 of the CCF processing unit 351 displays the multimedia hieroglyph created by the hieroglyph creation process in step S131 on the input / output device 135, for example, and presents it to the user. Read out aloud.

  In step S152, the multimedia hieroglyph presenting unit 354 guides the user to the cyber object that will perform an action by using at least one method in which a screen pet on the screen as an example of the cyber object uses voice or screen display. Proceeding to step S153, if the actions of all the cyber objects described in the multimedia hieroglyph are not completed, the multimedia hieroglyph presenting unit 354 proceeds to step S154.

  In step S154, the multimedia hieroglyph presentation unit 354 acquires the cyber object ID of the multimedia word arranged as the subject of the multimedia hieroglyph. In step S155, the multimedia hieroglyph presentation unit 354 searches the motion command conversion function table using the cyber object ID acquired in step S154, and acquires the motion command conversion function.

  FIG. 35 is a configuration diagram of an example of the operation command conversion function table. The operation command conversion function table of FIG. 35 is configured to include data items such as a cyber object ID and an operation command conversion function. The cyber object ID is an identifier of a cyber object that transmits an operation command. The operation command conversion function is a function for converting a multimedia hieroglyph into an operation command.

  Proceeding to step S156, the multimedia hieroglyph presenting unit 354 converts the multimedia hieroglyph into a cyber object motion command, as will be described later, using the motion command conversion function acquired in step S155. Here, the types of operation commands converted according to the cyber object are as follows.

  For example, when the cyber object is a human, the action command is for causing the device capable of expressing the content of the action to be executed by the human by at least one of visual, auditory, and tactile information to express the content of the action. When the cyber object is a machine or device such as a robot or a miniature that can be operated from a computer, the operation command is a signal (for example, infrared rays, Bluetooth, etc.) for operating the device or the device. In addition, when the cyber object is an object such as a 2D or 3D character on the screen, the operation command is a computer command for operating the object (for example, in the case of a video performer, a computer command for acquiring and playing the corresponding video clip). )

  In step S157, the multimedia hieroglyph presentation unit 354 transmits the motion command converted in step S156 directly or indirectly to the cyber object. Proceeding to step S158, the cyber object executes an action in accordance with the operation command received directly or indirectly.

  Then, the process proceeds to step S159, and if the action of all cyber objects described in the multimedia hieroglyph has not ended, the multimedia hieroglyph presentation unit 354 returns to step S153 and continues the hieroglyph presentation process. The multimedia hieroglyph presentation unit 354 terminates the hieroglyph presentation process if the actions of all the cyber objects described in the multimedia hieroglyph 321 have been completed.

  The hieroglyph presentation process of FIG. 34 will be described with a specific example. For example, the hieroglyph presentation process of the multimedia hieroglyph 32 shown in FIG. 3 for giving an operation command to a toy dog 34 that can be operated by infrared rays from a computer will be described as an example. In the multimedia hieroglyph 32, a multimedia word representing “dog” as a subject and a multimedia word representing “aruku” as a predicate are arranged.

  The hieroglyph presentation process for presenting such a multimedia hieroglyph 32 is as follows. In step S151, the multimedia hieroglyph presenting unit 354 of the CCF processing unit 351 sends the multimedia hieroglyph 32 “The dog is there” and the Japanese text “The dog is walking” to the input / output device 135, for example. By displaying it, it is presented to the user, and the screen pet 33 reads out the content “Dog walks” by voice.

  Proceeding to step S152, the multimedia hieroglyph presentation unit 354 causes the screen pet 33 to give an instruction "Look at the dog" by at least one of voice or screen display, and to the toy dog 34 that will perform an action from now on. Guide users. When the user is wearing a head mounting display, guidance using augmented reality is possible, such as the screen pet 33 actually moving near the toy dog 34 and pointing.

  Proceeding to step S153, the multimedia hieroglyph presentation unit 354 proceeds to step S154 because the actions of all the cyber objects described in the multimedia hieroglyph 32 have not been completed. In step S154, the multimedia hieroglyph presentation unit 354 acquires the cyber object ID of the toy dog 34 according to the multimedia word arranged as the subject of the multimedia hieroglyph 321.

  In step S155, the multimedia hieroglyph presentation unit 354 searches the motion command conversion function table using the cyber object ID of the toy dog 34 acquired in step S154, and operates the motion command conversion function fdog (h ) To get. Here, h represents the multimedia hieroglyph 32.

  For example, it is assumed that the operation command conversion function fdog (h) is a function for retrieving an operation command from the operation command correspondence table according to the hieroglyph ID of the multimedia hieroglyph 32. FIG. 36 is a configuration diagram of an example of the operation command correspondence table. The operation command correspondence table is configured to include data items such as a hieroglyph ID and an operation command. The hieroglyph ID is an identifier of the multimedia hieroglyph 32. The action command is information for issuing an action command to the cyber object.

  In this case, the process proceeds to step S156, and the multimedia hieroglyph presentation unit 354 acquires an infrared signal for walking the toy 34 of the dog from the hieroglyph ID of the multimedia hieroglyph 32 that “the dog walks”. In step S157, the multimedia hieroglyph presentation unit 354 transmits the infrared signal acquired in step S156 to the dog toy 34 via the input / output device 135, for example.

  In step S158, the dog toy 34 receives an infrared signal from the input / output device 135, for example, and executes an action of “walking” in response to the infrared signal. In step S159, the multimedia hieroglyph presenting unit 354 terminates the hieroglyph presenting process because the actions of all the cyber objects described in the multimedia hieroglyph 32 have been completed. The hieroglyph presentation process of FIG. 34 can actually cause the dog toy 34 to walk in accordance with the contents of the multimedia hieroglyph 32 “the dog walks”.

  Then, in the evaluation process of step S133 in FIG. 31, it is possible to confirm with the user whether or not the action executed by the cyber object in step S132 was what the user expected using the evaluation screen in FIG.

  As described above, the evaluation process in step S133 can provide clear feedback of success or failure to the user. With this evaluation process, people who are close to the user (for example, teachers and parents) can participate in learning, such as asking "How was it?" The effect can be expected.

  According to the second embodiment, the user can learn by associating the virtual world with the real world, so that the learning effect can be improved.

1 is a schematic diagram of an example system that presents data and interfaces. FIG. It is a conceptual diagram of one embodiment of an interface that is implemented by the interface presentation method of the present invention. It is a specific image diagram of interfaces implemented by the interface presentation method of the present invention. It is an image figure of an example of the frame switched to the operation mode. It is an image figure of an example of the frame switched to description mode. It is an image figure showing control of the cyber film at the time of control mode. It is an image figure for demonstrating the concept of a cyber film. It is an image figure of an example of the interface using the monitor contained in an input / output device. It is a block diagram of an example of a film table. It is a block diagram of an example of a scene table. It is a block diagram of an example of a frame table. It is a figure for demonstrating an example of the adaptation process of a cyber film. It is an image figure of an example of the frame which changed arrangement | positioning of the cyber object displayed on a flame | frame according to an input / output device. It is a system configuration | structure figure of one Example of the interface presentation system by this invention. It is a hardware block diagram of one Example of a client. It is a function block diagram of one Example of a client. It is a flowchart of an example of the process of the client using CIF by this invention. It is a block diagram of an example of a user environment model. It is a flowchart of an example of an adaptive process of CIF. It is a block diagram of an example of the user environment model utilized by FCS. It is the table showing the relationship between the data item of a user environment model, and the adaptive process of CIF. It is a block diagram showing the specific example of the user environment model. It is a block diagram showing the specific example of the film table. It is a block diagram showing the specific example of the scene table. It is an image figure showing the example of the frame. It is an image figure of an example of the frame showing arrangement | positioning of the cyber object when the data item "use input / output device" is a one button switch. It is an image figure of an example of a one button switch. It is an image figure of an example of the multimedia hieroglyph which has arranged the multimedia word. It is a function block diagram of one Example of a client. It is a flowchart of an example of a process of the client which implement | achieves the learning method using a multimedia hieroglyph. It is a flowchart of one Example of the learning method using a multimedia hieroglyph. It is a block diagram of an example of the word list table which linked | related the cyber object and the multimedia word. It is a flowchart of an example of a hieroglyph creation process. It is a flowchart of an example of a hieroglyph presentation process. It is a block diagram of an example of an operation command conversion function table. It is a block diagram of an example of an operation command corresponding | compatible table.

Explanation of symbols

1 Specific information DB
2 Data presentation function block 3 Interface presentation function block 4 Data group DB
5 Data filtering device 6 Data organization device 7,12 Computer 8 Human or machine 9 Interface group DB
DESCRIPTION OF SYMBOLS 10 Interface filtering apparatus 11 Interface organization apparatus 13 Input / output apparatus 14 User 60 View 61,62 Multi view 63 Frame 64 Cyber film 100 Interface presentation system 101 Client 102 Film database server (film DB server)
103 Authentication Server 104 Content Server 105 User Environment Model Database (User Environment Model DB)
108 Network 111 Input Device 112 Output Device 113 Drive Device 114 Recording Medium 115 Auxiliary Storage Device 116 Memory Device 117 Arithmetic Processing Device 118 Network Connection Device 131 Communication Control Unit 132 CIF Processing Unit 134 Computer 135 Input / Output Device 351 CCF Processing Unit 352 Frame Processing Part 353 Multimedia Hieroglyph Processing Part 354 Multimedia Hieroglyph Presentation Part B Bus

Claims (9)

An interface presentation method for presenting an interface that allows a user to operate or browse data adapted to a user from a group of data representing one or more characteristics of virtual and real world objects and processes, comprising:
The computer is
A first step of calling the singular information from first storage means storing singular information comprising at least one of profile information, operation information, location information, time information, reason information and means / method information relating to the user;
Second storage means storing an interface group, which is a set of the interfaces, including a frame representing the interface, a scene in which the frame is grouped by at least one of meaning and function, and a film in which the scenes are collected. A second step of selecting the film, the scene and the frame suitable for the user based on the specific information;
An adaptive process for the user that determines at least one of the number of objects and processes to be presented at one time, the arrangement, the presentation order, and the presentation time for the frame selected in the second step based on the specific information. A third step to perform;
According to the result of the adaptive process in the third step, the frame selected in the second step is presented regardless of the input information input from the user, or the frame is changed according to the input information. fourth interface presentation method and executes and presenting Te. The data is
Perception information by the five senses,
Information about internal structures and materials,
Information about the external structure or part of it, and
Information on usage, behavior / action, control, history,
Information about virtual and real world objects and processes that are closely related to semantic, spatial and temporal,
Information about virtual and real world objects and processes with opposite semantic, spatial and temporal meanings;
2. The interface presenting method according to claim 1, comprising at least one piece of explanation information depending on culture and language.   2. The interface presentation according to claim 1, wherein the computer updates the specific information as needed based on information acquired from at least one of an input / output device, a sensor, and a network specified in the means / method information. Method.   2. The interface presenting method according to claim 1, wherein the computer updates the specific information as needed according to a user operation history, a system state, and information from a sensor.   The interface selects data corresponding to the input / output device from the third storage means in which the data group is stored based on the input / output device usable by the computer, and uses the data to select the input / output device. 2. The interface presenting method according to claim 1, wherein control is performed to exchange information between the computer and the user.   2. The interface presenting method according to claim 1, wherein the virtual world and real world objects constituting the interface have classification information in the interface group and are searchable. Before Kikari virtual world and the real world object has a pointer to the data groups stored in the third storing means for representing the feature, in response to a request from the user, was adapted to the user The interface presentation method according to claim 1, wherein an explanation by data is provided.   The interface expresses the user's will from at least one of the subject, predicate, place, time, reason / condition / target, means / method, and the computer is represented according to the user's will expressed from the viewpoint. The interface presenting method according to claim 1, further comprising: exchanging information with the person or machine via the interface.   9. An interface presentation system that executes the interface presentation method according to claim 1.

Images