JP2000224565A - Method and system for controlling sequential presentation - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a method and a system for controlling a sequential presentation by changing it in accordance with a tacit feed-back from a viewer. SOLUTION: In the method, a sensory reaction including the tacit feed-back from the viewer 204 is detected during the presentation at first. Then the feed- back is transmitted to a presentation sequence controller 216. Then the feed-back is characterized from the viewpoint of a way to progress the presentation in the controller 216. Finally, the presentation is changed optimally in accordance with how the viewer understand it and react to it in real-time in response to the characterization of the tacit feed-back.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention generally relates to a viewer participation system. In particular, the present invention relates to methods and systems that utilize sensory feedback to control a presentation sequence. More particularly, the present invention relates to a method and system for providing implicit viewer feedback to change control a sequential presentation.

[0002]

BACKGROUND OF THE INVENTION Interactive / modifiable movies require conventional input by the viewer by asking the viewer to answer a question or then select an instruction to the scene. The input request occurs undesirably. I mean,
Be aware of specific choices and select them (for example, "Selection A
Or pointing to option B ") and more prudent physical work (eg, pointing). Thus, those methods of viewer interaction with the presentation will add to the viewer's presentation experience. This problem is especially disturbing when an interactive presentation attempts to entertain viewers.To get the maximum entertainment benefit from a cinematic entertainment presentation, the viewer must It must be engrossed enough, for the reasons mentioned above, this is very difficult to achieve using current interactive cinema technology.

From the foregoing, it can be seen that there is a need for a method and system that utilizes sensory feedback from viewers to control a sequential presentation. Such a method and system, if implemented, would be useful by allowing the presentation sequence to be modified in a way that best suits the viewer's implicit response to the presentation.

[0004]

One object of the present invention is to provide an improved viewer participation scheme.

Another object of the present invention is to provide a method and system that utilizes sensory feedback to control a presentation sequence.

It is still another object of the present invention to provide a method and system for providing implicit viewer feedback that controls the change of a sequential presentation.

[0007]

The above and other objects are to provide:
This is achieved as described below. A method and system for controlling a sequential presentation by modifying the sequential presentation according to implicit feedback from a viewer is disclosed.
First, during the presentation, sensory responses including implicit viewer feedback from the viewer are detected.
Next, the implicit viewer feedback is sent to the presentation sequence controller. Thereafter, the implicit viewer feedback is characterized in terms of how the presentation can proceed within the presentation sequence controller.
Finally, the presentation is modified in response to the characterization of the implicit feedback, so that it can be adapted in real time and optimally how the viewer understands and reacts to the presentation, so that it can be adapted sequentially.

[0008]

DETAILED DESCRIPTION OF THE INVENTION The method and system of the present invention tie implicit viewer feedback in the form of both spontaneous and involuntary reactions to a presentation sequence. Spontaneous reactions that can be used in embodiments of the present invention include:
Includes (but is not limited to) eye movements, hand gestures, respiratory rate, etc. The involuntary response utilized by the present invention can include changes in blood pressure, heart rate, sweating, and the like.
Some measurable physiological responses are due to a combination of spontaneous and involuntary responses to the presentation. An example of such a parameter is eye movement. In particular, gaze tracking is very useful for obtaining extensive information about both the viewer's spontaneous (conscious) and involuntary (unconscious) reactions to a presentation.
Gaze tracking technology currently exists as part of a rapidly developing technology, and this gaze tracking technology is incorporated herein.

Referring now to the figures, and particularly to FIG. 1, there is shown a simplified diagram of a data processing system 10 in which the present invention may be advantageously employed. Data processing system 1 of FIG.
0 is a configuration that includes all functional components of the computer and its associated hardware. Data processing system 10 includes a central processing unit ("CPU") 24, such as a conventional microprocessor, and a number of other units interconnected via a system bus 26.
CPU 24 is a data processing system 10 that controls the operation of all computer systems, including the execution of arithmetic and logic functions contained in a particular computer program.
Contains a part of. Although not shown in FIG.
CPUs such as U24 typically include a control unit that organizes data and program storage in computer memory and transfers data and other information between many parts of the computer system. Such CPUs also generally include arithmetic units that perform arithmetic and logical operations such as addition, comparison, multiplication, and the like.

The data processing system 10 includes a RAM 28
, A ROM 30, a display adapter 32 connecting the system bus 26 to the display device 14, and peripheral devices (for example, the disk and tape drive 3).
I / O adapter 3 for connecting 3) to system bus 26
4 is further included. RAM 28 is a typical example of a memory configured so that the location of stored data is independent of its contents. Further, any location in RAM 28
It can be accessed directly without having to do anything from scratch. RO
M30 is a typical example of a memory that holds information permanently and whose stored information is not changed by a computer program or normal operation.

The display device 14 is the visible output of the data processing system 10. The display device 14
It may also be a cathode ray tube (CRT) type video display well known in the computer hardware arts. However, for portable or notebook computers, the display device 14 can be replaced by a liquid crystal (LCD) or gas plasma flat panel display. The data processing system 10 includes a keyboard 16, a mouse 20, a speaker 38, a microphone 40, and / or other user interface devices such as a touch screen device (not shown) on the system bus 26. And a user interface adapter 36 for connecting to Using the communication adapter 42,
The data processing system 10 can be connected to an electronic sensor as described in connection with the sensory feedback system of FIG. Although data processing system 10 is shown as including only one CPU and one system bus, the present invention provides a computer system having multiple CPUs and a plurality of CPUs each performing various functions. It will be understood that the present invention can be similarly applied to a computer system having three buses.

Data processing system 10 also includes an interface resident in the machine-readable medium to direct the operation of data processing system 10. Suitable machine-readable media can hold interfaces such as RAM 28, ROM 30, magnetic disks, magnetic tape, optical disks (the last three are provided on disk and tape drives 33). A suitable operating system and associated interface (eg, Microsoft Windows) can direct CPU 24. For example, AIX operating system and AIX
The Windows window system can instruct CPU 24. The AIX operating system is IBM's implementation of the UNIX operating system. Other technologies, such as touch screen technology or voice control, can also be utilized with CPU 24.

Those skilled in the art will appreciate that the hardware depicted in FIG. 1 can be modified for a particular sensory feedback processing application. For example, optical disk media, audio adapters, computer
PAL or EPR well-known in the art such as hardware
Other peripheral devices, such as a chip programming device, such as an OM programming device, can be utilized in addition to or in place of the previously described hardware. Further, the main memory 44 is connected to the system bus 26 and includes a control program 46.
Control program 46 includes instructions that reside in main memory 44 and, when executed by CPU 24, perform the operations described in connection with FIGS. 2 and 3 described in more detail below.

Although the present invention has been described (and will be described) with reference to a full function computer system,
Those skilled in the art will appreciate that the present invention can be distributed as program products in various forms, and that the present invention applies equally, regardless of the particular type of signal-bearing medium used to actually distribute it. Keep in mind that
is important. Examples of signal-bearing media include recordable media, such as floppy disks, hard disk drives, CD-ROMs, and transmission media, such as digital and analog communication links.

Referring now to FIG. 2, a sensory feedback interface system 200 in accordance with the teachings of the present invention.
Is shown in a simplified block diagram. The sensory feedback interface system 200 includes a display device 202 and a sequence controller 21.
6, data processing system 212, including several sensors. In the embodiment of the present invention shown in FIG. 2, sensors are provided to viewer 204 so that each sensor can detect the physiological response of viewer 204 with minimal interference with viewer's cognitive attention. Connected or arranged.

The line-of-sight tracker 214 can measure the eye movement of the viewer 204 in real time.
12 to the viewer 204. In addition to the gaze information, the data processing system 212
Collects other real-time viewer response information from a heart rate detector (eg, an electrocardiograph) 206, a posture sensor 210, and a blood pressure sensor 208. It should be noted that the sensors shown in FIG. 2 make up one possible sensor arrangement and are not a complete list of possible types of sensors available within the spirit and scope of the present invention.

Display device 202 provides a visual representation of the sequential presentation to viewer 204. Those skilled in the art will recognize other sensory output devices, such as audio speakers, alone or on display device 2.
It will be appreciated and understood that the extensions required to implement the method and system of the present invention when used with a visual display device such as 02. In the example shown in FIG. 2, the sequential presentation may be in the form of a movie in which the display device 202 provides a visual output on a projection screen. Display device 202 may be a CRT or other analog display screen device in other embodiments of the present invention.

In a preferred embodiment of the present invention,
The data processing system 212 includes sensors 214, 208,
Includes instructions resident in control program 46 that collects and processes real-time sensory feedback from 210,206. The collection of this information from the sensors 214, 208, 210, 206 is accomplished using an input information adapter, such as the communication adapter 42 of FIG. The processing performed by the data processing system includes interpreting the collected sensory feedback information according to the teachings of the present invention. Since not disturbing the viewer is an important purpose of the sensory feedback interface system 200, interpreting sensory feedback executes a direct command, such as a command executed by a computer processing system in response to direct input from a user. It is not a trivial operation. Therefore, the data processing system 2
The control program resident in the memory 12
Fuzzy logic or deterministic methods suitable for handling implicit or implicit "commands" from 4,208,210,206 may be included. This process may include characterizing the received viewer feedback on how best to proceed with the presentation. In a preferred embodiment of the invention, this characterization may include assigning a relative weight value to each received sensory response.

In the exemplary embodiment of the present invention shown in FIG. 2, the data processing system 212 includes sensors 206, 2
The signals received from 14, 208, 210 are processed so that the sequential presentation can be adapted in real time and optimally to the response of the viewer 204 to the presentation. Data processing system 21
2 from the sequence controller 21 during the sequential presentation.
Sent to 6. Although the sequence controller 216 is shown in FIG. 2 as a separate entity, in other embodiments of the present invention, the sequence controller 216 may be incorporated into the data processing system 212.
The sequence controller 216 has a sequential
It has a script database 218 that contains encoded pictorial information containing the presentation. This encoded presentation information can be immediately accessed according to the processed feedback information received from the data processing system 212 and stored on a magnetic or optical recording medium so as to be sequentially arranged.
In an important aspect of the present invention, the sequential presentation displayed on display device 202 must be changeable in response to real-time feedback. Within the sensory feedback device 200, this change is realized by the combined functionality of the data processing system 212 and the sequence controller 216. For example, sequence controller 216 can include program instructions for predicting a viewer's physiological response to stimuli provided by the presentation. The sequence controller 216
Further, instructions can be included that correlate those predicted physiological responses to the viewer's physiological responses.

Referring now to FIG. 3, a flowchart of a sensory feedback loop 300 is shown in accordance with the teachings of the present invention. The sensory feedback loop 300 begins at step 302, which represents the initialization of a sequential presentation. In the embodiment illustrated in FIG. 3, this sequential presentation is C
A visual presentation displayed on a visual display such as an RT or similar mechanism. The initialization represented in step 302 may involve providing an initial sequence of a series of visual scenes. The display of the current scene is represented in step 304 and is performed by the sequence controller 216 and the script database 218.
And a presentation output device including the display device 202. Next, step 30
As represented by 6, a response of a viewer, such as viewer 204, is detected. This detection is performed by the sensors 214, 208,
This can be done using any one or all of 206, 210 and other devices capable of detecting a physiological response that can be interpreted as implicit feedback from the viewer.

Step 308 represents the processing of the sensory feedback information detected as a physiological response in step 306. The basis for the concept underlying the method and system of the present invention is that the feedback obtained from the viewer is implicit so that interference with the viewer's cognitive attention to the presentation is minimized. . Thus, in the absence of explicit commands to control changes in the sequential presentation, some type of artificial intelligence, such as that used by fuzzy logic programs, may be used in step 308 to interpret the raw sensory data. it can. This interpretation involves characterizing certain reactions in terms of how the presentation should be continued. Also, if multiple reactions are detected simultaneously, the interpretation at step 308 may include assigning a relative weight value to each reaction. Finally, step 310 represents the selection of a scene that follows the current scene represented in step 304. In a preferred embodiment of the present invention, the selection at step 310 includes a change from the current scene according to the interpretation represented at step 308.

Preferred embodiments of the present invention include implementation as a computer system programmed to perform one or more of the methods described herein, and as a program product. According to an implementation of a computer system, the set of instructions for performing the methods and systems of the present invention may be implemented in one or more computer systems.
Resident in a storage device such as M. Until requested by the computer system, the instruction set may be in other computer memory, for example, a disk drive (which may include removable memory such as an optical disk or a floppy disk utilized in the disk drive). Within, can be stored as a computer program product.

The computer program product also comprises:
It can be stored on another computer and transmitted when required by the user's workstation over a network or external communication network. One skilled in the art can appreciate that the physical storage of the instruction set physically changes the medium stored to carry computer readable information. The change is an electrical, magnetic, chemical, or other physical change. While it is convenient to describe the invention in terms of instructions, symbols, characters, and the like, the reader should remember that all of these and similar terms relate to the appropriate physical components. Thus, the method for performing the steps described in connection with FIG. 3 can be implemented on a computer-aided device. In such a method, data stored in a memory unit of the data processing system may represent a step in a method for implementing a preferred embodiment of the present invention.

The embodiments and examples described herein are set forth to best explain the invention and its practical applications, and thereby to enable any person skilled in the art to make and use the invention. However, those skilled in the art will understand that the foregoing description and examples have been presented for the purpose of illustration and example only.
The above description is not intended to be complete and is not intended to limit the invention as disclosed. Many modifications and variations are possible in light of the above teaching without departing from the spirit and scope of the appended claims.

In summary, the following is disclosed regarding the configuration of the present invention.

(1) A method for controlling a sequential presentation by modifying the sequential presentation according to an implicit feedback from a viewer, wherein the sequential presentation is controlled by at least one viewer during a sequential presentation. Detecting a sensory response including viewer feedback, sending the implicit viewer feedback to a presentation sequence controller, and providing the presentation sequence controller with a view to how the presentation sequence can proceed. Characterizing the implicit viewer feedback; and, in response to the characterizing step, the viewer being in real-time and
Modifying the sequential presentation so that the sequential presentation can be adapted as best suited to how to understand and react to the presentation.

(2) The step of detecting a sensory response from at least one viewer further comprises the step of detecting a physiological response indicative of a physiological response of the at least one viewer to the sequential presentation. (1) The method according to the above.

(3) The step of detecting a physiological response from the at least one viewer further includes detecting spontaneous and involuntary physiological responses of the at least one viewer. 2) The method as described.

(4) The step of detecting a sensory response from the at least one viewer is such that the sensory response is transparent such that the at least one viewer can maintain maximum cognitive attention to the sequential presentation. The method according to the above (1), further comprising the step of:

(5) The method of (1) above, wherein characterizing the implicit viewer feedback further comprises interpreting the implicit viewer feedback utilizing fuzzy logic.

(6) The method of (1) above, further comprising detecting a plurality of sensory responses from the at least one viewer, wherein the plurality of sensory responses include the viewer feedback.

(7) The method of (6), wherein the step of characterizing the implicit viewer feedback further comprises assigning a relative weight value to each of the plurality of sensory responses.

(8) A system for modifying a sequential presentation according to implicit sensory feedback from a viewer, the sequential presentation being provided to at least one viewer using a display device, and A sensor for detecting a physiological response of a viewer including viewer feedback, and interpreting the physiological response from the viewpoint of how the sequential presentation can proceed according to the physiological response optimally. A system comprising: means; and means for modifying the sequential presentation according to a result from the interpreting means.

(9) The system according to the above (8), wherein the physiological response includes an implicit command from the viewer.

(10) The system of (8) above, further comprising a transparent sensory interface for sending said physiological response of said at least one viewer to said sensor.

(11) The system according to (10) above, wherein said transparent sensory interface is characterized by a group consisting of a gaze tracker, a blood pressure detector, a pulse rate detector, and a posture detector.

(12) The system according to (8), further comprising means for detecting a plurality of physiological reactions.

(13) The system according to the above (12), further comprising means for assigning a relative weight value to each of the plurality of physiological responses.

(14) The system according to (8), wherein said display device is characterized by a group consisting of an audio output device and a video output device.

(15) The system according to (8), wherein said presentation sequence is stored in a digital storage medium.

(16) The system according to (15), wherein said storage medium is characterized in a group consisting of a magnetic storage device or an optical storage device.

(17) The system according to (8), wherein said display device is a computer screen, and said at least one viewer is one computer user.

(18) A program product for controlling the sequential presentation by changing the sequential presentation according to the implicit feedback from the viewer, wherein the program product is controlled by at least one viewer during the sequential presentation. Means for detecting a sensory response including explicit viewer feedback; means for sending the implicit viewer feedback to a presentation sequence controller; and means for storing the presentation stored on a signal-bearing medium in the presentation sequence controller. Command means for characterizing the implicit viewer feedback in terms of being able to proceed; stored in a signal-bearing medium, in real-time and How if perfect or react understand to presentation, so that it can be adapted to sequentially program product comprising an instruction means for changing the sequential presentation according to the characterizing step.

(19) Instruction means stored on a signal-bearing medium in the presentation sequence controller and characterizing the implicit viewer feedback:
The program product of claim 18, further comprising instruction means stored on a signal-bearing medium for interpreting the implicit viewer feedback utilizing fuzzy logic.

(20) Instruction means stored on a signal-bearing medium for modifying the sequential presentation further comprises instruction means for predicting a sensory response of the at least one viewer to a stimulus provided by the presentation. The program product according to the above (18).

[Brief description of the drawings]

FIG. 1 shows a simplified representation of a data processing system utilized in implementing the methods and systems of the present invention.

FIG. 2 shows a simplified flow diagram of a sensory feedback loop that can be used in a preferred embodiment of the present invention.

FIG. 3 shows a simplified block diagram of the functional elements and devices that make up a changeable presentation sequence control system according to a preferred embodiment of the present invention.

[Explanation of symbols]

10 Data Processing System 14 Display Device 16 Keyboard 20 Mouse 24 CPU 26 System Bus 28 RAM 30 ROM 32 Display Adapter 33 Disk and Tape Drive 34 I / O Adapter 36 User Interface Adapter 38 Speaker 40 Microphone 42 Communication Adapter 44 Main Memory 46 control program 200 sensory feedback interface system 202 display device 204 viewer 206 heart rate detector (electrocardiograph) 208 blood pressure sensor 210 attitude sensor 212 data processing system 214 gaze tracker 216 sequence controller 218 script database

 ────────────────────────────────────────────────── ─── Continued on the front page (72) Inventor Winslow Scott Burson United States 94306 Palo Alto Georgia Avenue 4173 California 4172 (72) Inventor John Martin Malary United States 78731 Austin, Texas Sidehill Pass 3903

Claims (20)

[Claims] 1. A method for controlling a sequential presentation by modifying the sequential presentation according to implicit feedback from a viewer, the method comprising: controlling a sequential presentation from at least one viewer during the sequential presentation. Detecting a sensory response including viewer feedback, sending the implicit viewer feedback to a presentation sequence controller, and providing the implicit viewer feedback in terms of how the presentation sequence controller can proceed with the presentation. Characterizing dynamic audience feedback; and in response to the characterizing step, the viewer Modifying the sequential presentation so that the sequential presentation can be adapted to best understand and react to it. 2. The method of claim 1, wherein detecting a sensory response from at least one viewer further comprises detecting a physiological response indicative of a physiological response of the at least one viewer to the sequential presentation. The method of claim 1. 3. The method of claim 2, wherein detecting a physiological response from the at least one viewer further comprises detecting spontaneous and involuntary physiological responses of the at least one viewer. The described method. 4. The method of claim 1, wherein the step of detecting a sensory response from the at least one viewer comprises: translating the sensory response transparently so that the at least one viewer can maintain maximum cognitive attention to the sequential presentation. The method of claim 1, further comprising the step of detecting. 5. The step of characterizing the implicit viewer feedback further comprises interpreting the implicit viewer feedback utilizing fuzzy logic.
The method of claim 1. 6. The method of claim 1, further comprising the step of detecting a plurality of sensory responses from the at least one viewer, wherein the plurality of sensory responses include the viewer feedback.
The described method. 7. The method of claim 6, wherein the step of characterizing the implicit viewer feedback further comprises assigning a relative weight value to each of the plurality of sensory responses. 8. A system for modifying a sequential presentation according to implicit sensory feedback from a viewer, the system comprising: a sequential presentation provided to at least one viewer using a display device; A sensor for detecting a physiological response of a viewer including viewer feedback; and a means for interpreting the physiological response in terms of how the sequential presentation can proceed in accordance with the physiological response optimally. And a means for modifying the sequential presentation according to a result from the interpreting means. 9. The system of claim 8, wherein said physiological response includes an implicit command from said viewer. 10. The system of claim 8, further comprising a transparent sensory interface for sending said physiological response of said at least one viewer to said sensor. 11. The transparent sensory interface includes a gaze tracker, a blood pressure detector, a pulse rate detector,
The system of claim 10, wherein the system is characterized in a group consisting of attitude detectors. 12. The system of claim 8, further comprising means for detecting a plurality of physiological responses. 13. The system of claim 12, further comprising: means for assigning a relative weight value to each of said plurality of physiological responses. 14. The system of claim 8, wherein said display device is characterized in a group consisting of an audio output device and a video output device. 15. The system of claim 8, wherein said presentation sequence is stored on a digital storage medium. 16. The system of claim 15, wherein said storage medium is characterized in a group consisting of magnetic storage or optical storage. 17. The system of claim 8, wherein said display device is a computer screen and said at least one viewer is a computer user. 18. A program product for controlling a sequential presentation by altering the sequential presentation according to implicit feedback from a viewer, the program product controlling the sequential presentation from at least one viewer during the sequential presentation. Means for detecting sensory responses including viewer feedback; means for sending the implicit viewer feedback to a presentation sequence controller; and how the presentation is stored on a signal-bearing medium in the presentation sequence controller. Command means for characterizing the implicit viewer feedback in terms of what is possible; stored on a signal-bearing medium, in real-time and allowing the viewer to Command means for modifying the sequential presentation in response to the characterizing step so that the adaptation can be sequentially adapted in an optimal manner in how to understand and respond to the presentation. 19. The presentation sequence system
Instruction means stored on the signal-bearing medium in the controller and characterizing the implicit viewer feedback further comprise instruction means stored on the signal-bearing medium for interpreting the implicit viewer feedback utilizing fuzzy logic. The program product according to claim 18. 20. The instruction means stored on a signal-bearing medium for modifying the sequential presentation further comprises instruction means for predicting a sensory response of the at least one viewer to a stimulus provided by the presentation. Item 19. The program product according to Item 18.