KR20000062474A - Sensory feedback for sequential presentation adaptation - Google Patents

Abstract

The present invention relates to a method and system for controlling a sequential presentation by adaptively changing the sequential presentation according to the implicit feedback of the audience members. First, a sensory response including implicit audience feedback of audience members is detected during the presentation. Next, the implicit audience feedback is passed to the presentation sequence controller. The implicit audience feedback is then characterized according to how the presentation proceeds within the presentation sequence controller. Finally, in response to the characterization of this implicit feedback, the presentation is altered to best suit the way in which the audience member perceives and responds to the presentation and to adapt the presentation sequentially in real time.

Description

Sensory feedback for sequential presentation adaptation {SENSORY FEEDBACK FOR SEQUENTIAL PRESENTATION ADAPTATION}

The present invention relates to a method of audience participation. In particular, the present invention relates to a method and system for using sensory feedback to control a presentation sequence. More specifically, the present invention relates to a method and system for providing implicit audience feedback for adaptively controlling a sequential presentation.

Interactive / adaptive movies typically require input from the viewer that is obtained by having the viewer answer questions about the next scene or select a command for the next scene. This viewer's input is cumbersome as it requires a conscious selection of a particular view by the viewer (eg, "Choose A or B") and a deliberate physical action of the viewer (eg, an act of selection). Thus, this interactive way of interacting with the audience greatly hinders viewers from viewing the presentation. This problem is particularly acute in interactive presentations intended to entertain the audience. In order to get the maximum entertainment effect from a recreational presentation such as a movie, the viewer must be deeply immersed in the presentation. However, this is very difficult with current interactive movie technology for the reasons mentioned above.

Thus, it will be appreciated that a method and system for using sensory feedback of audience members is needed to control sequential presentation. If such a method and system could be implemented, it would be very useful to adapt the presentation sequence to best suit the audience member's implicit response to the presentation.

Accordingly, it is a first object of the present invention to provide an improved audience participation method.

It is a second object of the present invention to provide a method and system for using sensory feedback to control a presentation sequence.

It is a third object of the present invention to provide a method and system for providing implicit audience feedback for adaptively controlling a sequential presentation.

The above and other objects are achieved by the following description. Hereinafter, a method and system for controlling the sequential presentation by adaptively changing the sequential presentation according to the implicit feedback of the audience members will be described. First, a sensory response including implicit audience feedback of audience members is detected during the presentation. Next, implicit audience feedback is passed to the presentation sequence controller. The implicit audience feedback is then characterized according to how the presentation proceeds within the presentation sequence controller. Finally, in response to the characterization of this implicit feedback, the presentation is altered to best suit the way in which the audience member perceives and responds to the presentation and to adapt the presentation sequentially in real time.

1 is a schematic diagram of a data processing system used to implement the method and system of the present invention.

2 is a schematic block diagram of a functional entity and apparatus for forming an adaptive presentation sequence control system according to a preferred embodiment of the present invention.

3 is a schematic flowchart of a sensory feedback loop used in a preferred embodiment of the present invention.

[Description of Symbols for Main Parts of Drawing]

10: data processing system

14: display device

200: sensory feedback interface system

202: display device

204: audience members

212 data processing system

216 sequence controller

The novel features of the invention are set forth in the appended claims. However, not only the preferred embodiment of the present invention, but also the invention itself, other objects and advantages thereof will be best understood from the detailed description of the following embodiments with reference to the accompanying drawings.

The method and system of the present invention use implicit audience feedback in the form of both spontaneous and involuntary responses to the presentation sequence. Spontaneous reactions that can be used in embodiments of the present invention include, but are not limited to, eye movements, hand gestures, respiration rates, and the like. Involuntary reactions used in the present invention may include changes in blood pressure and heart rate, sweating, and the like. Some measurable physiological responses are obtained by combining spontaneous and involuntary responses to the presentation. An example of such a parameter is eye movement. In particular, eye-gaze tracking will be very useful for obtaining comprehensive information about both the voluntary (conscious) and involuntary (unconscious) responses of the audience members to the presentation. Eye tracking technology is a rapidly developing technology field and forms part of the present specification.

1, there is shown a schematic form of a data processing system 10 in which the present invention may be preferably used. The data processing system 10 of FIG. 1 is a configuration that includes all the functional components of a computer and associated hardware. The data processing system 10 includes a central processing unit (CPU) 24 such as a conventional microprocessor and a number of other units interconnected via a system bus 26. The CPU 24 includes the execution of the arithmetic and logic functions contained in a particular computer program and forms part of the data processing system 10 that controls the operation of the entire computer system. Although not shown in FIG. 1, CPUs, such as the CPU 24, typically comprise a control unit that organizes data and program storage locations within computer memory and transfers the data and other information between various parts of the computer system. . In addition, such CPUs typically include arithmetic units that perform arithmetic and logical operations, such as additions, comparison operations, multiplications, and the like.

The data processing system 10 may include a random access memory (RAM) 28, a read only memory (ROM) 30, and a display device 14 on a system bus 26. An additional I / O adapter 34 is provided to connect the display adapter 32 and peripheral devices (e.g., disk and tape drive 33) to the system bus 26 for connection. Include. RAM 28 is a form of memory designed such that the location of data stored therein is independent of the content of the data. In addition, any position of RAM 28 can be directly accessed without accessing all positions sequentially from the start position. The ROM 30 permanently stores information, and is a memory form in which the stored information cannot be changed by a normal operation of a computer or a program.

The display device 14 is an image output device of the data processing system 10. Display device 14 may be a cathode ray tube (CRT) based video display known in the computer hardware art. However, in a portable or notebook-based computer, the display device 14 may be replaced with a liquid crystal display (LCD) based or gas plasma-based flat panel display. The data processing system 10 may include a keyboard 16, a mouse 20, a speaker 38, a microphone 40 and / or a touch-screen device (not shown). It further includes a user interface adapter 36 that connects other user interface devices, such as < RTI ID = 0.0 > The communication adapter 42 may be used to connect the data processing system 10 to an electronic sensor as described in connection with the sensory feedback system of FIG. 2. Although the data processing system 10 of FIG. 1 is shown as including only a single CPU and a single system bus, the present invention is equally applicable to a multi-CPU computer system and a multi-bus computer system that each performs different functions in different ways. .

The data processing system 10 may also include an interface stored in a machine readable medium to direct the operation of the data processing system 10. Suitable machine-readable media include, but are not limited to, RAM 28, ROM 30, magnetic disks, magnetic tapes or optical disks, where magnetic disks, magnetic tapes and optical disks are located within disks and tape drives 33. It may include an interface. The CPU 24 may be directed to any suitable operating system and associated interface (eg, Microsoft Windows). For example, the CPU 24 may be directed to the AIX operating system and the AIX windowing system. AIX operating system is UNIX The operating system is implemented by IBM. In addition, other technologies such as touch screen technology or voice control technology may be used in combination with the CPU 24.

Those skilled in the art will appreciate that the hardware shown in FIG. 1 can be modified depending on the particular sensory feedback processing application. For example, a peripheral device such as an optical disk medium, an audio adapter, or a chip programming device such as a PAL or EPROM programming device known in the computer hardware industry may be added to or substituted for the hardware shown in FIG. . In addition, a main memory 44 including a control program 46 is connected to the system bus 26. The control program 46 resides in the main memory 44 and includes instructions which, when executed in the CPU 24, perform an operation described in greater detail below with respect to FIGS. 2 and 3.

Although the present invention has been described with reference to fully functional computer systems throughout this specification, those skilled in the art will appreciate that the present invention may be distributed in various forms of program products, and in fact the signal transmission used for that distribution. It should be noted that this applies equally regardless of the particular form of signal-bearing medium. Examples of signal carrier media include recordable media such as floppy disks, hard disk drives and CD-ROMs, and transmission media such as digital and analog communication links.

2 shows a schematic block diagram of a sensory feedback interface system 200 in accordance with the present invention. The sensory feedback interface system 200 includes a display device 202, a sequence controller 216, a data processing system 212 and some sensors. In the embodiment of the present invention shown in FIG. 2, the sensor is configured to allow the sensor to detect the physiological response of the audience member 204 while minimizing interference with the cognitive attention of the audience member 204. Or connect to

The gaze tracker 214 is positioned with respect to the audience member 210 to determine in real time the gaze movement of the audience member 210 and transmit it to the data processing system 212. In addition to the gaze information, the data processing system 212 collects other real-time audience response information from the heart rate detector 206, the position sensor 210, and the blood pressure sensor 208. It should be noted that the sensor shown in FIG. 2 is one of the possible sensor configurations and is not an exhaustive list of sensor types available within the scope of the present invention.

The display device 202 provides the audience member 204 with a sequential presentation of the video display. Those skilled in the art will appreciate that other sensory output devices, such as audio speakers, may be applied alone or in combination with video display devices, such as display device 202, to the methods and systems of the present invention. In the embodiment shown in FIG. 2, the sequential presentation may be in the form of a movie in which the display device 202 provides an image output on a projection screen. In alternative embodiments of the invention, the display device 202 may be a cathode ray tube or other similar display screen device.

In a preferred embodiment of the present invention, data processing system 212 belongs to control program 46 and includes instructions for collecting and processing real-time sensory feedback from sensors 214, 208, 210, and 206. Information from these sensors 214, 208, 210, 206 can be collected using an input information adapter such as the communication adapter 42 of FIG. 1. The processing in the data processing system according to the present invention involves the interpretation of the collected sensory feedback information. Since non-intrusiveness is an important purpose of the sensory feedback interface system 200, the interpretation of sensory feedback is as much as the execution of direct commands, such as those executed by a computer operating system in response to a user's direct input. It's not a trivial move. Thus, a control program residing within data processing system 212 may be deterministic or fuzzy logic suitable for processing unspecified or implicit "commands" from sensors 214, 208, 210, and 206. It may include. This process may include the characterization of received audience feedback in accordance with how to optimally process the presentation. In a preferred embodiment of the invention, this characterization comprises assigning a relative importance to each sensory response received.

In the exemplary embodiment of the present invention shown in FIG. 2, the data processing system 212 is best suited for the response of the audience member 204 to the presentation and the sensors 206, 214 so that the sequential presentation can be adapted in real time. Process the signals received from 208,210. This processed signal from data processing system 212 is delivered to sequence controller 216 during the sequential presentation. Although the sequence controller 216 is shown as a separate object in FIG. 2, in alternative embodiments of the invention, the sequence controller 216 may be integrated into the data processing system 212. Sequence controller 216 includes a script database 218 that contains encoded picture information organized in sequential presentations. This encoded presentation information may be stored in a magnetic storage medium or an optical storage medium to be quickly accessed and sequentially aligned in accordance with the processed feedback information received from the data processing system 212. In an important aspect of the invention, the sequential presentation displayed on the display device 202 should be changeable in response to runtime feedback. Within the sensory feedback device 200, this change is achieved through the combined functionality of the data processing system 212 and the sequence controller 216. For example, sequence controller 216 may include program instructions for predicting the physiological response of the audience member to the stimulus provided by the presentation. Sequence controller 216 may further include instructions for associating this predicted physiological response with the physiological response of the audience member.

Referring now to FIG. 3, shown is a flow diagram of a sensory feedback loop 300 in accordance with the present invention. This sensory feedback loop 300 begins at step 302, which indicates the initialization of a sequential presentation. In the embodiment illustrated in FIG. 3, this sequential presentation is an image presentation displayed on an image display device such as a cathode ray tube (CRT) or equivalent device thereof. The initialization step indicated at 302 will involve presenting a first sequence of a series of video scenes. Any current scene is displayed using a presentation output device, which may consist of a display device 202 coupled with a sequence controller 216 and a script database 218, at a step indicated with 304. FIG. Then, as indicated at step 306, the response of the audience member, such as the audience member 204, is detected. This audience member's response can be detected using one or both of the sensors 214, 208, 206, 210 and any other device capable of detecting a physiological response that can be interpreted as implicit feedback from the audience member. have.

Step 308 denotes the processing of sensory feedback information detected as a physiological response in step 306. The fundamental underlying concept of the method and system of the present invention is that the feedback obtained from the audience member must be implicit in order to minimize the interference of the audience member's perceptual attention to the presentation. Thus, in the absence of explicit instructions to control the adaptation of the sequential presentation, some form of artificial intelligence, such as used by the fuzzy logic program to interpret the original sensory data, may be used in step 308. This interpretation involves characterizing specific responses according to how the presentation proceeds. Also, if multiple responses are detected at the same time, the interpretation at step 308 may include assigning a relative importance for each response. Finally, step 310 denotes the selection of a scene following the current scene indicated in step 304. In a preferred embodiment of the present invention, the selection at step 310 includes a change from the current scene according to the interpretation at step 308.

Preferred embodiments of the invention include those implemented with a computer system and a program product programmed to perform the method or methods described herein. In accordance with a computer system implementation, a set of instructions that perform the methods and systems of the present invention reside in a storage device, such as RAM, of one or more computer systems. The instruction set may be stored as a computer program product in another computer memory, such as a disk drive (including removable memory such as an optical or floppy disk that is ultimately used in a disk drive) until required by the computer system. .

The computer program product may also be stored on another computer and, if desired, transmitted to a user's workstation by a network or an external communications network. Those skilled in the art will appreciate that the physical storage location of an instruction set physically alters the medium on which the instruction set is stored such that the medium carries computer readable information. This change may be an electrical, magnetic, chemical or some other physical change. While it is easy to describe the invention in instructions, symbols, letters, and the like, it should be remembered that all such and similar terms should be associated with appropriate physical elements. Thus, the method of implementing the steps described in connection with FIG. 3 may be accomplished with a computer-aided device. In such a method, data stored in a memory unit of a data processing system, such as data processing system 212, may represent steps in a method of implementing a preferred embodiment of the present invention.

The embodiments and examples disclosed herein best illustrate the invention and its practical application, thereby enabling those skilled in the art to use the invention. However, one of ordinary skill in the art will recognize that the foregoing description and examples are merely illustrative. The description set forth herein is not intended to be exhaustive or to limit the invention to the precise form disclosed. Various modifications and variations are possible in light of the above description without departing from the spirit and scope of the appended claims.

According to the present invention, by adaptively controlling the sequence of the interactive presentation using sensory feedback of the audience receiving the presentation, the audience viewing the presentation can focus on the presentation without interference, such as input selection of the next scene.

Claims (20)

A method of controlling a sequential presentation by adaptively changing the sequential presentation according to an implicit feedback of an audience member, the method comprising: a) during the sequential presentation, detecting a sensory reaction comprising implicit audience feedback from at least one audience member; b) conveying the implicit audience feedback to a presentation sequence controller; c) in the presentation sequence controller, characterizing the implicit audience feedback in accordance with a presentation progress method; d) in response to the characterization step, modifying the sequential presentation so that an audience member is best suited to the method of sensing and responding to the sequential presentation and so that the sequential presentation can be sequentially adapted in real time. Control method of the sequential presentation comprising a. The method of claim 1, wherein detecting the sensory response from the at least one audience member further comprises detecting a physiological response indicative of the physiological response of the at least one audience member to the sequential presentation from the at least one audience member. How to control sequential presentations. 3. The method of claim 2, wherein detecting the physiological response from the at least one audience member further comprises detecting spontaneous and involuntary physiological responses of the at least one audience member. 2. The method of claim 1, wherein detecting the sensory response from the at least one audience member comprises applying the sensory response in a transparent manner such that the at least one audience member maintains the maximum cognitive attention to the sequential presentation. And detecting the sequential presentation. 2. The method of claim 1, wherein characterizing the implicit audience feedback further comprises interpreting the implicit audience feedback using fuzzy logic. The method of claim 1, further comprising detecting a plurality of sensory responses comprising audience feedback from the at least one audience member. 7. The method of claim 6, wherein characterizing the implicit audience feedback further comprises assigning a relative importance to each of the plurality of sensory responses. In a system that adaptively changes a sequential presentation in response to an audience's suggestive sensory feedback, a) a sequential presentation provided to at least one audience member by a display device, b) a sensor for detecting a physiological response comprising audience feedback of said at least one audience member, c) means for interpreting the physiological response in terms of how the sequential presentation can proceed most appropriately for the physiological response; d) means for changing the sequential presentation according to the results of the interpreting means System for adaptively changing the sequential presentation according to the audience's implicit sensory feedback. The system of claim 8, wherein the physiological response comprises an implicit command from an audience member. The system of claim 8, further comprising a transparent sensory interface that communicates the physiological response of the at least one audience member to the sensor. The system of claim 10, wherein the transparent sensory interface is selected from the group consisting of eye trackers, blood pressure detectors, heart rate detectors, and position detectors. The system of claim 8, further comprising means for detecting a plurality of physiological responses. 13. The system of claim 12, further comprising means for assigning relative importance to each of the plurality of physiological responses. The system of claim 8, wherein the display device is selected from the group consisting of an audio output device and a video output device. The system of claim 8, wherein the presentation sequence is stored on a digital storage medium. 16. The system of claim 15, wherein the storage medium is selected from the group consisting of magnetic storage devices or optical storage devices. The system of claim 8, wherein the display device is a computer screen and the at least one audience member is a single computer user. In a program product that controls the sequential presentation by adaptively changing the sequential presentation according to the implicit feedback of the audience members, a) means for detecting sensory responses including implicit audience feedback from at least one audience member during the sequential presentation, b) means for conveying the implicit audience feedback to a presentation sequence controller; c) instruction means stored on a signal carrier medium in the presentation sequence controller, the instruction means for characterizing the implicit audience feedback in terms of how the presentation proceeds; d) instruction means stored in a signal carrier medium for modifying the sequential presentation so that it is best suited to the way in which the audience member detects and responds to the sequential presentation in response to the characterization step and the presentation can be adapted sequentially in real time. Program product for controlling the sequential presentation by adaptively changing the sequential presentation in accordance with the implicit feedback of the audience members, including. 19. The program of claim 18, wherein the instruction means stored on the signal carrier medium in the presentation sequence controller to characterize implicit audience feedback further comprises instruction means stored on the signal carrier medium to interpret the implicit audience feedback with fuzzy logic. product. 19. The program product of claim 18, wherein the instruction means stored on the signal carrier medium to modify the sequential presentation further comprises instruction means for predicting the physiological response of at least one audience member to the stimulus provided by the sequential presentation.