WO2017077364A1

WO2017077364A1 - Information device with simultaneous collection of feedback, method of presentation of information

Info

Publication number: WO2017077364A1
Application number: PCT/IB2015/058535
Authority: WO
Inventors: Jakub BERČÍK; Ján BAJÚS; Ľudmila NAGYOVÁ; Elena HORSKÁ
Original assignee: Slovenská Poľnohospodárska Univerzita V Nitre; Centrum Vedecko-Technických Informácií Sr
Priority date: 2015-11-03
Filing date: 2015-11-04
Publication date: 2017-05-11
Also published as: SK289006B6; CZ32181U1; EP3371763A1; SK500742015A3

Abstract

At least one camera (5, 6) captures the person before the display unit (1 ), a reaction of the person is analyzed from the collected image and the settings of the presentation of the information are altered according to the reaction. The information device has a display unit (1 ) for display of the visually presented information, at least one speaker (2) and a computer (3) with a memory for the presented information. The device includes at least two cameras (5, 6). The stable camera (6) is connected to the module (7) for the recognition of the position of the person and for the determination of the position of the face; the output is connected to the control of the direction of the positionable camera (5). The output from the positionable camera (5) is connected to the computer (3) in which there is a facial expressions recognition unit (8). On the basis of the image data, the bodily position of the limbs and/or the facial expression and/or the change in the heartbeat and/or the sex and/or the approximate age of the person before the display unit (1 ), is determined, and the mode and/or content of the presentation is chosen on the basis of these data.

Description

Information device with simultaneous collection of feedback, method of presentation of information

Field of technology The invention concerns the information device which provides the information, activates the senses and produces a stimuli for the person in the vicinity of the device, whereby the device also records and eventually assesses the feedback of the person receiving the information, stimuli and perceptions, and it can also produce or affect the environment in the desired way. The received feedback can not only be recorded for further statistical processing, but it can also be used for immediate change in the mode of presentation of the information, including the change in some features of the environment.

Prior state of the art In many economic fields as well as in many social situations it is necessary to receive feedback from the persons receiving the information, perceptions and stimuli. Questionnaires are usually used for this purpose, but such method is protracted and marred by many systemic errors. In fact, questionnaires provide subjective view of the conscious projection with regard to the given question, but this usually does not correspond to the actual unconscious reaction of that person. The modern substitute for questionnaires are various electronic devices where the person chooses her answer by, for example, touching the given icon. Such devices, such as the one according to FR 2998077 A1 , provide only very limited data concerning the reaction without proper synchronization with the presented information. The gathered results are evaluated only afterwards, and only then the presentation of the information can be adjusted for other persons.

Publication AU 2014101196 A4 discloses a solution of gathering of the feedback by means of web-oriented application in the mobile phone, which, however, requires high level of cooperation from the person. According to US 201 1218930 A1 the feedback is gathered by single-purpose electronic devices which are firstly distributed to the customers, which is - however - complicated and it narrows the amount of evaluated persons.

What is needed is a device which will present the various desired information and which will also gather the immediate reaction, feedback from the participating person, and which will eventually also adapt the mode of presentation according to the particular situation and reaction of the participating person.

Subject matter of the invention

The abovementioned deficiencies are significantly remedied by the information device with simultaneous collection of feedback - where the device includes a display unit for displaying the visually presented information, at least one speaker and a computer with the memory where the presented information are stored, whereby the display unit is connected to this computer - according to this invention, which essence lies in the fact that the device includes two cameras, one camera is positionable and other is stable, whereby both cameras are oriented to the space before the display unit. The stable camera is connected with the module for recognition of the position of the person and for determination of the position of the face; the output of this module is connected with the control of the direction of the positionable camera. The output of the positionable camera is connected to the computer in which there is a unit for recognition of the face expressions.

In this basic arrangement the visual and sonic information are presented to the person or persons in the vicinity of the device. If the person is in the angle of view of the stable camera the module recognizes her position and evaluates the actual position of the face of the person. The positionable camera focuses on this position and takes the face of the person into the view. Image data are gathered from this view in order to assess the facial expressions; the results are temporarily synchronized with the presented information. Known algorithms can be used for the evaluation of the expressions and micro-expressions of the face. The positionable camera can alter the focal length, which is changed according the recognized size of the face in the view of the stable camera. This size depends mainly on the actual distance of the person from the displaying unit.

The presented information can be of any nature and can have any content; usually they will be video clips with sound. They can be of an educational, presentational or propagation nature or they can serve some other purpose.

The module for recognition of the position of the person, which is connected with the stable camera, can be adjusted for the evaluation of the gestures of the person and the movement of her limbs. The person standing before the displaying unit reacts to the received information by altering the position of her body, too. The change of the position of the body also hints at the reaction of the person to the received information.

It is preferable if both cameras are located on the front side of the device where the displaying unit is, whereby they are located above the display unit. The device can have outer casing in form of an independent panel designed for independent placing on the floor or hanging on the wall.

From the point of view of easy construction of the device it will be preferable if the commercially available device used in gaming industry - capable of preliminary production of the vector model of the figure - is used as the stable camera with the module for recognition of the position of the person. The body of the person with the respective position of the limbs, eventually also with the position of the hands and fingers, is recognized and described in the three-dimensional system. Some devices are able to analyze the person's pulse, sex, age and basic micro-emotions. The data from the stable camera are used not only for the direction of the movable camera, but they are also sent to the computer as the data for the evaluation of the feedback.

Pursuant to the feedback the content or form of the presented information - such as the loudness, brightness, zoom and so on - can be adjusted. With the help of the neuroscience we can assess the optically and remotely recorded functions of the human body in order to construct the attitude towards the presented theme, for example the particular good or service. The investigation of the emotions of the persons who are being presented with the information is all the more important. According to the neuroscientific the decision-making of the persons is not very rational and it is based on the unconscious processes in the mind. The gathered data are interpreted and put into relation with the judgment, emotion and decisionmaking of the person. In the preferable arrangement the aromatizer releasing the odors in the vicinity of the device can be part of the device. Multiple aromatizers can be used, or multiple dispensers of various fragrances. It is preferable if the sensor of the presence of the person in the vicinity of the device is part of the device. The sensor is connected to the computer. The start of the presentation on the display unit, or the operation of the device as a whole, can be controlled by this sensor of the presence. It can be a motion sensor detecting the entrance of the person to the vicinity of the device; it can also be a sensor operating on the different basis of the detection, for example noise sensor, thermal imaging sensor, or pressure sensor in the pad, for example under the carpet before the device. In order to determine the presence of the person in the vicinity of the device an analysis of the image data from the camera can be used; these data can identify the image of the figure of a man and his movement in the field of view.

Apart from the display unit for the display of the multimedia content, an alphanumeric displaying element can be part of the device; this element's purpose will be the display of the time, date and the temperature of the environment.

The chromaticity of the environment affects the human perception; the device can have RGB LED lighting with the adjustable chromaticity, which is controlled according to the content of the multimedia presentation or on the basis of the gathered feedback. The lighting can be set according to the surrounding climate - or micro-climate, respectively - whereby for example the air pressure in the immediate vicinity or the weather in the locality of the device is taken into account. The data concerning the weather can be gathered from the own sensors or from the foreign source, for example from the data available on the internet. RGB LED lighting will be usually placed on the sides of the device.

Excitation indicator can be also part of the device. The excitation indicator can graphically accompany the sounding of the device; for example, it can be rhythmically connected with the musical background of the presentation in such a way that the luminous points of the excitation indicator will shine in the rhythm of the accompanying music. The excitation indicator can indicate the degree of the interaction of the person with the device pursuant to the evaluation of the feedback. In such case the excitation indicator will signal - for example - the degree of the ability of the device to determine the facial expression.

In order to be able to take into account the effects of the weather during the setting of the operation of the individual elements and during the evaluation of the feedback of the person, it is preferable if the device has the respective sensor or multiple sensors. This can concern the sensor of the barometric pressure, the external temperature sensor, the external humidity sensor, and the outer lighting sensor. Based on the data they provide the information concerning the actual weather is gathered. This information can be got from the internet for the particular localization of the device; this requires the respective internet connection of the device.

The aromatizer, the sensor of the presence, RGB LED lighting with the adjustable chromaticity, the excitation indicator, the sensors of the environment in the vicinity of the device, or the exterior sensors can be connected to and controlled from the computer either directly or through one or multiple microcontrollers. It will be preferably if one or two micro-controllers controlling the group of the related peripheral elements are used. The deficiencies in the prior state of the art are also significantly remedied by the method of the presentation of the information by means of a display unit and a speaker according to this invention, which essence lies in the fact that at least one camera captures the person before the display unit, on the basis of the retrieved image a reaction of the person is analyzed, and the settings of the presentation are changed pursuant to this reaction.

The analysis of the reaction of the person can concern the changes of the physical position of the limbs, the change of facial expression, the change of the heartbeat and so on. Sex and approximate age of the person before the display unit can also be determined.

The change of the setting of the presentation of the information can include the choice of the background sound, the change in volume of the sound, the creation of the lighting atmosphere vicinity of the device, the releasing of fragrances or change of the content of the presentation. The influences of the environment such as the barometric pressure, the weather and so on can be also taken into account when adjusting the presentation.

The device according to this invention allows to effectively determine and evaluate the attitudes of the consumer and to affect her by means of the presentation which adjusts itself according to her feedback. For example, if the person before the device reacts positively to the accompanying music - which can be manifested by the rhythmical movements of the body of the person - the device can increase the volume of the accompanying music or choose another similar music track in the next course of the presentation.

The device has a wide-scale use in various sorts of organizations with both research as well as practical orientation (hospitals, companies, shops, services, public institutions).

The device offers the platform for the connection of the state-of-the-art knowledge from the field of the consumer neuroscience (neuromarketing), communication technologies and information technologies. The device is able to identify the public by means of an image analysis (sex, approximate age) and monitor its interest by means of the directionally adjustable camera. On the basis of the micro-emotion determined from the facial expressions it can set an atmosphere of the environment by means of a various colors, fragrances and sounds, whereby it thoroughly takes the exterior weather and the interior temperature into account. The public is being offered the information concerning the actual time, temperature and date. The device at the same time communicates the activities of the organization (presentation video) to the public. The device offers the researchers the sufficient information concerning how engaging the presentation is and concerning the reaction of the public under the influence of the activated factors affecting the perception of the environment (colors, music, smells). The connection of the communication, information and research fields into one functional whole with the modern design can produce the desirable interior atmosphere and psychological well- being on the work-place. This mutual interconnection of the functional wholes can be verified in detail, evaluated and then adjusted in order to satisfy the public. The motion sensors allow activation of the chromatic lighting, the sounds and the spreading of fragrances in the vicinity of the panel. The control unit of the panel will take the moments of presence (or absence) of the potential public fully into account in order to save the energy as much as possible.

The device can on a given day and time present the activities, projects or advertisements related to or taking place in the given locality, for example at the workplace. This communication work will be supplied by various factors affecting the overall perception of the environment (colors, smells, sounds) and psychological well-being (micro-emotions) of the public. In case of the movement around the device this will be capable of detecting the presence of the persons by means of the motion or presence sensor, respectively, and instantly adjust the environment by means of colors, sounds and fragrances after taking the weather and environment temperature, and also the sex and approximate age of the passerbies into account. The advantage of this device is that it can at the same time record the reaction of the public (mainly on the basis of the facial expressions) for the purposes of the presentation and the flexibility of the device. The gathered feedback allows a better analysis of the reactions and the subsequent adjustments which help to better the presentation itself, or to better the communication of the device and the production of the engaging environment. Because the device has the appropriate peripheral elements for the influencing of the atmosphere of the presentation, the device is able not only to collect the data for further evaluation of the effects of the presentation, but also to immediately adjust the actual conditions of the presentation.

The device and the method according to this invention can be used in the marketing departments of the business companies, in all business fields, in research-marketing agencies, in advertising agencies, in the research and scientific centers, in non-profit organizations, in public institutions (offices, stations, hospitals, universities, schools), and so on. The device with the connection to the communication network can be fully remotely controlled; the presented content as well as the criteria of the presentation can also be changed remotely. At the same time the gathered data and information can be got from the device by means of a remote access. This allows processing multiple devices by means of a central point.

The advantage of this invention is the automatic process of gathering of the feedback and the ability to affect the conditions of the presentation. There exists a broad scale of various presentation devices, but all of them operate on the principle of one-sided providing of the information stored in advance or of remotely received information. This invention breaks this stereotype and offers technical means for a more flexible provision of information, where the presentation can adjust itself to the immediate interaction with the public and the device can with some probability gauge the attitude of the public and "live" with it interactively. This makes the participants more interested in the content of the presentation. The device and method according to this invention is empathetic, which the public unconsciously finds close and human. Brief description of the drawings

The invention is further disclosed by the drawings 1 to 3. The shape of the panel as well as the precise position of the individual elements within the device are for illustration purposes only and cannot be understood as limiting the scope of protection.

Figure 2 shows the independently standing device which has complex equipment with multiple peripheral elements.

Figure 2 shows the connection diagram of the basest arrangement of the information device with simultaneous gathering of the feedback.

Figure 3 is a connection diagram with the more complex arrangement.

Examples of realization

Example 1

In this example according to the figure 2 the information device has a display unit 1 in the form of a flat wide screen, which is connected to a miniature PC 3. Speakers 2 are part of the display unit 1_. The computer 3 has a memory 4, in which a file of multiple presentations is stored, among other things.

A stable camera 6, which has two optical members with relative lateral distance, is connected to the computer 3. The image output is analyzed in its own module 7 of the recognition of the position of the person. Information concerning the actual position of the person before the display unit 1 arrive to the computer 3 from this module 7.

The controlling program is in the computer 3; this program directions the directionally positionable camera 5 on the basis of the determined position of the person. This camera 5 records the image of the face of the person; the image data are analyzed in the facial expressions evaluation unit; the result is stored in the memory 4 alongside the precise time with regard to the presented multimedia content with various information. According to the image of the face this unit 8 gauges the age and sex of the person before the display unit 1_. The results of the assessment of the facial expressions are not only stored in the memory, but the presentation of the multimedia files is also altered according to them. The feedback is assessed by the set criteria, for example according to the gauged age and sex the music accompanying the presentation of the information changes. If the person moves in its rhythm, the device assesses this situation as an agreement with the choice of music and increases its volume.

Example 2

Information device according to the figures 1 and 3 has an outer form of the independently standing panel 9. In this panel the individual elements of the device are stored. In the front wall there is the display unit 1, above it there are two stable cameras 6, a directionally positionable camera 5, alphanumeric displaying element 12 and the excitation indicator 14. Below the display unit 1 there is a motion sensor H of the presence. On the sides of the panel 9 there are strips with RGB LED lighting 13, the output of the aromatizer 10 and side speakers 2.

The display unit 1, directionally positionable camera 5, communication element 9, two speakers 2, two stable cameras 6 with the module 7 of the recognition of the position of the person (Kinect system) and the excitation indicator 14 are connected to the mini computer 3 which is stored in the casing of the panel 9. The other elements of the device are connected to the mini computer 3 indirectly by two microcontrollers 17. One microcontroller 17 controls the alphanumeric display element 12 and a temperature sensor 16. RGB LED lighting 13, motion sensor H of the presence, sensor of the barometric pressure 15 and aromatizer 10 are connected to the second microcontroller 17.

The excitation indicator 14 in this example operates in two basic regimes. In first regime it operates as a luminous accompaniment to the sound, where it dynamically shines according to the rhythm and changes in the tones of the reproduced music. In the second regime it displays an excitation based on the device's capability to recognize the facial expressions or other assessed feature. For example, five shining green LEDs indicate a great contact, that is, very good feedback. Three orange LEDs indicate good contact; single red LED indicates minimal contact with the person before the device. A particular luminous manifestation can be different in another example, with different number and color of the lights; what is important is that the excitation indicator 14 is an externally observable dynamic manifestation of the function of the device.

In the stand-by mode the motion sensor V\_ of the presence is active. After the person approaches the device, the sensor V\_ of the presence activates the multimedia presentation. After the initial part of the presentation the approximate age and the sex of the person before the displaying unit 1 is determined by means of a stable camera 6 and a recognition module 7. This information determines the choice of the subsequent content of the presentation. For example, the information is presented in an appropriately simpler form to the children.

Information about the position of the person, the limbs and the face are continuously sent from the module 7 of recognition of the position of the person to the mini computer 3. The directionally positionable camera 5 is directed to the position of the face and the emotions and attitudes of the person are analyzed from its image. The evaluation of the image data from the cameras 5, 6 are gathered from the reactions of the person to the presented information. These reactions are stored in the memory 4 together with the time synchronization. These reactions are also used for simultaneous influencing of the mode of presentation. According to the pre-programmed criteria the color and intensity of the RGB LED lighting 13 changes, the fragrance is released by the aromatizer 0, the accompanying sound and its intensity changes. During these changes the temperature datum from the temperature sensor 16 and the datum of the barometric pressure - roughly determining the immediate weather - are taken into account. The weather affects the mood of the person and pursuant to the setting it can also change the content of the presented information.

The communication element 19 is connected the mini computer 3 in this example; the communication element 19 allows the wireless transfer in both directions; in one direction it loads the multimedia files and criteria of presentation and in the other direction it transfers raw gathered data or processed data concerning the feedback. The communication element 19 can be a network card by which the device is connected to the internet. The internet connection can be used not only for the bi-directional communication with the control of the device, but also for gathering of the data concerning the local weather from the respective websites. All measured data and the recorded video can be wirelessly transferred by means of a communication element 9. By means of the internet connection, the content of the presentation can be remotely altered or supplied, and the software controlling the device and mutual relations between elements and functions can be updated.

The device also has free ports for the connection of the additional peripheral elements such as, for example, external sensors of the humidity, temperature and lightning.

In this example the directionally positionable camera 5 has a remote control 8; it can be controlled from the vicinity of the device. Directionally positionable camera 5 can be rotated manually, too.

Industrial applicability

The industrial applicability is obvious. According to this invention it is possible industrially and repeatedly construct and use the information device, which simultaneously gathers feedback and changes the surrounding perceptions and mode of multimedia presentation of the information. List of related symbols

1 - display unit

2 - speaker

3 - computer

4 - memory

5 - directionally positionable camera

6 - stable camera

7 - person's position recognition module 8 - facial expressions' evaluation unit

9 - panel

10 - aromatizer

1 1 - presence sensor

12 - alphanumeric display element 13 - RGB LED lighting

14 - excitation indicator

15 - barometric pressure sensor

16 - temperature sensor

17 - microcontroller

18 - remote control of the camera 19 - communication element

PC - personal computer

Claims

PATENT CLAI MS

1. Information device with a simultaneous collection of a feedback, where the device includes a display unit (1) for displaying of visually presented information, at least one speaker (2) and a computer (3) with a memory (4) for a presentation of the information, where the display unit (3) is connected to the computer (3), is characterized by the fact, that

it includes two cameras (5, 6), whereby one camera (5) is directionally positionable and other camera (6) is stable,

both cameras (5, 6) are oriented to a space before the display unit (1), the stable camera (6) is connected to a module (7) for recognition of a person's position and for determination of a position of a face, an output from this module (7) is connected to a control of a direction of the positionable camera (5),

and an output from the positionable camera (5) is connected to the computer (3), in which there is a unit (8) for an evaluation of facial expressions.

2. Information device with the simultaneous collection of the feedback according to claim 1, is characterized by the fact, that both cameras (5, 6) are located on a front side of the device where the display unit (1) is, whereby the cameras (5, 6) are located above the display unit (1).

3. Information device with the simultaneous collection of the feedback according to claim 1 or 2, is characterized by the fact, that it has an outer casing in form of an independent panel (9), which is designed for an independent standing on a floor or for hanging on a wall.

4. Information device with the simultaneous collection of the feedback according to any of the claims 1 to 3, is characterized by the fact, that it has an aromatizer (10).

5. Information device with the simultaneous collection of the feedback according to any of the claims 1 to 4, is characterized by the fact, that it has a sensor (11) of a presence connected to the computer (3), preferably the presence sensor (11) is a motion sensor or a noise sensor or a thermal imaging sensor or a pressure sensor in a pad before the device.

6. Information device with the simultaneous collection of the feedback according to any of the claims 1 to 5, is characterized by the fact, that it has an alphanumeric display unit (12).

7. Information device with the simultaneous collection of the feedback according to any of the claims 1 to 6, is characterized by the fact, that it has an RGB LED lighting (13) with an adjustable chromaticity, preferably the RGB LED lighting (13) is placed on sides of the panel (9).

8. Information device with the simultaneous collection of the feedback according to any of the claims 1 to 7, is characterized by the fact, that it has an excitation indicator (14).

9. Information device with the simultaneous collection of the feedback according to claim 8, is characterized by the fact, that the excitation indicator (14) is designed for a luminous accompaniment to a sound, preferably following a rhythm or changes in a pitch.

10. Information device with the simultaneous collection of the feedback according to claim 8 or 9, is characterized by the fact, that the excitation indicator (14) is designed for a display of a degree of an ability to recognize the facial expression.

11. Information device with the simultaneous collection of the feedback according to any of the claims 1 to 10, is characterized by the fact, that it has a sensor (15) of a barometric pressure

12. Information device with the simultaneous collection of the feedback according to any of the claims 1 to 11, is characterized by the fact, that it has a temperature sensor (16) and/or a humidity sensor.

13. Information device with the simultaneous collection of the feedback according to any of the claims 1 to 12, is characterized by the fact, that it has at least one microcontroller (17) for a connection of peripheral elements with the computer (3).

14. Information device with the simultaneous collection of the feedback according to any of the claims 1 to 13, is characterized by the fact, that it has a communication element (19) which is connected to the computer (3).

15. Information device with the simultaneous collection of the feedback according to the claim 14, is characterized by the fact, that the communication element (19) is connected to a network for a remote control of a presented content and/or for the remote collection of data gathered by the device.

16. Information device with the simultaneous collection of the feedback according to any of the claims 1 to 15, is characterized by the fact, that it has a remote control (18) of the directionally positionable camera (5).

17. A method of presentation of information by means of a display unit (1) and a speaker (2) is characterized by the fact, that at least one camera (5, 6) captures a person before the display unit (1) and on a basis of a recorded image a reaction of the person is analyzed and settings of the presentation of the information are altered on a basis of the reaction.

18. The method of presentation of the information by means of the display unit (1 ) and the speaker according claim 17 is characterized by the fact, that the camera (5, 6) records a bodily position of limbs and/or a facial expression and/or changes in a heartbeat.

19. The method of presentation of the information by means of the display unit (1) and the speaker according claims 17 or 18 is characterized by the fact, that on the basis of the image data a sex and an approximate age of the person before the display unit (1) is determined and on the basis of these data a mode and/or a content of the presentation is chosen.

20. The method of presentation of the information by means of the display unit (1) and the speaker according any of the claims 17 to 19 is characterized by the fact, that a change of the mode of presentation includes the change of an accompanying sound and/or the change of a volume and/or the creation of a luminous atmosphere in the vicinity of the device and/or releasing of fragrances.

21. The method of presentation of the information by means of the display unit (1) and the speaker according any of the claims 17 to 20 is characterized by the fact, that an actual weather is determined by measuring a barometric pressure and pursuant to the weather the settings of the presentation of the information are altered.

22. The method of presentation of the information by means of the display unit (1) and the speaker according any of the claims 17 to 21 is characterized by the fact, that by means of a connection to the internet the weather in a locality of the device is found out and pursuant to the weather the settings of the presentation of the information are altered.

23. The method of presentation of the information by means of the display unit (1) and the speaker according any of the claims 17 to 22 is characterized by the fact, that according to instructions from a remote center the presented content and/or the conditions of presentation are changed by means of the internet connection.

24. The method of presentation of the information by means of the display unit (1) and the speaker according any of the claims 17 to 23 is characterized by the fact, that the data collected by the device are transferred to the remote center by means of the internet connection, preferably the data from multiple devices are transferred to the single remote center.