KR20100083519A - Actual feeling book systme for multimodal immersive interaction and method therefor - Google Patents

Abstract

PURPOSE: An actual feeling book system for multimodal immersive interaction and a method thereof are provided to together experience sensation contents when a user reads print media, thereby maximizing learning, experience and play effect. CONSTITUTION: An interest area recognition unit(120) recognizes an interest area of a user about print media based on the action of the user. A main controller(130) searches and prints sensibility contents corresponding to the interest area from a contents database(140). A renderer(150) renders the sensibility contents according to the control of the main controller. The rendered contents is outputted to a video, an audio, or a haptic device(158-162).

Description

Actual Feeling Book Systme for Multimodal Immersive Interaction and Method Therefor}

The present invention is directed to sensory systems and methods for multiple immersive interactions. More specifically, the present invention relates to a system and method for immersive interaction for multi-immersive interaction that enables a user to experience physical analogue print media as well as sensory content such as video, audio, and haptic associated with the print media. will be.

Based on the development of various information technologies, technologies are being developed to make the contents of actual books into digital contents so that the contents can be accessed through the terminal anytime and anywhere.

In its most basic form, an E-Book is simply a replacement for an analog book with digital content, which does not provide the reader with realistic interaction effects.

Billinhurst et al. Applied the Augmented Reality technology through the concept of MagicBook, attaching AR Marker on the real book and augmenting the virtual object to the real book to give the reader a realistic experience. A proposed concept was proposed (M. Billinghurst, H. Kato, and I. Poupyrev, "The MagicBook: a traditional AR intreface", Proc. Of Computer and Graphics, vol. 25, pp. 745-753, 2001.) . Back et al. Conducted research to enhance the sound when readers read using electrical sensors (M. Back, J. Cohen, R. Gold, S. Harrison, and S. Minneman, "Listen Reader: an electonically augmented paper-based book ", Proc. of CHI'01, pp 23-29, 2001).

However, in the existing technology, only the visual or auditory effects are mainly provided for books or printed materials, but the sense of touch, which is an important sensory organ of the human body, was not considered. In addition, the prior art is to allow the user to manually watch the audio-visual content or only limited operation. Such a technique has a disadvantage in that the reader cannot be immersed in the provided content sufficiently and the learning book does not have sufficient learning effect.

The present invention provides a multi-immersive interaction system and method for maximizing learning, experience, and play effects by allowing a user to experience sensory content such as video, audio, and haptic when reading a print medium such as a book. The purpose is to provide.

In order to achieve the above object, the present invention is a system for providing multiple immersive interaction using a print medium, the region of interest recognition unit for recognizing the user's region of interest on the print media based on the user's behavior ; A main control unit which controls to output the sensory content corresponding to the ROI from a content database; And a renderer that renders the sensory content under the control of the main controller, so that the rendered content is output to a video device device, an audio device device, or a haptic device. To provide.

Preferably, the sensory book system further includes a print medium recognition unit recognizing a print medium based on an image or a marker of a print medium, wherein the print medium recognition unit comprises: a camera for acquiring an image of the print medium, the And at least one of a bar code reader for recognizing a bar code formed on a print medium or an optical page recognizing device for recognizing a page number formed on the print medium.

Preferably, the ROI recognition unit may include an ultrasound receiver configured to receive an ultrasound transmitted from an ultrasound transmitter worn by the user and determine a position.

Preferably, the ROI recognition unit may include any one of a marker for estimating an ROI worn by the user, an inertial sensor worn by the user, a sensing panel based on an electromagnetic induction method, or a touch pad provided below the printing medium. Recognizing the region of interest of the user by recognizing or having one.

Preferably, the content database includes at least one of video content, audio content, haptic content, and olfactory content, and the main control unit determines whether the corresponding content exists in the content database based on a recognition result of the ROI recognition unit. Search.

Meanwhile, the multi-immersive interaction system may be configured to include a table-shaped body for disposing the print media.

In addition, the main control unit, it is preferable to provide the user with a method of immersing the content in a visual or auditory way.

According to still another embodiment of the present invention, there is provided a method of providing multiple immersive interaction using a print medium, comprising: (a) recognizing a region of interest of a user with respect to the print medium; (b) retrieving from the content database whether there is content corresponding to the region of interest; (c) rendering the content if it exists as a result of performing step (b); And (d) providing the rendered content to a user.

Preferably, before the step (a), further comprising the step of recognizing the print medium, the image of the print medium to track and recognize the marker, to recognize the barcode formed on the print medium Or recognizing a page number formed on the printing medium.

Preferably, the step (a) may include receiving an ultrasonic wave transmitted from an ultrasonic transmitter worn by the user, tracking a marker for estimating a region of interest worn by the user, and a sensing panel using an electronic flow method. It may be performed by one of the methods of recognizing using the recognition, or by having a touch pad below the print medium.

Preferably, as a result of performing the step (b), the user is informed of the form or a sense method presented according to the type of the searched content in the form of video or audio.

The content stored in the content database may be set based on the type and page information of the print media and the area information of a specific page.

According to the present invention, by recognizing the type of analog print media such as books and corresponding pages, by tracking the user's region of interest, the sensory contents such as video and audio, tactile content, and olfactory content are provided to the user. There is an effect that allows the user to provide a more realistic and immersive experience.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings. First, in adding reference numerals to the components of each drawing, it should be noted that the same reference numerals are assigned to the same components as much as possible, even if shown on different drawings. In the following description of the present invention, a detailed description of known functions and configurations incorporated herein will be omitted when it may make the subject matter of the present invention rather unclear. In addition, the following will describe a preferred embodiment of the present invention, but the technical idea of the present invention is not limited thereto and may be variously modified and modified by those skilled in the art.

1 is a block diagram of a multi-immersive interaction system in accordance with a preferred embodiment of the present invention.

The present invention provides multiple immersive interactions that allow a user to integrally enjoy analog print media 10, such as actual physical books, and immersive content such as audiovisual and tactile content associated with the print media 10. It features.

Multi-immersive interaction system 100 according to a preferred embodiment of the present invention for this purpose, the print medium recognition unit 110 for recognizing the print medium, and the region of interest in the print medium 10 Region of interest recognition unit 120 that recognizes the control unit, a main control unit 130 that controls to output the corresponding content from the content database 140 based on the recognition result, and video and audio under the control of the main control unit 130. Or a ladder 150 for rendering haptic content. The multi-immersive interaction system 100 further includes a content database 140 that stores and provides video, audio, or haptic content.

The print medium 10 includes all physical media such as books, drawings, catalogs, and posters. However, hereinafter, the present invention will be described with reference to a book composed of a plurality of pages.

The print medium recognition unit 110 performs a function of recognizing the print media currently viewed by the user. Further, the print medium recognition unit 110 performs a function of recognizing the page when the print medium consists of several pages. As an example of the print medium recognition unit 110, the print medium recognition unit 110 is an image tracking unit (not shown) for determining the type of the print medium 10 from the camera (not shown) and the image input from the camera. Can be done. To this end, the print medium 10 may be provided with a marker such as a barcode. The marker may be provided in a visible or invisible form. In addition, the marker may be a feature point of a printing medium in addition to the form of a barcode. The feature point of the printing medium may be a page number displayed on a specific area of the printing medium, a picture formed on the printing medium, or the like. An invisible marker may be, for example, a marker represented by ink containing carbon. In this case, the camera may be configured to include an infrared light source and recognize an image generated according to the infrared light source.

However, in the practice of the present invention, the print medium recognition unit 110 may be omitted. This includes information that the content storage memory 272 identifies the print media when the user enters a particular print media through the system or through the external connection port 270 described below. This particular print medium can be identified.

The ROI recognizer 120 performs a function of identifying a region of interest to the print media 10. The user may express the degree of interest by pointing the region of interest through a hand or finger. Alternatively, the user may indicate an ROI using an operation tool such as a TUI (Tangible User Interface).

A detailed example of recognizing and determining the ROI of the user by the ROI recognizer 120 will now be described.

As an example, a user may include an ultrasonic transmitter in a finger, and the ROI recognizer 120 may include an ultrasonic receiver configured to receive the ultrasonic wave generated from the ultrasonic transmitter to track the position of the ultrasonic wave generated from the ultrasonic transmitter. Estimate the position of.

As another example, an ROI marker may be attached to a user's hand or finger, and the ROI recognizer 120 may estimate the position of the user's finger by tracking the position of the ROI marker in the image input from the camera. Can be. However, when the marker is used as the tracking means, a lot of influence of the illumination may occur, and a vibration may occur due to its characteristics, and thus, a finger position may not be accurately determined. In addition, if there is a camera covering, it may become impossible to recognize the marker. Accordingly, in the present invention, as described above, the user may have an ultrasonic transmitter on the finger, or as shown in the following description, to grasp the user's intention from the physical means or the user's gesture to experience the viewing tactile content in the book. It may be desirable.

As another example, a sensing panel using an electromagnetic induction method may be installed on a table to estimate a current hand or finger position.

As another example, the touch pad may be provided under the print medium 10 to estimate a position where the user touches the finger while viewing the print medium.

As another example, the user may wear an inertial sensor on the hand or finger to detect the rotation of the hand or finger by estimating the hand or finger rotation to estimate page turn, interest in the corresponding content, or to obtain the user's The image of the hand may be extracted to receive an operation signal from the movement of the hand.

 The main control unit 130 controls to output the corresponding sensory content based on the results recognized by the print media recognition unit 110 and the ROI recognition unit 120.

The content database 140 stores sensory content that a user can feel with the five senses. Such sensational sensory content may be video content 142, audio content 144, or haptic content 146, and may also include content for providing olfactory information. The content database 140 may classify and store corresponding content according to a specific area or content of the print medium 10 and the print medium 10. Also, such contents are preferably stored together with an output condition and an output time at the time of storage. The content database 140 may be built in the system or may use an external memory device, and may be provided through a separate server device connected through a network such as the Internet.

The renderer 150 renders the corresponding content under the control of the main controller 130 and outputs the corresponding content to the display device. This renderer 150 includes video content 152 that renders video content, audio content 154 that renders audio content, or haptic renderer 156 that renders haptic content.

Meanwhile, the haptic content may be set to provide tactile information or kinesthetic information.

The content rendered by the renderer 150 is output to the video display device 158, the audio display device 160, or the haptic device 162 according to its type.

2 is a diagram showing an actual implementation of a multi-immersive interaction system according to a preferred embodiment of the present invention.

The multiple immersive interaction system 200 shown in FIG. 2 includes a table-shaped body 210, a video display device 158 and a haptic device 162 disposed around the body 210. The audio display device 160 such as a speaker may be configured separately from the main body 210, but may be mounted on the main body 210 as shown in FIG. 2. On the other hand, the haptic display device 162 may be provided in the same model as SensAble's OMNI ^TM series, preferably provided with a tactile display unit 164 composed of a piezoelectric element, a pneumatic element, etc. for transmitting the tactile sense.

The print medium 10 is configured to be placed on an upper portion of the main body 210, and the main body 210 is provided with an indication mark 212 to indicate a criterion for aligning the print medium 10.

The multi-immersive interaction system 200 according to FIG. 2 is a configuration corresponding to the print media recognition unit 110 in FIG. 1, and includes a camera 220, an optical page recognition device 230, and a barcode reader 240. ). In addition, the ultrasound receiver 250 is illustrated as a configuration corresponding to the ROI recognizer 120.

The camera 220 performs a function of acquiring an image including the print medium 10. The image acquired from the camera 220 may be output to the video display device 158. In addition, the video display device 158 may be augmented and output with the video content 142 selected from the content database 140 together with the image acquired from the camera 220. On the other hand, when the print medium recognition unit 110 is provided with a marker for identifying the print medium 10 on the print medium 10, the print medium 10 tracks the marker from an image input from the camera 220. It is also possible to identify.

The optical page recognition apparatus 230 may be used as a means for recognizing a page of the print medium 220. Preferably, a number or a sign indicating a page is displayed in a specific area of the print medium 220, and the optical page recognition device 230 is arranged at a position capable of recognizing the number or sign indicating the page.

The barcode reader 240 performs a function of recognizing the barcode provided in the print medium 10 to determine the print medium 10. 3 shows an example of a printing medium provided with a barcode on a cover, in which a barcode 300 is provided on the cover of the printing medium 10 and a barcode reader 240 is provided on the upper surface of the main body 210 as shown in FIG. In the case where the print media 10 is placed on the main body, the barcode reader 240 may recognize the barcode 300. In the meantime, when the barcode reader 240 recognizes the barcode 300, the orientation information of the barcode 300 may be grasped to determine the position and orientation information of the printing medium 10. When the position and orientation information of the print medium 10 is determined based on the position and orientation information of the barcode 300, the user may be interested in any part of the print medium 10 only with position information such as the user's finger. There is an advantage that can be determined.

On the other hand, the main body 210 may be equipped with a fragrance capsule 260 as a configuration for transmitting the smell to the user, and may be provided with a tuyere 262 for delivering the fragrance of the fragrance capsule 260. In this case, when it is necessary to transmit the olfactory information under the control of the main control unit 130, the fragrance of the fragrance capsule 260 can be transmitted to the user through the operation of the tuyere 262.

Next, the configuration including the ultrasound receiver 250 to recognize the ROI of the user will be described.

4 is a diagram illustrating a state in which an ultrasonic transmitter is attached to a user's finger.

The ultrasound receiver 250 determines the location of the finger by determining the position of the finger by receiving the ultrasound generated by the ultrasound transmitter 252. Meanwhile, in order to recognize the ROI of the user, the camera 220 provided at one side of the main body may be used. The user's interest level may be determined by acquiring an image of a user's finger or hand through the camera 220 and the ROI recognition unit 120 recognizes the image.

An external connection port 270 may be provided at one side of the main body 210, and a content storage memory 272 provided with the print medium 10 may be inserted into the external connection port 270. The content storage memory 272 stores content such as video, audio, etc. corresponding to the print medium 10, which may be understood as replacing the content database 140 of FIG. 1.

Next, a multi-immersive interaction method according to a preferred embodiment of the present invention will be described.

5 is a flowchart of a multi-immersive interaction method according to a preferred embodiment of the present invention.

The print medium recognition unit 110 recognizes a corresponding page when the type of the print medium 10 and the print medium 10 are composed of a plurality of pages (S500). The type of the printing medium 10 can be determined by the barcode reader 24 recognizing the barcode 330 printed on the front cover of the printing medium 10 as described above. In addition, after acquiring an image of the print medium 10 with the camera 220, the marker and the page of the print medium 10 may be recognized by tracking the marker. In addition, the optical page recognizing apparatus 230 may be used to recognize the page display number of the print medium 10.

Next, the ROI recognizer 120 determines which area of the print media 10 the user is interested in (S502). As described above, when the user wears the ultrasound transmitter 252 on the finger, the ultrasound receiver 250 may recognize the ROI of the user by receiving the ultrasound signal transmitted from the ultrasound transmitter 252. However, the recognition of the ROI of the user is not limited thereto, and various embodiments have been described above.

The main control unit 130 refers to the content database 140 to search whether there is content corresponding to the ROI of the user (S504), and determines the type of content (S506) when the corresponding content exists. In this case, the main controller 130 may inform the user in a visual or audio manner what kind of content is presented to the user and how the content can be realized according to the kind of content to be provided to the user. desirable. As a method for this purpose, the present invention defines a realistic story telling (story telling). Realistic storytelling refers to providing the user with information about how to feel the haptic content. For example, a haptic icon may be provided to a user to display a method of haptic content. The haptic cone is given as an icon that is stabbed by a finger, an icon that is rubbed by a finger, and an icon that is grabbed and pulled. The haptic cone is induced so that a user can effectively perceive the haptic content by looking at the icon. Alternatively, the author or teacher of the sensational book may provide a guide for providing a sensational method of haptic content in video or audio. As another method, the main controller 130 may control the haptic device 162 linked with the sensory book to perform a trial operation so that the user may demonstrate the haptic content.

If the corresponding content is the video content 142, the main controller 130 acquires the video content 142 (S508), causes the video renderer 152 to perform rendering, and outputs the video content to the video display device 158. (S510).

If the corresponding content is the audio content 144, the main controller 130 obtains the audio content 144 (S512), causes the audio renderer 154 to perform rendering, and then outputs the audio content to the audio display device 160. (S514).

If the corresponding content is the haptic content 146, the main controller 130 obtains the haptic content 146 (S516), causes the haptic renderer 156 to perform rendering, and outputs the haptic device 162 to the haptic device 162. (S518). In this case, the haptic content 146 may be in the form of providing a sense of force depending on whether or not the collision with the virtual object, or in the form of providing a tactile feeling such as surface roughness, pressure or vibration, and thus also the haptic device 162 It may be a feeling-sensing device or a haptic device using a vibrator or the like. Since the user may not be able to feel the haptic content 146 when the user is pointing to a specific area of the print medium 10 while the user wears the ultrasonic transmitter 252 on the hand, the haptic content 146 In the case of presentation, it is preferable to inform the user that the haptic content 146 is or is being presented through the video display device 158 or the audio display device 160.

If the corresponding content is olfactory content, the olfactory content information is obtained (S520), and the fragrance of the fragrance capsule 260 is discharged through the air blowing unit 262 so that the user can feel the olfactory content (S522). .

On the other hand, the content stored in the content database 140, it is preferable to set the conditions to be output based on the type and page information of the print media 10 and the area information of a specific page. To this end, in generating content, it is preferable to include ID information of a book, a page, a planar position (eg, x and y coordinates) to be disposed on the page, and time information to be presented.

The above description is merely illustrative of the technical idea of the present invention, and various modifications, changes, and substitutions may be made by those skilled in the art without departing from the essential characteristics of the present invention. will be. Accordingly, the embodiments disclosed in the present invention and the accompanying drawings are not intended to limit the technical spirit of the present invention but to describe the present invention, and the scope of the technical idea of the present invention is not limited by the embodiments and the accompanying drawings. . The protection scope of the present invention should be interpreted by the following claims, and all technical ideas within the equivalent scope should be interpreted as being included in the scope of the present invention.

1 is a block diagram of a multi-immersive interaction system in accordance with a preferred embodiment of the present invention;

2 illustrates an actual implementation of a multi-immersive interaction system in accordance with a preferred embodiment of the present invention.

3 is a diagram showing an example of a printing medium provided with a barcode on a cover sheet;

4 is a view illustrating a state in which an ultrasonic transmitter is attached to a user's finger;

5 is a flowchart of a multi-immersive interaction method according to a preferred embodiment of the present invention.

<Description of reference numerals for the main parts of the drawings>

10: print medium 110: print medium recognition unit

120: region of interest recognition unit 130: subject control unit

140: Content Database 150: Renderer

158: video display device 160: audio display device

162: haptic device 210: main body

220: camera 230: optical page recognition device

240: barcode reader 250: ultrasonic receiver

252: ultrasonic transmitter 260: aroma capsule

300: Barcode

Claims (13)

A system for providing multiple immersive interactions using print media, A region of interest recognition unit configured to recognize a region of interest of the user with respect to the print media based on a user's behavior; A main control unit which controls to output the sensory content corresponding to the ROI from a content database; And Renderer for rendering the sensory content under the control of the main controller And a immersive interaction system for multiple immersive interactions that includes outputting content rendered in the renderer to a video device device, an audio device device, or a haptic device. The method of claim 1, A print medium recognizing unit may further include a print medium recognizing unit recognizing a print medium based on an image or a marker of a print medium. The print medium recognizing unit may include a camera acquiring an image of the print medium and a barcode reader recognizing a bar code formed on the print medium. Or an optical page recognizing apparatus for recognizing a page number formed on the printing medium. The method of claim 1, The ROI recognition unit, Sensing system for multi-immersive interaction, characterized in that it comprises an ultrasonic receiver for determining the position by receiving the ultrasonic wave transmitted from the ultrasonic transmitter worn by the user. The method of claim 1, The ROI recognition unit, The user's interest by recognizing or providing any one of a marker for estimating a region of interest worn by the user, an inertial sensor worn by the user, a sensing panel by an electromagnetic induction method, or a touch pad provided under the printing medium A sensory book system for multiple immersive interactions characterized by domain recognition. The method according to any one of claims 1 to 4, The content database includes at least one of video content, audio content, haptic content, and olfactory content, and the main control unit searches whether the corresponding content exists in the content database based on a recognition result of the ROI recognition unit. A sensory book system for multiple immersive interactions. The method of claim 5, The multiple immersive interaction system, And a table-shaped body for disposing the print medium. The method of claim 5, And the content database is stored in a server device embedded in the system, stored in an external memory device, or connected to a network, and read out under the control of the controller. The method of claim 5, The main control unit, the sensory book system for multi-immersive interaction, characterized in that to provide the user with a sensation method of the content in a visual or auditory way. A method of providing multiple immersive interaction using a print medium, (a) recognizing a region of interest of the user for the print media; (b) retrieving from the content database whether there is content corresponding to the region of interest; (c) rendering the content if it exists as a result of performing step (b); And (d) providing the rendered content to a user Multiple immersive interaction method comprising a. The method of claim 9, Before the step (a), further comprising the step of recognizing the print medium, the image of the print medium to track and recognize the marker, to recognize the barcode formed on the print medium, or the printing A method for multi-immersive interaction, characterized in that it is carried out in any one of the manner of recognizing the page number formed on the medium. The method of claim 9, In the step (a), receiving the ultrasonic wave transmitted from the ultrasonic transmitting unit worn by the user, tracking the marker for estimating the region of interest worn by the user, and using a sensing panel using an electronic flow method. Multi-immersive interaction method characterized in that it is carried out in any one of the way, or by having a touch pad below the printing medium to recognize. The method according to any one of claims 9 to 11, And (b) informing the user of a form or a sensation method presented according to the type of the retrieved content as a result of the step (b). The method according to any one of claims 9 to 11, And the content to be output in the content database is set based on the type and page information of the print media and area information of a specific page.

Images