KR101051433B1 - Game machine that responds to user's state judged by eye image and its game providing method - Google Patents

Abstract

Provided are a game machine and a game providing method which responds to a user state determined through an eye image. The game machine receives an eye-image of a game user's eyes from an external photographing apparatus, determines a user-state based on the eye-state measured based on the received eye-image, and determines the determined user-state. Control game content as a basis. As a result, it is possible to provide an intelligent game that matches the mood of the game user and does not cause fatigue. In addition, by measuring the emotion information of the game user in real time, by providing the game content suitable for the emotion, it is possible to maximize the interesting elements of the game.

Game, System, Pupil, Blinking, Emotion, Fatigue

Description

Game apparatus for interacting according to user's status determined by eye image and method for providing game}

The present invention relates to a game machine and a game providing method thereof, and more particularly, to a game machine and a game providing method that responds intelligently according to the user's state.

Real-time multimedia content output system and method using the EEG signal (application number: 10-2007-0060599) measures the brain wave from the user of the multimedia content, and applied to a predefined model to measure the emotional state, according to the multimedia signal ( Music or video).

However, in order to enjoy multimedia contents, that is, an additional sensor for measuring brain waves must be attached to each part of the user's body, the user may feel uncomfortable.

The present invention has been made to solve the above problems, an object of the present invention is to control the game content on the basis of the user-state determined based on the eye-state measured based on the eye-image of the game user To provide a game machine.

In addition, an object of the present invention is to enable the objective sensitivity and fatigue measurement through the wearable eye-imaging device. In addition, the present invention is to provide a game content suitable for the measured emotion to enjoy a game corresponding to the current mood state of the user.

In addition, the present invention provides a game that does not feel fatigue by adjusting the game screen configuration and the image quality parameter suitable for the measured fatigue degree.

In accordance with an aspect of the present invention, a game machine includes: a receiver configured to receive an eye-image of a game user's eyes from an external photographing apparatus; A measuring unit measuring an eye state based on the eye-image received by the receiving unit; A determination unit determining a user-state based on the eye-state measured by the measurement unit; And a controller for controlling game contents based on the user-state determined by the determination unit.

The eye-state preferably includes at least one of pupil size, pupil adaptation rate, eye blink frequency and eye blink rate.

The user-state preferably includes at least one of sensitivity and fatigue.

The determination unit may determine the emotion of the game user with reference to the pupil size and the blinking speed, and determine the fatigue level of the game user with reference to the blinking frequency and the pupil adaptation speed.

The determination unit may set threshold values for the pupil size, the eye blinking speed, the eye blinking frequency, and the pupil adaptation in the calibration process before the game starts.

The controller may adjust at least one of a configuration and a screen state of game content based on the user-state.

On the other hand, the game providing method according to the invention, the step of receiving an eye-image photographing the eyes of the game user from the external shooting device; Measuring an eye state based on the eye image received in the receiving step; Determining a user-state based on the eye-state measured in the measuring step; And controlling game contents based on the user-state determined in the determining step.

The eye-state preferably includes at least one of pupil size, pupil adaptation rate, eye blink frequency and eye blink rate.

The user-state preferably includes at least one of sensitivity and fatigue.

In the determining step, it is preferable to determine the emotion of the game user with reference to the pupil size and the eye blinking speed, and to determine the fatigue degree of the game user with reference to the eye blinking frequency and the pupil adaptation speed.

In the determining step, a threshold value for the pupil size, the eye blinking speed, the eye blinking frequency, and the pupil adaptation can be set through the calibration process before starting the game.

In the controlling step, it is preferable to adjust at least one of the configuration of the game content and the screen state based on the user-state.

On the other hand, the eye-image capturing apparatus according to the present invention, the infrared illumination for emitting infrared light to the eyes of the game user; An infrared camera for photographing the eyes of a game user whose infrared light is emitted by the infrared light to generate an eye-image; And a transmission unit for transmitting the eye-image generated by the infrared camera to an external game machine.

The infrared illumination and the infrared camera may be installed in the eye-image capturing apparatus so as to face the eyes of the game user.

The eye-imaging device may be implemented in at least one of an HMD, stereo glasses, goggles, and a helmet.

Meanwhile, the eye-image capturing method according to the present invention comprises the steps of: emitting infrared rays to the eyes of a game user; Photographing the eyes of the game user from which the infrared rays are emitted to generate an eye-image; And transmitting the eye-image to an external game machine.

As described above, according to the present invention, it is possible to provide an intelligent game that matches the mood of the game user and does not cause fatigue. In addition, by measuring the emotion information of the game user in real time, by providing the game content suitable for the emotion, it is possible to maximize the interesting elements of the game.

Existing sensitivity and fatigue analysis was often conducted through a subjective questionnaire, but in the present invention, by determining the user's eye image in real time, the emotion and fatigue information not damaged by the user's prejudice may be measured.

In addition, according to the present invention, by measuring the fatigue degree of the game user in real time, by providing a method to reduce the fatigue or recommend a break, it is possible to build an intelligent game system capable of context awareness (context awareness).

In addition, the apparatus and method for fatigue and sensitivity measurement may be usefully used in other fields that require the measurement of fatigue and sensitivity information.

Hereinafter, with reference to the drawings will be described the present invention in more detail.

1 shows a game-system to which the present invention is applicable. As shown in FIG. 1, the game system is constructed by connecting the eye-image capturing apparatus 100, the game machine 200, and the TV 300 to communicate with each other.

In FIG. 1, the eye-image capturing apparatus 100 and the game machine 200 are illustrated to be wirelessly connected. However, when it is not possible to connect them wirelessly, it is also possible to connect the eye-image photographing apparatus 100 and the game machine 200 so as to communicate by wire.

In addition, the game machine 200 and the TV 300 is shown as being connected to enable communication by wire. However, they can also be connected to communicate wirelessly.

The eye-image capturing apparatus 100 transmits the eye-image obtained by photographing the game user's eyes to the game machine 200. The eye-image capturing apparatus 100 may be implemented in various forms, and may be implemented in the form of a device in which infrared-based lighting and a camera are attached to a head mounted display (HMD), stereo glasses, goggles, and a helmet.

If the eye-imaging device 100 is implemented in the form of an HMD, the TV 300 may be omitted and the function may be absorbed into the HMD.

2 shows an example of an eye-image capturing apparatus, an HMD having an infrared light and an infrared camera attached thereto for eye-image capturing. The eye-image capturing apparatus 110 illustrated in FIG. 2 includes an HMD 111, an infrared light 112, and an infrared camera 113.

The HMD 111 functions as a display for providing a game image to a game user. In addition, the infrared light 112 is directed toward the eyes of the user and emits infrared rays to the eyes of the game user. In addition, the infrared camera 113 also faces the user's eyes, and photographs the game user's eyes to generate an eye-image.

And, although not shown in Figure 2, the eye-image generated by the infrared camera 113 is transmitted to the game machine 200 through the transmission means.

Meanwhile, FIG. 3 illustrates another example of an eye-image capturing apparatus, stereo glasses with an infrared light and an infrared camera for eye-image capturing. The eye-image capturing apparatus 120 illustrated in FIG. 3 includes stereo glasses 121, an infrared light 122, an infrared camera 123, and a mirror 124.

The mirror 124 is attached to the front of one lens of the stereo glasses 121 at a 45 degree angle. The mirror 124 reflects infrared rays into the eyes of the game user, but transmits visible light.

Since the mirror 124 transmits visible light and is transparent to the user's eyes, there is no problem for the game user to look forward.

Infrared light 122 emits infrared light toward mirror 124, ultimately irradiating infrared light into the eyes of the game user.

In addition, the infrared camera 123 is emitted from the infrared light 122 and reflected from the game user's eyes, and then receives the infrared rays reflected back from the mirror 124 to photograph the game user's eyes.

In addition, although not shown in FIG. 3, the eye-image generated by the infrared camera 123 is transmitted to the game machine 200 through the transmission means.

In the present embodiment, infrared rays are used to photograph the user's eyes, in order to minimize the deterioration of visibility of the boundary between the pupil and the iris according to the melanin pigment of the iris muscle that controls the enlargement and reduction of the pupil. In addition, since infrared rays are not sensed by the eyes of the user, the infrared rays are also suitable for measuring the sensitivity and fatigue of the user to be described later.

In addition, the above-described infrared cameras 113 and 123 may be implemented by removing an infrared cut filter from an existing camera and attaching an infrared transmissive filter to block visible light.

Hereinafter, the game machine 200 will be described in detail with reference to FIG. 4. 4 is a detailed block diagram of the game machine shown in FIG.

As shown in FIG. 4, the game machine 200 includes a receiver 210, an eye-state measuring unit 220, a user-state determining unit 230, a game-controlling unit 240, a video generating unit 250, The video output unit 260, the audio generator 270, and the audio output unit 280 are provided.

The receiver 210 receives an eye-image from the eye-image capturing apparatus 110 and transmits the eye-image to the eye-state measuring unit 220.

The eye-state measuring unit 220 measures the eye-state of the game user by using the eye-image received from the receiving unit 210. Specifically, the eye-state measuring unit 220 measures an eye state, 1) pupil size, 2) pupil adaptation speed, 3) eye blink frequency, and 4) eye blink rate, which will be described in more detail below. .

1) Pupil size measurement

The eye-state measuring unit 220 first extracts the pupil from the eye-image for specifying the pupil size, and a pupil extraction process is illustrated in FIG. 5.

In order to detect the position of the pupil in the eye-image, the eye-state measuring unit 220 performs a circular detection algorithm under the assumption that the pupil is circular. The circular detection algorithm detects a circle shape for the entire eye-image while changing the size of a template composed of two circles.

The eye-state measuring unit 220 determines the location of the pupil having the largest difference in brightness between the inner circle and the outer circle of the template, which is shown in FIG.

On the other hand, the circular detection algorithm is not accurate when the shape of the pupil is an ellipse or crushed shape. Thus, as shown in (b) of FIG. 5, the eye-state measuring unit 220 sets an area around the area detected through the circular detection algorithm and performs binarization on the set area.

At this time, since the region can be classified into two categories, a dark pupil region and a relatively bright iris region, the threshold for binarization is possible through the automatic threshold determination method proposed by Gonzalez or Otzu.

When performing binarization, an area other than the pupil may be determined as a black pixel by an eyebrow or a shadow. In order to remove the noise element, the state measuring unit 220 labels the binarization result, leaves only the largest continuous area, and then rests. By removing, the result as shown in (c) of FIG. 5 can be obtained.

However, since the result is a hole in the pupil due to the reflected light of the illumination, the eye-state measuring unit 220 is a 'region filling' through a morphological operation such as 'closing (closing)' (region filling) 'Process, it is possible to extract the correct pupil area as shown in (d) of FIG.

In addition, the eye-state measuring unit 220 may measure the size of the pupil area by counting the number of black pixels in the extracted pupil area.

2) pupil adaptation speed measurement

The eye-state measuring unit 220 measures the pupil adaptation speed per unit time by substituting the pupil area area extracted in real time into Equation 1 below.

In Equation 1, p is the pupil adaptation speed, Sn is the pupil area area inferred in the n-th eye-image, r is the eye-image acquisition speed per unit time.

3) Frequency of blinking eyes

The pupil area is extracted when the user opens his eyes, but when the user closes his eyes, the pupil area having a relatively dark brightness is not extracted because it is covered by the bright eyelid skin area.

That is, when the user closes the eyes, the number of black pixels as a result of binarization of the set area is remarkably small. Therefore, in this case, the eye-state measuring unit 220 determines that the game user is closing his eyes.

In addition, the eye-state measuring unit 220 determines that the game user blinks once when it is determined that the game user closes the eyes and opens the eyes again.

In this way, the eye-state measuring unit 220 divides the number of eye blinks by the observation time to measure the frequency of eye blinks per unit time.

4) blink rate measurement

The eye-state measuring unit 220 divides the number of eye-images determined to close the eyes for a unit time by the total number of eye-images acquired in unit time, and measures the speed of eye blinking.

This is a method using the fact that the pupil blinks for a long time when the blinking speed is slow.

So far, the eye-state measuring unit 220 has been described in detail. Hereinafter, referring again to FIG. 4, the user-state determination unit 230 will be described again.

The user-state determination unit 230 determines the state of the game user based on the eye-state measured by the eye-state measurement unit 220. In detail, the user-state determination unit 230 determines 1) the sensitivity and 2) the fatigue degree as the user-state, which will be described in more detail below.

1) Emotional judgment

The user-state determination unit 230 determines the emotion of the game user with reference to the pupil size and the blink rate.

In general, it is known that a person decreases the size of the pupil when he sees something disgusting, and that the size of the pupil increases when he sees a subject with a crush. Therefore, the user-state determination unit 230 may determine the opposing emotions of 'denial' and 'crush' through the size of the pupil.

Specifically, the user-state determination unit 230 determines that i) the game user feels rejection when the pupil size decreases, and ii) determines that the game user feels that the pupil size increases.

On the other hand, when you feel sleep, the time for closing eyes increases, and the speed of eye blinking is significantly slowed down. Accordingly, the user-state determination unit 230 may determine the opposite emotions of 'bored' and 'interesting' through the size of the pupil.

Specifically, the user-state determination unit 230, i) determine that the speed of the eye blink slow sleep, that is, the game user feels bored, ii) the speed of the eye blink decreases the sleep bath, In other words, it is determined that the game user is interested.

2) Determination of fatigue

The user-state determination unit 230 determines the degree of fatigue of the user with reference to the blinking frequency and the pupil adaptation speed.

Frequent blinking of eye and slowing of pupil adaptation are cases of a large amount of fatigue. Therefore, the user-state determination unit 230 determines that visual fatigue increases as the frequency of blinking increases. The threshold for fatigue determination using the blinking frequency can be set to blink once every 5 seconds for the general public, and can be set to a value close to this or through a calibration process before the game starts.

On the other hand, when a person's exercise muscles become tired, the speed of contraction and relaxation slows down, and the same principle applies as the irises that govern the size of the pupil are also muscles. Therefore, the user-state determination unit 230 determines that the fatigue degree is increased when the pupil adaptation speed becomes slow.

The threshold of fatigue determination using pupil adaptation speed can also be set based on existing researches, and can be set through the calibration process before the game starts.

On the other hand, the game-control unit 240 controls the game content based on the emotion and fatigue of the game user in the user-state determined by the user-state determination unit 230. In this process, the game-control unit 240 adjusts the game configuration and the screen state.

Specifically, when it is determined by the user-state determination unit 230 that the game user feels rejection, the game-control unit 240 controls to block or mosaic the contents of the game content.

In addition, when it is determined that the user of the game is bored by the user-state determination unit 230, the game-control unit 240 may cause an interest such as changing the type of game content or changing the configuration of the screen. Controls to stretch.

And, if it is determined that the user of the game is tired by the user-state determination unit 230, the game-control unit 240 changes the image quality parameters of the game screen, such as brightness, contrast, hue, saturation, so as to reduce the user fatigue To do that.

In addition, if it is determined that the degree of fatigue of the game user is severe, the game-control unit 240 may output a message recommending a break to the game user.

The video generator 250 generates a video in which the game content is displayed according to the control content of the game-control unit 240. The video generator 250 outputs the generated video to the TV 300 through the video output unit 260. Accordingly, the video generated by the video generator 250 is displayed on the TV 300.

Meanwhile, the audio generator 270 generates audio of game content according to the control content of the game-control unit 240 and outputs the generated audio to the TV 300 through the audio output unit 280. Accordingly, the audio generated by the audio generator 270 is output to the TV 300.

Hereinafter, a process of playing a game in the game-system described so far will be described in detail with reference to FIG. 6. 6 is a view provided to explain a game providing method according to an embodiment of the present invention.

The eye-image capturing apparatus 100 obtains an eye-image by capturing the eyes of a game user (S310). The eye-image photographing apparatus 100 transmits the acquired eye-image to the game machine 200 (S320).

The eye-state measuring unit 220 of the game machine 200 measures the pupil size using the eye-image received in step S320 (S330), and measures the pupil adaptation speed (S340).

In addition, the eye-state measuring unit 220 measures the eye blink frequency (S350) by using the eye-image received in step S320, and measures the eye blink speed (S360).

Thereafter, the user-state determination unit 230 determines the emotion of the game user with reference to the pupil size measured in step S330 and the blinking speed measured in step S360 (S370).

The user-state determination unit 230 determines the fatigue level of the game user with reference to the pupil adaptation speed measured in step S340 and the blinking frequency measured in step S350 (S380).

Then, the game-control unit 240 controls the game content based on the emotion of the game determined in step S370 and the fatigue degree of the game user determined in step S380 (S390).

Thereafter, the video generator 250 and the audio generator 270 generate video and audio of the game content according to the control content of the game controller 240 and transmit the generated video and audio to the TV 300 (S400).

On the other hand, in the present embodiment, the technique relating to the sensitivity and fatigue measurement can be used not only for games but also for evaluation of display image quality.

In addition, although the preferred embodiment of the present invention has been shown and described above, the present invention is not limited to the specific embodiments described above, but the technical field to which the invention belongs without departing from the spirit of the invention claimed in the claims. Of course, various modifications can be made by those skilled in the art, and these modifications should not be individually understood from the technical spirit or the prospect of the present invention.

1 shows a game-system to which the present invention is applicable;

2 is an example of an eye-image capturing apparatus, illustrating an HMD attached with an infrared light and an infrared camera for eye-image capturing;

3 is a view showing another example of an eye-image capturing apparatus, stereo glasses attached with an infrared light and an infrared camera for eye-image capturing;

4 is a detailed block diagram of the game machine shown in FIG. 1;

5 is a view illustrating a pupil extraction process, and

6 is a view provided to explain a game providing method according to an embodiment of the present invention.

Claims (16)

A receiver for receiving an eye-image of the game user's eyes from an external photographing apparatus; A measuring unit measuring an eye state based on the eye-image received by the receiving unit; A determination unit determining a user-state based on the eye-state measured by the measurement unit; And And a controller for controlling game contents based on the user-state determined by the determination unit. The method of claim 1, The eye state is And at least one of pupil size, pupil adaptation speed, blink frequency, and blink rate. 3. The method of claim 2, The user-state is Game machine comprising at least one of emotion and fatigue. The method of claim 3, The determination unit, The game machine, characterized in that the emotion of the game user is determined by referring to the pupil size and the eye blinking speed, and the fatigue degree of the game user is determined by referring to the eye blinking frequency and the pupil adaptation speed. The method of claim 4, wherein The determination unit, And a threshold value for the pupil size, the eye blink rate, the eye blink frequency, and the pupil adaptation in the pupil size through the calibration process before starting the game. The method of claim 1, The control unit, And adjust at least one of a configuration and a screen state of game content based on the user-state. Receiving an eye-image of the game user's eyes from an external photographing apparatus; Measuring an eye state based on the eye image received in the receiving step; Determining a user-state based on the eye-state measured in the measuring step; And And controlling the game content based on the user-state determined in the determining step. The method of claim 7, wherein The eye state is And at least one of pupil size, pupil adaptation speed, eye blink frequency, and eye blink speed. The method of claim 8, The user-state is Game providing method comprising at least one of emotion and fatigue. The method of claim 9, The determining step, And determining the emotion of the game user by referring to the pupil size and the blink rate of the eye, and determining the fatigue degree of the game user by referring to the eye blink frequency and the pupil adaptation speed. The method of claim 10, The determining step, And a threshold value for the pupil size, the eye blink rate, the eye blink frequency, and the pupil adaptation in the pupil size through the calibration process before starting the game. The method of claim 7, wherein The control step, And adjusting at least one of a configuration and a screen state of game content based on the user-state. Infrared illumination emitting infrared light into the eyes of the game user; An infrared camera for photographing the eyes of a game user whose infrared light is emitted by the infrared light to generate an eye-image; And And a transmission unit for transmitting the eye-image generated by the infrared camera to an external game machine. The method of claim 13, The infrared light and the infrared camera, Eye-image recording apparatus, characterized in that installed in the eye-image capturing device to face the eye of the game user. The method of claim 13, The eye-imaging device, Eye-imaging device, characterized in that implemented in at least one form of HMD, stereo glasses, goggles and helmet. Emitting infrared light into the eyes of the game user; Photographing the eyes of the game user from which the infrared rays are emitted to generate an eye-image; And And transmitting the eye-image to an external game machine.

Images