KR102556886B1 - Apparatus for controlling mood lighting in accordance with game and method therefor - Google Patents

Abstract

A game animation type lighting mood control device and method are disclosed. When an image is input, the controller generates an extended image of the game screen using the first artificial intelligence model that outputs an extended image for the image, extracts at least one color from the extended image of the game screen, and outputs the extracted color. A control command is output to the lighting device to do so.

Description

Game animation type lighting mood control device and its method {Apparatus for controlling mood lighting in accordance with game and method therefor}

An embodiment of the present invention relates to a method and apparatus for generating and controlling a lighting mood suitable for a game.

Games are developing into a sports (ie, e-sports) industry that aims to compete in front of a large audience beyond simple personal entertainment. Since the game is played in the limited space of the monitor screen, there is a limit to decorating the stage environment for gamers only with the game screen. Accordingly, in general, e-sports implement various directing effects in the gamer's stage environment in order to provide a more splendid appearance to the audience or viewers. However, the stage directing effect has nothing to do with the game screen, and in some cases, it may reduce the immersion in the game.

A technical problem to be achieved by embodiments of the present invention is to provide a method and apparatus for controlling the mood of ambient lighting to match the game so as to improve immersion in the game.

An example of a game animation type lighting mood control method according to an embodiment of the present invention for achieving the above technical problem is a game using a first artificial intelligence model that outputs an extended image for the image when an image is input. generating an extended image of the screen; extracting at least one color from the extended image of the game screen; and outputting a control command to the lighting device to output the extracted color.

An example of a game animation type lighting mood control device according to an embodiment of the present invention for achieving the above technical problem is a game using a first artificial intelligence model that outputs an extended image for the image when an image is input. An extended image generating unit for generating an extended image of the screen; a color extraction unit extracting at least one color from the extended image of the game screen; and a control unit outputting a control command to the lighting device to output the extracted color.

According to an embodiment of the present invention, the degree of immersion in the game can be improved by controlling the ambient light according to the game screen. At large events such as e-sports, you can create a stage environment suitable for games. Game developers can contribute to the development of the furniture industry through various types of furniture (for example, gaming chairs, etc.) capable of developing games linked to the game's surrounding environment and directing lighting related to games.

1 is a diagram schematically showing an overall system for controlling a game fairy tale lighting mood according to an embodiment of the present invention;
2 is a diagram showing an example of a method for determining an output color of a lighting device according to an embodiment of the present invention;
3 is a diagram showing a learning process of an artificial intelligence model used for generating an extended image according to an embodiment of the present invention;
4 is a flowchart showing an example of a method for controlling a game fairy tale lighting mood according to an embodiment of the present invention;
5A is a diagram showing an example of a method of extracting emotions from game music according to an embodiment of the present invention;
5B is a diagram showing an example of a method of extracting game music using a window to identify emotions according to an embodiment of the present invention;
6 is a diagram showing an example of a mapping relationship between emotions and movement patterns according to an embodiment of the present invention;
7 is a flowchart illustrating another example of a method for controlling a game fairy tale lighting mood according to an embodiment of the present invention;
8 is a diagram showing an example of a method for controlling the strength and weakness of a movement pattern according to an embodiment of the present invention;
9 is a diagram showing an example of a method of mapping and managing a color table with an index according to an embodiment of the present invention; and
10 is a diagram showing the configuration of an example of a control device according to an embodiment of the present invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, an apparatus and method for controlling a lighting mood for a game animation according to an embodiment of the present invention will be described in detail with reference to the accompanying drawings.

FIG. 1 is a diagram schematically illustrating an overall system for controlling a lighting mood of a fairy tale game according to an embodiment of the present invention.

Referring to FIG. 1, the system for controlling the lighting mood of the game's fairy tale collects various game information such as the game screen or game music displayed on the display device 100 and controls the lighting device 122 accordingly ( 110).

As an example, in the case of a general computer-based game, the controller 110 may be implemented in a general computer, and in the case of a mobile game, the controller 110 may be implemented in a mobile device such as a smartphone on which the mobile game operates. . As another example, the control device 110 may be implemented as a separate computing device connected to a computer or mobile device in which a game is operated by wired or wireless communication.

The lighting device 1220 may be located on various types of furniture 120 located around a place where a game is played. For example, the lighting device 122 of the present embodiment is applied to a chair 120 on which a professional gamer sits at an e-sports event, a chair 120 used by a general individual at home, or a chair 120 used in an PC room. can exist In addition to this, the lighting device 122 of this embodiment may be located on one side of the stage to direct the stage of an e-sports event, or may be located in various furniture (desks, bookshelves, wardrobes, etc.) in houses or PC rooms. . As another example, the above salpin control device 110 may be implemented integrally with the lighting device 122 or may be located on one side of the furniture 120 or the stage where the lighting device 122 is located.

The lighting device 122 is a device that outputs at least one color. For example, the lighting device 122 may be composed of light emitting diodes (LEDs) outputting a plurality of different colors or may be composed of lasers outputting a plurality of different colors. Alternatively, the lighting device 122 may be implemented as a device that outputs colors or images to a certain area, such as a projector. In addition to this, various conventional devices that output light may be used as the lighting device 122 of this embodiment.

For ease of understanding, this embodiment shows an example in which the lighting device 122 located on the back of the chair 120 outputs light to a certain area 124 on the back of the chair. This chair 120 can be used for stage directing in e-sports and the like. As another example, the lighting device 122 of the chair 120 may be disposed to output light to the front, side, or back of the chair 120 . For example, game immersion can be improved by having the lighting device 122 of the chair 120 output game-related mood lighting to a wall in front of the chair.

Since the location of the lighting device 122 or the direction or area in which the lighting device 122 outputs light can be variously modified according to embodiments, hereinafter, for convenience of description, a chair as shown in the embodiment of FIG. 1 A case in which the lighting device 122 is present in 120 will be described.

2 is a diagram illustrating an example of a method for determining an output color of a lighting device according to an embodiment of the present invention.

Referring to FIG. 2, the control device uses an extended image for extracting at least one representative color associated with the game screen 200 using the first artificial intelligence model 210 that outputs an extended image for this when an image is input ( 220) can be created. For example, the controller 110 captures the game screen 200 currently in progress at regular intervals (eg, in units of several seconds) and inputs the captured image to the first artificial intelligence model 210 to display the game screen 200 An extended image 220 located on the outer edge of can be obtained. The controller 110 may extract at least one color to be output through the lighting device 122 based on the extended image 220 located on the edge of the game screen 200 .

The extended image 220 of this embodiment is in the form of a square frame located outside the game screen 200, but this is only one example. It may be variously modified depending on the example. However, for convenience of explanation, the extended image 220 will be described assuming a rectangular frame shape as shown in FIG. 2 .

The extended image 220 is not simply an enlarged game screen, but an image composed of various colors related to the game screen, and in order for the first artificial intelligence model 210 to output such an extended image, a learning process must be preceded. . The learning process of the first artificial intelligence model 210 will be described in FIG. 3 .

3 is a diagram illustrating a learning process of an artificial intelligence model used to generate an extended image according to an embodiment of the present invention.

Referring to FIG. 3 , the first artificial intelligence model 210 may be learned through a learning network such as a convolutional neural network (CNN), a recurrent neural network (RNN), or a generative adversarial network (GAN). In addition to this, various conventional learning networks can be applied to this embodiment and are not limited to a specific learning method.

The learning data 300 of the first artificial intelligence model 210 may be generated by capturing the game screen 310 . For example, after capturing various types of game screens 310 , each game screen 310 may be divided into an inner area 312 and an edge area 314 . A size ratio between the inner region 312 and the edge region 314 may vary according to embodiments. The first artificial intelligence model 210 outputs a predicted border area 320 when receiving learning data 300 including a game screen 310 labeled as an inner area 312 and an edge area 314. The first artificial intelligence model 210 performs a learning process of adjusting the parameters of the learning network by comparing the predicted border area 320 and the border area of the training data (ie, ground truth) 314.

4 is a flowchart illustrating an example of a method for controlling a lighting mood of a game animation type according to an embodiment of the present invention.

Referring to FIG. 4 , the control device 110 generates an extended image for the game screen using the first artificial intelligence model that has been learned (S400). An example of a method of generating an extended image for a game screen is shown in FIG. 2 . The control device 110 may capture a game screen at a predetermined period and generate an extended image for the corresponding game screen.

The control device 110 extracts at least one color from the extended image (S410). For example, the control device 110 may extract a certain number of colors as representative colors in order of maximum frequency after recognizing frequencies of various colors existing in the entire extended image or in a predefined specific region. As another example, the control device 110 may extract the entire extended image or an area image of a certain size as it is. For example, the control device 110 may select all or part of the extended image in the form of a square frame and output it through the lighting device. In other words, when the background image of the game screen is in the forest, the control device may output all or part of the extended image as it is through the lighting device to make the game space look like it is in the forest. Therefore, hereinafter, at least one color extracted by the controller 110 refers to data for at least one color (ie, RGB values or CMYK values, etc.) or data for an image of a certain size composed of a plurality of pixels. can be interpreted as pointing to

The controller 110 generates a control command for the lighting device based on the extracted color (S420). For example, if green is extracted as a representative color from the extended image, the control device may generate a control command for green output and transmit it to the lighting device. As another example, when the control device 110 extracts a plurality of colors from the extended image, the control device outputs a plurality of colors according to a certain order or outputs a control command to arrange and output the plurality of colors in a pattern or pattern. It can be created and transmitted to the lighting device. As another example, when an image of a certain size is extracted from an extended image, the control device 110 may generate a control command for output of the corresponding image and transmit it to the lighting device. In addition to this, the control device 110 may generate a control command for outputting the extracted color in various forms or shapes. An example of a method of controlling the controller 110 to output a color according to a movement pattern based on game music will be described in FIGS. 5 to 8 .

5A is a diagram illustrating an example of a method of extracting emotions from game music according to an embodiment of the present invention.

Referring to FIG. 5A , the second artificial intelligence model 510 is an artificial intelligence model that outputs emotions about music. The second artificial intelligence model 510 may be implemented with various conventional learning networks such as CNN, RNN, and RAN.

The learning data 500 of the second artificial intelligence model 510 may be generated using game music. For example, learning data 500 may be generated by dividing various game music into predetermined interval units (eg, several seconds) and then labeling emotion information to the music in each interval unit. As another embodiment, learning data 500 may be generated by defining a window of a certain size rather than by a certain section, and labeling emotion information about music of a certain section belonging to the window while moving the window. An example in case of using a window is shown in FIG. 5B.

Emotions may be classified into various numbers according to embodiments. For example, emotions may be defined in various ways according to embodiments such as cheer, depression, happiness, comfort, and tension, and emotion information may be mapped to each section of game music included in the learning data 500 .

When the second artificial intelligence model 510 receives the learning data 500, it outputs a predicted emotion 520. In other words, the second artificial intelligence model 510 predicts the emotion 520 for music for each given section (or window), compares it with the emotion of the learning data 500, and adjusts the parameters of the network. carry out the learning process;

When the learning of the second artificial intelligence model 510 is completed, the control device 110 inputs the game music output during the real-time game to the second artificial intelligence model 510 to grasp the real-time emotion of the game music. can

FIG. 5B is a diagram illustrating an example of a method of extracting game music using a window in order to identify emotions according to an embodiment of the present invention.

Referring to FIG. 5B , the window 560 has a certain size t. For example, the size of the window 560 may be 10 second intervals. The size of the window 560 may be variously changed according to embodiments. Window 560 moves over time. The controller 110 extracts a section of the game music 550 where the window 560 is located and inputs it to the second artificial intelligence model 510 to predict emotion. Then, the control device 110 moves the window 560 to the right at a certain interval (eg, 1 second interval, etc.), extracts a section of the game music 550 located in the moved window 570, and extracts it. Emotion is predicted by inputting it to the second artificial intelligence model 510 again.

6 is a diagram illustrating an example of a mapping relationship between emotions and movement patterns according to an embodiment of the present invention.

Referring to FIG. 6 , the control device 110 may register and store movement patterns for each emotion in advance. For example, the movement pattern may be in various forms such as a form in which one or more squares move, a form in which one or more rectangles move, and a wave form. In this embodiment, the motion pattern is defined as a concept including a fixed pattern (eg, a wave pattern or a square pattern) rather than a pattern necessarily accompanied by motion.

The movement pattern can be created using the extended image shown in FIG. 2 . In other words, it is possible to capture the game screen at regular intervals and make the change in the generated extended image into a movement pattern. For example, the control device 110 may generate a movement pattern as a change between images appearing in the entire extended image or in a predetermined area. The control device 110 may register the mapping relationship between the thus created movement pattern and emotion. As another example, when generating a control command for the lighting device in FIG. 4 , the control device 110 may generate a control command for outputting color using a movement pattern recognized through a change in an extended image.

7 is a flowchart illustrating another example of a method for controlling a lighting mood of a game animation type according to an embodiment of the present invention.

The example of FIG. 4 relates to a method of controlling a lighting device based on a game screen, whereas the example of FIG. 7 relates to a method of controlling a lighting device by simultaneously utilizing a game screen and game music. This embodiment assumes that at least one color is extracted by the method of FIG. 4 .

Referring to FIG. 7 , the control device 110 determines the emotion of the game music using the second artificial intelligence model that has been learned (S700). For example, the control device 110 may input real-time game music to the second artificial intelligence model that has been learned through the learning process of FIG. 5 to determine emotion.

The control device 110 identifies a movement pattern corresponding to the identified emotion (S710). As shown in FIG. 6 , a mapping relationship between emotion and movement pattern is set in advance, and the control device 110 can grasp the movement pattern for emotion through this mapping relationship.

The controller 110 generates a control command for outputting at least one color determined based on the game screen according to a movement pattern, and transmits the generated control command to the lighting device (S720). For example, if the green color is extracted from the extended image of the game screen through the method of FIG. 4 and the square-shaped movement pattern is identified from the emotions for the game music, the control device outputs green as a square-shaped movement pattern. A control command is generated and transmitted to the lighting device.

8 is a diagram illustrating an example of a method for controlling strength and weakness of a movement pattern according to an embodiment of the present invention.

Referring to FIG. 8 , the controller 110 can determine the height or strength of game music (S800). For example, in the case of detecting emotions from game music in 10-second increments through the second artificial intelligence model, the control device may determine the height or intensity of the game music in 10-second increments, which is the same interval. Various conventional methods for determining the pitch or intensity of music may be applied to this embodiment.

The controller 110 may determine the intensity or speed of the movement pattern based on the pitch or intensity of the game music (S810). For example, after defining a plurality of sections for the pitch of the music and the strength and weakness of the movement pattern for each section (eg, the intensity of the movement pattern or the thickness of the shape or line constituting the pattern) in advance, the controller controls the game music. The strength and weakness of the movement pattern according to the height can be determined. Alternatively, as the intensity of the game music increases, the change speed of the movement pattern may be increased.

The control device 110 may generate a control command including strength or speed of the movement pattern identified through the process of FIG. 7 and transmit the control command to the lighting device (S820).

9 is a diagram illustrating an example of a method of mapping and managing a color table with an index according to an embodiment of the present invention.

Referring to FIG. 9 , mapping relationships between various colors and indices are predefined and stored in the control device 110 and the lighting device 122, respectively. When the control device 110 transmits a control command including information on a small number of colors and movement patterns thereof to the lighting device 122, the size of data is not large, so information on colors (eg, RGB data) etc.) can be transmitted as is. However, when a control command including information on tens or hundreds of colors is transmitted, since the data capacity is large, index information can be used to reduce it. For example, the index information may consist of numbers, and when the control device 110 transmits index information '2' for the second color, the lighting device 122 displays '2' based on a pre-stored color table. It can be seen that the color for is the second color.

10 is a diagram showing the configuration of an example of a control device according to an embodiment of the present invention.

Referring to FIG. 10, the control device 110 includes a first learning unit 1000, an extended image generating unit 1010, a color extraction unit 1020, a first movement pattern extraction unit 1025, and a second learning unit ( 1030), an emotion detection unit 1040, a second movement pattern extraction unit 1050, and a control unit 1060. Each component of the control device can be software implemented. For example, the controller may include a memory for loading each configuration implemented in software and a processor for executing each configuration stored in the memory. The control device 110 may be implemented by omitting some of the components. For example, the control device 110 may include only one of the first movement pattern extraction unit 1025 and the second movement pattern extraction unit 1050 .

The first learning unit 1000 trains a first artificial intelligence model that generates an extended image for a game screen. An example of the learning process of the first artificial intelligence model is shown in FIG. 3 . As an embodiment, the control device 110 may not include the first learning unit 1000 but may include a first artificial intelligence model that has been learned.

The extended image generator 1010 generates an extended image for the game screen using the first artificial intelligence model that has been learned. Referring to FIG. 2 , the extended image generating unit 1010 may generate an extended image for the game screen by capturing a game screen while a real-time game is in progress and then inputting the captured image to the first artificial intelligence model.

The color extraction unit 1020 extracts at least one color from the extended image. The color extraction unit 1020 may extract various colors or images related to the game screen in various ways, and details thereof are illustrated in FIG. 4 .

The first motion pattern extraction unit extracts the motion pattern of the extended image. For example, the first motion pattern extraction unit may extract a color, pattern, or pattern change of the entire extended image or a predetermined area.

The second learning unit 1030 trains a second artificial intelligence model for extracting emotions from game music. An example of the learning process of the second artificial intelligence model is shown in FIG. 5 . As an embodiment, the control device 110 may not include the second learning unit 1030 and may include a second artificial intelligence model that has been learned.

The emotion detection unit 1040 uses the second artificial intelligence model that has been learned to determine the emotion of the game music. An emotion detection period of the emotion detection unit 1040 and an extended image generation period of the extended image generator 1010 may be synchronized with each other.

The second movement pattern extraction unit 1050 identifies a movement pattern corresponding to the emotion identified by the emotion detection unit 1040 . A mapping relationship between emotions and movement patterns may be preset as shown in FIG. 6 .

The control unit 1060 controls to output at least one color extracted by the color extraction unit 1020 through the movement pattern recognized by the first movement pattern extraction unit 1025 or the second movement pattern extraction unit 1050. Generates a command and sends it to the lighting device. As another example, as shown in FIG. 8 , the control unit 1060 may determine the intensity or speed of the movement pattern based on the pitch or intensity of the game music, and then generate a control command reflecting the determination and transmit it to the lighting device.

The present invention can also be implemented as computer readable codes on a computer readable recording medium. A computer-readable recording medium includes all types of recording devices in which data that can be read by a computer system is stored. Examples of computer-readable recording media include ROM, RAM, CD-ROM, magnetic tape, floppy disk, and optical data storage devices. In addition, the computer-readable recording medium may be distributed to computer systems connected through a network to store and execute computer-readable codes in a distributed manner.

So far, the present invention has been looked at with respect to its preferred embodiments. Those skilled in the art to which the present invention pertains will be able to understand that the present invention can be implemented in a modified form without departing from the essential characteristics of the present invention. Therefore, the disclosed embodiments should be considered from an illustrative rather than a limiting point of view. The scope of the present invention is shown in the claims rather than the foregoing description, and all differences within the equivalent scope will be construed as being included in the present invention.

Claims (14)

Generating an extended image of the game screen using a first artificial intelligence model that outputs an extended image for the image when an image is input;
extracting at least one color from the extended image of the game screen;
recognizing emotions about game music by using a second artificial intelligence model that outputs emotions about the music when music is input; and
extracting a movement pattern corresponding to the emotion of the game music according to a predefined mapping relationship between emotion and movement pattern; and
and outputting a control command to a lighting device to output the extracted color according to the motion pattern. delete The method of claim 1, wherein generating the control command comprises:
determining the intensity or speed of the movement pattern according to the pitch or intensity of the game music; and
Generating a control command including strength or speed of the movement pattern with the extracted color; The method of claim 1, wherein the step of extracting the color comprises:
Extracting at least one color based on color frequency from the extended image of the game screen, calculating a representative color based on at least one color located in a predefined area in the extended image of the game screen, or A method for controlling game dynamic lighting mood, comprising the step of extracting some or all of the colors of an extended image. According to claim 1,
Further comprising: grasping a movement pattern of color for a part or the whole of the extended image of the game screen;
The step of outputting the control command includes generating a control command for outputting the extracted color according to the motion pattern. According to claim 1,
Capturing a plurality of game screens;
Separating each game screen into an inner area and a border area; and
and learning the first artificial intelligence model for generating an extended image for a basic image using learning data composed of the inner region and the border region. According to claim 1,
Capturing a plurality of game music;
Classifying the game music into pre-defined sections and receiving an emotion for each section; and
Learning the second artificial intelligence model using learning data including game music and emotions in units of intervals; game fairy tale lighting mood control method further comprising. The method of claim 1, wherein outputting the control command comprises:
Generating a control command including an index number for the extracted color; a game fairy tale lighting mood control method comprising a. According to claim 1,
The lighting device is a game fairy tale lighting mood control method, characterized in that located on one side of the gaming chair. An extended image generator for generating an extended image of a game screen using a first artificial intelligence model that outputs an extended image for the image when an image is input;
a color extraction unit extracting at least one color from the extended image of the game screen;
an emotion grasping unit that grasps emotions about game music using a second artificial intelligence model that outputs emotions about the music when music is input; and
a motion pattern extraction unit extracting a motion pattern corresponding to the emotion of the game music according to a predefined mapping relationship between emotion and motion pattern; and
and a control unit for outputting a control command to the lighting device so as to output the extracted color according to the motion pattern. delete According to claim 10,
The first artificial intelligence model that captures a plurality of game screens, divides each game screen into an inner area and a border area, and generates an extended image for the basic video using learning data composed of the inner area and the border area. Game animation type lighting mood control device further comprising a first learning unit for learning. According to claim 10,
After capturing a plurality of game music, classifying the game music into predefined sections and receiving emotions for each section, using learning data including game music and emotions in section units, the second artificial intelligence model A second learning unit for learning; a game animation type lighting mood control device further comprising a. delete

Images