KR20200008227A - System and method for controlling gesture based light dimming effect using natural user interface - Google Patents

Abstract

Provided is a system for controlling a gesture-based lighting effect using a natural user interface (NUI), comprising: at least one light source which is installed inside or outside a virtual reality (VR) booth for a realistic experience and irradiates light to correspond to a preset event in order to implement a special effect according to motion data including a motion or a gesture of a game subject, or a game progress situation; a dimmer controller which increases or decreases a percentage of light output of the at least one light source by using a digital addressable lighting interface (DALI) protocol and independently gives an address to each of the at least one light source to perform individual control; at least one infrared camera which performs photographing to sense motion data of the game subject; and a control device which, when motion data or a preset event occurs, controls the at least one light source through the dimmer controller based on motion data input from the at least one infrared camera.

Description

Lighting effect control system and method using gesture-based NUI {SYSTEM AND METHOD FOR CONTROLLING GESTURE BASED LIGHT DIMMING EFFECT USING NATURAL USER INTERFACE}

The present invention relates to a method of controlling a lighting effect using a gesture-based NUI, and provides a method of outputting a neon-mapped special effect mapped to an event generated in a gesture or game.

Recently, researches on virtual reality games, the growth engine of the 4th industrial revolution, and game systems using user gestures have received a lot of attention, and beyond the simple interface methods such as keyboards and mice, gesture recognition, face, which is an emotional recognition technology between machines and people Human interface related technologies using recognition and speech recognition technologies have been actively studied. Among them, gesture recognition using hands is the most commonly used interface in daily life. It has the advantage of fast communication through designation of objects or hand movements. It has the advantage of being highly interactive.

In this case, a method of mapping a gesture to an NUI in a game and recognizing the gesture is performed by mapping a gesture and EMG data to game data. In this regard, Korean Patent Laid-Open Publication No. 2017-0059300 (published May 30, 2017), which is a prior art, defines a set command as at least one gesture and stores it in a database, and inputs the user's motion and the EMG data of the user. Receives gestures that are mapped to user's motion, evaluates recognized gestures using input user's motion and EMG data, selects the most suitable gesture, and allows users to control the game comfortably while sitting It is.

However, in addition to recognizing or evaluating the gestures of the game in virtual reality, the VR booth installed in the arcade area attracts a variety of customers to watch the game video with the users who play the game. Unreal VR games are less interesting and marketing effects are reduced, and it is difficult to give realistic effects to users who play games.

An embodiment of the present invention, to control the on and off and output level of at least one light source to correspond to the event of the game occurring in the virtual reality or the hand motion, motion and gesture of the game subject, and separate equipment through the NUI mapping People who do not manipulate the human's natural movements are reflected in the game or operate as input values to output special effects, and use the DALI protocol to individually drive at least one light source and watch the game around the VR booth. For example, it is possible to provide a lighting effect control method using a gesture-based NUI, which can achieve an interest factor and marketing effect due to light emission of a light source. However, the technical problem to be achieved by the present embodiment is not limited to the technical problem as described above, and other technical problems may exist.

As a technical means for achieving the above-described technical problem, an embodiment of the present invention, is installed inside and outside the VR booth for a realistic experience to implement the special effects according to the motion data or game progress including the motion or gesture of the game subject To increase or decrease the light output of at least one light source using at least one light source for irradiating light to correspond to a predetermined event, the digital addressable lighting interface (DALI) protocol, and at least one light source independently A dimmer controller which gives an address and performs individual control, at least one infrared camera photographing to detect motion data of a game subject, and at least one through a dimmer based on motion data input from at least one infrared camera. A single light source with motion data It includes a control device for controlling when the set event occurs.

Another embodiment of the present invention is installed in the VR booth for a realistic experience taken from at least one infrared camera to receive movement data including the motion or gesture of the game subject, the movement data and pre-stored NUI mapping (Natural User) Interface mapping (DIM) comparing the data to determine whether to implement the special effect, when the result of the motion data that the effect is to be implemented, using the Digital Addressable Lighting Interface (DALI) protocol to independently control at least one light source Transmitting a control signal to a dimmer controller; and receiving feedback results of outputting a signal for controlling at least one light source from the dimmer as feedback data to perform PID (Proportional Integral Derivative) control.

According to any one of the problem solving means of the present invention described above, to control the on-off and output level of the at least one light source to correspond to the event of the game occurring in the virtual reality or the hand motion, motion and gesture of the game subject, NUI mapping It allows the human's natural movements to be reflected in the game or outputs special effects without any additional equipment, and operates at least one light source individually using the DALI protocol, and around the VR booth. In addition, people watching the game can enjoy interesting factors and marketing effects due to light emission.

1 is a view for explaining a lighting effect control system using a gesture-based NUI according to an embodiment of the present invention.
FIG. 2 is a block diagram illustrating a control device included in the system of FIG. 1.
3 is a view for explaining an embodiment of the lighting effect control system using a gesture-based NUI according to an embodiment of the present invention.
4 is a diagram illustrating an embodiment in which a lighting effect control system using a gesture-based NUI according to an embodiment of the present invention is applied to a VR booth.
5 is a flowchart illustrating a lighting effect control method using a gesture-based NUI according to an embodiment of the present invention.

DETAILED DESCRIPTION Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings so that those skilled in the art may easily implement the present invention. As those skilled in the art would realize, the described embodiments may be modified in various different ways, all without departing from the spirit or scope of the present invention. In the drawings, parts irrelevant to the description are omitted in order to clearly describe the present invention, and like reference numerals designate like parts throughout the specification.

Throughout the specification, when a part is "connected" to another part, it includes not only "directly connected" but also "electrically connected" with another element in between. . In addition, when a part is said to "include" a certain component, which means that it may further include other components, without excluding other components, unless specifically stated otherwise, one or more other features It is to be understood that the present disclosure does not exclude the possibility of adding or presenting numbers, steps, operations, components, parts, or combinations thereof.

As used throughout the specification, the terms "about", "substantially", and the like, are used at, or in close proximity to, numerical values when manufacturing and material tolerances inherent in the meanings indicated are intended to aid the understanding of the invention. Accurate or absolute figures are used to assist in the prevention of unfair use by unscrupulous infringers. As used throughout the specification of the present invention, the term "step of" or "step of" does not mean "step for".

In the present specification, the term 'unit' includes a unit realized by hardware, a unit realized by software, and a unit realized by both. In addition, one unit may be realized using two or more pieces of hardware, and two or more units may be realized by one piece of hardware.

Some of the operations or functions described as being performed by the terminal, the apparatus, or the device may be performed instead in the server connected to the terminal, the apparatus, or the device. Similarly, some of the operations or functions described as being performed by the server may be performed by the terminal, apparatus or device connected to the server.

In the present specification, some of the operations or functions described as mapping or matching with the terminal mean that the identification number of the terminal or identification information of the individual, which is identification data of the terminal, is mapped or matched. Can be interpreted as

Hereinafter, the present invention will be described in detail with reference to the accompanying drawings.

1 is a view for explaining a lighting effect control system using a gesture-based NUI according to an embodiment of the present invention. Referring to FIG. 1, the lighting effect control system 1 using a gesture-based NUI includes at least one infrared camera 100, a control device 300, a dimmer 400, and at least one light source 500. can do. However, since the lighting effect control system 1 using the gesture-based NUI of FIG. 1 is only an embodiment of the present invention, the present invention is not limitedly interpreted through FIG. 1.

In this case, each component of FIG. 1 is generally connected through a network 200. For example, as shown in FIG. 1, at least one infrared camera 100 may be connected to the control device 300 through the network 200. In addition, the control device 300 may be connected to the at least one infrared camera 100, the dimmer 400, and the at least one light source 500 through the network 200. In addition, the dimmer 400 may be connected to the control device 300 through the network 200. At least one light source 500 may be connected to the control device 300 and the dimmer 400 through the network 200.

Here, the network refers to a connection structure capable of exchanging information between each node, such as a plurality of terminals and servers, and examples of such a network include RF, 3rd Generation Partnership Project (3GPP) network, and Long Term (LTE). Evolution network, 5th Generation Partnership Project (5GPP) network, World Interoperability for Microwave Access (WIMAX) network, Internet, Local Area Network (LAN), Wireless Local Area Network (WLAN), Wide Area Network (WAN) , PAN (Personal Area Network), Bluetooth (Bluetooth) network, NFC network, satellite broadcasting network, analog broadcasting network, DMB (Digital Multimedia Broadcasting) network and the like, but is not limited thereto.

In the following description, the term “at least one” is defined as a singular and plural term, and each component may exist in the singular or plural, even though the term “at least one” does not exist, and may mean the singular or plural. It will be self explanatory. In addition, the singular or plural elements may be changed according to embodiments.

At least one infrared camera (100) is a charge coupled device (CCD) camera having a sufficient sensitivity to infrared light. When photographing wild animals and plants at night, it may be a camera that photographs a game object by attaching an infrared filter to a light with strong light condensation. The at least one infrared camera 100 may be installed inside or outside the VR booth for a realistic experience, and may be an integrated infrared camera using a near infrared light source using a laser and an image intensifier (II).

Infrared images use invisible infrared radiation emitted directly from the surface of the object, and the higher the outside temperature of the moving object, the higher the amount of radiant energy is emitted. It can be expressed in gray scale. Moving objects that emit more infrared radiation than their surroundings can be easier to find than moving color-based cameras. Accordingly, even a remote moving object can be tracked and the position of the moving object can be shown even in the absence of visible light such as night, fog and saturation. The method using the max-mean and max-median filters of the infrared image preserves the contours of the cloud and the surrounding environment, and the anti-mean and anti-median operations are blurred. It is possible to take advantage of the fact that it is useful for the detection of moving objects having different characteristics from the site. The image obtained by the maximum average and maximum intermediate filters may use the original frame and the difference image to detect moving objects, and the number of pixels whose threshold may be the moving object may be added. In addition, a method using a morphological filter may be used to process an infrared image, which uses a new detection technique for detecting and tracking a moving object represented by moving points in an infrared image frame. After preprocessing the original image using the -hat operator, the moving object can be detected by reducing the three-dimensional space-time detection technique to the two-dimensional space detection dimension, and the moving object has a constant false alarm rate detector. Can be. Alternatively, a temporal filter method using a continuous wavelet transform (CWT) may be used for temporal pixel profiles in an infrared image. In order to distinguish a moving object from a background clutter, a Mexican hat CWT is used to generate a constant luminance value. If a pixel is held for a certain period of time, it is recognized as a background and removed from the area that can be a moving object. Of course, it is not limited to the above-described method, it will be apparent that various methods can analyze the image obtained from the infrared camera 100 according to the embodiment.

The control device 300 receives an image photographing a game subject from at least one infrared camera 100 and controls the position and radius of motion data in the image in order to control lighting effects using a gesture-based NUI. The device may be configured to determine which special effects are to be output by grasping the events stored in the pre-stored NUI (Natural User Interface) mapping. Accordingly, when the special effect is extracted from the NUI mapping data, the control device 300 can control the on / off and illuminance of the at least one light source 500 by controlling the dimmer 400. It may be a device. At this time, since the method for distinguishing the infrared image by the control device 300 is as described above, a detailed description thereof will be omitted. Here, the control device 300 may be implemented as a computer that can be connected to a server or a terminal in a remote place through a network. Here, the computer may include, for example, a navigation, a laptop equipped with a web browser, a desktop, a laptop, and the like.

The dimmer 400 may be a device that individually turns on and off the power of the at least one light source 500 from the control device 300 by using a digital addressable lighting interface (DALI) protocol. At this time, DALI is a dimming control method of the open standard according to IEC 62386, and when a suitable protocol is used, it is a protocol that allows products of different companies to operate in conjunction with each other. It is basically operated by a digital communication protocol for transmission and reception, but a simple two-wire method may be used as a communication medium. In the case of lighting control, since the data does not need to process a large amount of data quickly, such as Ethernet, the data rate may be set to 1200 bits per second, but is not limited thereto. In order to reduce the effects of malfunction due to noise during transmission, the voltage corresponding to the high level is defined as 9.5 V to 22.5 V and the voltage recognized as the low level is specified as -6.5 V to 6.5 V. It is obvious that various embodiments are possible without being obvious. In addition, the DALI protocol can adjust the brightness of 246 levels and these control the light source in relation to the logarithmic function rather than linear values, i.e. show a greater increase in the amount of light at high dimming levels and less light at low dimming levels. You can allow an increase. The reason is that the human eye is more sensitive at a lower light level than a high light level, so that it can be recognized as a linear relationship from the human perspective.

In addition, the DALI protocol has the flexibility to program a variety of commands to suit the desired lighting task and the given situation, allowing the lighting to be changed without a separate wiring for changing workspaces. Through this, at least one light source 500 having a maximum of 64 addresses may be independently controlled through a DALI loop or a bus configured by the dimmer 400, and bidirectional communication may be performed. When the at least one light source 500 is combined and controlled in groups, the at least one light source 500 may be divided into 16 groups by the dimmer 400, and the dimmer 400 may also operate as a standalone operation. It can be connected to a network of VR booths and operated as a subsystem of bidirectional communication. Conductors that can be used as DALI buses can use ordinary conductors (twisted or shielded cables). Here, the final control of the at least one light source 500 may be performed by a power converter (not shown). In the case of LEDs, the driver may be referred to as a control gear. DALI control requires a dedicated driver that works with the DALI protocol, which can be used with microcontrollers in addition to existing drivers or ballasts. The DALI driver may receive and execute a control command for the light source 500 through the DALI bus, transfer the data related to the state of the light source 500 to the bus, and transmit the data to the dimmer 400. Each data can be communicated as a signal electrically isolated by a photocoupler or transformer for the electrical separation between the signal and power control systems, and the DALI driver has its own optical output, fade time, and accessible address. Parameters for and groups can be stored. The information received by the dimmer 400 from the driver is related to the current illumination state, the output level of the light source 500, and the state of the lamp and the ballast, through which the PID control may be possible in the control device 300. The dimming range of the at least one light source 500 may be set to 0.1% to 100% through the dimmer 400, and a minimum level value of dimming may vary depending on a product.

The at least one light source 500 may be a device that emits light by adjusting the on / off and output levels by the dimmer 400. At this time, the output level of the at least one light source 500 is transmitted to the dimmer 400 again as described above, so that the dimmer 400 can determine whether light is being irradiated according to a desired output level. In this case, the one light source 500 may be a flexible neon, an LED, an OLED, an LCD, or the like, but the present invention is not limited thereto and various configurations may be added according to various embodiments.

Referring to FIG. 2, the control device 300 may include a receiver 310, a determiner 320, a transmitter 330, and an implementer 340. Referring to FIG. 2, the receiver 310 may be installed in a VR booth for a realistic experience captured by at least one infrared camera 100 to receive motion data including a motion or a gesture of a game subject. At this time, the motion of the game subject during the motion or gesture is a figure drawing operation using a finger, left and right shaking, up and down touch, horizontal touch, fist, pinch, finger spread, palm touch, finger spreading, bending, and flipping the palm. It may be any one or at least one combination. In this case, the hand gesture may be connected and stored with data for controlling driving and output of the at least one light source 500 and NUI mapping. At this time, in addition to the at least one infrared camera 100 to detect the movement of the game subject at least one controller (Game Controller, not shown) located on at least one human body part of the game subject to detect the movement data of the game subject or It may further include an infrared sensor (not shown).

The determination unit 320 may determine whether to implement a special effect by comparing the motion data with previously stored NUI mapping data. In this case, in addition to NUI mapping based on NUI, user interface technology may use various approaches such as object-oriented specification using UML (Unified Modeling Language) and formal specification using Object-Z, Petri Net. have. In the NUI environment such as touch, gesture, and voice recognition, GDL (Gesture Definition Language), which is a domain specification language applicable to a multi-touch interface environment, may be used, and for flexible gesture recognition through a 3D camera, the Flexible Action and Articulated Skeleton Toolkit can be used. FAAST is a toolkit that can control gestures and VR applications and games by detecting gestures based on the user's skeleton model and converting them into mouse and keyboard events through Kinect sensors. It can support five types of gesture input formats such as position, angular, velocity, and time, and eight output formats such as keyboard, mouse, and time. Of course, it will be apparent that user interfaces of various NUI environments may be applied in addition to the above-described methods. In addition, the control of the special effects according to the gesture is disclosed in Patent No. 10-1794233 filed by the present applicant (announced on November 07, 2017), detailed description thereof will be omitted.

When the transmission unit 330 is the motion data to which the special effect is to be determined as a result of the determination, the dimmer controller 400 using a digital addressable lighting interface (DALI) protocol to independently control the at least one light source 500. The control signal can be transmitted. In this case, at least one infrared sensor (not shown) may be further included to collect three-axis data including X, Y, and Z of the game subject's motion data, and the control device 300 may include at least one infrared camera ( The protocol may be generated by analyzing photographing data photographed at 100) and three-axis data including X, Y, and Z of motion data of a game subject collected from at least one infrared sensor.

The implementation unit 340 may receive a result of outputting a signal for controlling the at least one light source 500 from the dimmer 400 as feedback data to perform PID (Proportional Integral Derivative) control. In this case, the control device 300 may detect and analyze the motion data of the game subject using the location information, the coordinate range, and the detection level of the game subject received from the at least one infrared camera 100. In this case, the at least one light source 500 may be formed of any one or a combination of flexible neon, LED, OLED, and LCD.

On the other hand, when the infrared camera 100 is configured as a stereo system, the control device 300 undergoes a stereo rectification process, and internal parameters such as focal length, pixel size, etc. of each infrared camera 100 are performed. ) And external parameters that define the spatial transformation state between the cameras and the cameras, and geometrically predict where the three-dimensional spatial coordinates projected on the stereo image are projected to each stereo image. The geometric correspondence between these stereo images is called epipolar geometry. The epipolar geometry is geometrically defining where one 3D spatial coordinate is projected on the left and right stereo images, which may vary depending on how the stereo cameras are spatially arranged. The most common camera layout is two cameras placed at the same distance and height from the object, looking parallel to each other vertically. This is called a parallel camera array. In the case of the parallel arrangement, since the position of the camera is shifted only in the left and right directions, the position of the projected image also differs only in the left and right horizontal directions. The epipolar geometry is therefore a horizontal line for each image. Stereo systems, unless otherwise noted, generally assume this parallel camera arrangement. When the arrangement between the stereo cameras is not parallel, each image is adjusted to have a parallel arrangement structure using camera parameters obtained through a camera correction process in advance. Since the adjusted stereo image is aligned on the x-axis when the corresponding point is estimated, only the scan-line having the same y coordinate needs to be searched. The control device 300 then performs stereo matching. That is, the above-described epipolar geometry defines a geometric relationship in which correspondences between left and right stereo images exist, and define the position of the stereo image in which the correspondence exists.

Stereo matching refers to a process of finding correspondence points of left and right stereo images based on the epipolar geometry. In a stereo system having a parallel camera arrangement, since the correspondence between left and right stereo images estimated through stereo matching is represented by the difference in pixel positions between corresponding points, stereo matching is also called disparity estimation. In order to convert depth information in parallax estimation, the actual size of 1 pixel may be measured. If you know the CCD size, focal length, angle of view and resolution of the camera image, you can calculate the size of 1 pixel using the trigonometric function. The geometrical interpretation of a parallel stereo camera is that the image plane is located in front of the focal point of the lens, which is called the central projection, and the straight line connecting the two lens centers is called the base line. The origin of the reference coordinate system (X, Y, Z) is located. For example, given the focal length f and the distance between the centers of the two lenses, i.e. the baseline distance b, the correlation between the reference coordinate system and the corresponding image coordinates (x, y) by the similar triangular properties You can get relations. Using the physical size or distance between pixels (mm), the distance between two cameras (reference distance, mm), and the parallax values for the object correspondence points estimated by stereo matching, the distance from the camera to the object is measured using a simple proportional equation. can do. Of course, when the plurality of infrared cameras 100 are not used, the above matching process may not be required.

In addition, when the smart glasses are provided, the control device 300 may use a hand gesture recognition technique based on a convolutional neural network (CNN) to detect a gesture of the hand. In MPEG IoMT, it is assumed that hand gesture detection and recognition are performed in a separate device, and an interface such as an API (Application Programming Interface) and a data format that can be interoperable between these modules is prepared. Therefore, the control device 300 according to an embodiment of the present invention is composed of an image processing-based hand contour detection part and a CNN-based hand gesture recognition part, and the hand gesture detection module obtains from an input stereo image of smart glasses. The contour of the hand can be detected as a binary image using depth information and color information, and the recognition module uses the gesture binary image obtained from the detection module as a data set for CNN learning and recognizes the hand gesture based on the learning result. The recognition result may be transmitted to the wearable device.

To this end, hand gesture detection, CNN-based gesture recognition, CNN optimization algorithm, and deep learning framework, which are executed in the control device 300, will be described below. First, the control device 300 acquires stereo video from the stereo camera or at least one infrared camera 100 of the smart glass for hand gesture detection, and finds the contour of the hand from it. In addition, the control apparatus 300 obtains a depth image through stereo matching on a difference between the left and right images (RGB image). Due to the characteristics of the VR booth, it is assumed that the user's hand is in a certain distance range (30 to 50 cm) from the camera, and the control device 300 detects the hand gesture area by applying a depth image and color information. Further morphology operations may be performed to obtain a more accurate area of the hand gesture and to remove noise.

Secondly, the control device 300 needs to perform CNN-based gesture recognition, and CNN is a kind of multilayer neural network, which shows very good results in the field of recognition, and CNN directly computes input source data unlike conventional neural networks. It is characterized by producing a result value through a feature extraction step and a classification step without outputting the data. The structure of CNN is composed of convolution layer, pooling layer, and fully-connected (FC) layer, and C1 and C2 are convolution layers and 5 × 5 image data input from the input layer. A convolution operation is performed at interval 1 (stride = 1) using a size filter, and features of the input hand contour image are extracted. S1 and S2 are pooling layers. Here, the Max Pooling technique can be used to reduce the size of the feature map to 2 × 2, that is, the horizontal and vertical sizes, respectively. The maximum pooling layer may have a size range of 2 × 2 in such a manner that the most characteristic portion within the set size range is designated as a representative value of the range. F1 and F2 are Fully-Connected layers, which can be computed on the neural network based on the extracted features to determine the output value. When the probability value is calculated, the control device 300 uses the Softmax function to recognize the gesture having the largest probability value as the gesture of the input image (result value '1'), and the remaining nine gestures have the result value '0'. Can be set with

Thirdly, the CNN optimization algorithm should be run. In order to minimize the cost function in the CNN learning process, the weight of each layer is learned, and the weight is multiplied by the input value to give an output value. In this case, the control device 300 may use an optimization algorithm to adjust the weight. Representative methods include gradient descent and momentum. The gradient reduction method uses a loss function, which is an error function between the output value of the network and the actual value. Is an algorithm that moves in the direction of the minimum value. In this case, the momentum is a method of additionally moving in the direction while remembering the result of the past movement in the process of moving by the slope reduction method. Detailed equations thereof are the same as the disclosed technology, and thus detailed descriptions thereof will be omitted.

Finally, deep learning frameworks include Tensorflow, Theano, MXnet, and Keras, and you can get good performance by choosing the right framework based on what you want to predict or analyze. In an embodiment of the present invention, the S / W language may use an R language specialized for statistics or parallel calculation, and may use MXnetR, Deepnet, H20, etc., which can be utilized in the R language, for each framework. The internal function type is set differently. For example, MXnet is a framework designed to maximize efficiency and flexibility by allowing a mix of symbolic and imperative programming, while Deepnet is relatively small compared to other frameworks, but it is designed to allow a variety of structural choices. have. The H20 can also utilize distributed computer systems and provides an interface to various programming as a predictive analytics platform. Of course, in addition to the above-described method, it will be apparent that the method of extracting the gesture of the hand may be added, modified, or changed according to the embodiment.

In addition, the control device 300 according to an embodiment of the present invention may further use a gesture detection method that is robust to changes in illumination. First, the control device 300 may use two-dimensional pattern information having a small amount of calculation to detect and recognize in real time by the movement of a hand, and convert the RGB color model into an HSI curl model to effectively find the skin color region in the image. The skin color region can be detected, and the detected skin color region is binarized, and then morphology operations can be used to remove noise and labeling to detect skin color regions such as face and hands. In addition, the control device 300 can find a correct hand area by searching for a face area using a Haar-like feature, classifying the face and hand areas, and removing the face area from the classified face and hand areas. Make sure

Next, the control device 300 may convert the RGB color model into an HSI color model of Hue, Saturation, and Intensity, so as to be robust to lighting changes. In addition, the hand region should be detected. In general, when the skin color is detected, the face region and the hand region are detected together, and a task of removing the face region through the classification of the face and the hand region is necessary to find the correct hand region. In addition, the hand area is fluid and not fixed in shape, so it is easier to find a face area where the eyes, nose and mouth are fixed. Even in the hand region where the face region is removed, the skin color detection region varies according to the length of the user's clothes. If you come to your wrist in long sleeves, the area of your hand is excluded, but if you are wearing short sleeves, the area of your hand that includes your arm is recognized. Therefore, the control device 300 needs a method of finding only a unique hand region from which a wrist part is removed for gesture recognition, and a non-moving area with less movement is required to find a wrist. The finger area is the most active area of the hand area, but the palm and wrist parts are very less mobile. In particular, the width of the wrists and arms is almost unchanged. So you can find your wrist using the width of your arms and palms, regardless of the shape of your hand. In addition, the line of the wrist can be detected by using the feature of narrowing gradually from the lower part of the arm to the upper part of the arm and rapidly increasing from the wrist part. Of course, in addition to the above-described method can be used a variety of methods that are robust to changes in illumination.

Hereinafter, an operation process according to the configuration of the control apparatus of FIG. 2 described above will be described in detail with reference to FIGS. 3 and 4. However, the embodiment is only any one of various embodiments of the present invention, but is not limited thereto.

Referring to FIG. 3, (a) the VR booth may be installed in an arcade, and the user may experience VR using an HMD, a controller, or the like. At this time, the lighting, such as a special effect is controlled on and off or output level when an event in the game occurs or the user's gesture or motion is detected as described above, and is installed in an arcade space, such as a theater, It can be operated without management as Standalone, so it is configured to achieve sales with the minimum number of managers. And, if the at least one light source 500 is made of neon, at the time of the night game, at least one of the light source 500 installed along the outer periphery of the booth on and off and light emission to the user as well as the customer to watch the effect And marketing. (b) shows that hand gestures are linked by special effects and NUI mapping. Of course, it will be apparent that the hand gestures may be added, deleted, or modified in addition to those described above.

Referring to FIG. 4, FIG. 4 is a modeling view of the VR booth viewed from various angles. The VR booth may be equipped with two chairs and all the equipment for two people to play the game, but may be designed to be operated by one person or operated by more than two people. In this case, when two people are playing a game, when each camera for photographing two people is provided in stereo, the process of matching stereo images is performed as described above.

2 and 3 that are not described with respect to the lighting effect control method using the gesture-based NUI from the same or described above with respect to the lighting effect control method using the gesture-based NUI through FIG. Since it can be easily inferred, a description thereof will be omitted.

FIG. 5 is a diagram illustrating a process of transmitting / receiving data between components included in a lighting effect control system using the gesture-based NUI of FIG. 1 according to an embodiment of the present invention. Hereinafter, an example of a process in which data is transmitted and received between each component will be described with reference to FIG. 5, but the present invention is not limited to the above-described embodiment, and is illustrated in FIG. 4 according to the various embodiments described above. It is apparent to those skilled in the art that the process of transmitting and receiving data may be changed.

Referring to FIG. 5, the control device is installed in a VR booth for a realistic experience captured by at least one infrared camera and receives movement data including a motion or a gesture of a game subject (S5100).

The control device compares the motion data with previously stored NUI mapping data to determine whether to implement the special effect (S5200), and when the result is the motion data to which the special effect is to be implemented, at least one. In order to independently control the light source, the control signal is transmitted to a dimmer controller using a digital addressable lighting interface (DALI) protocol (S5300).

In addition, the control device receives a result of outputting a signal for controlling at least one light source from the dimmer as feedback data and performs PID (Proportional Integral Derivative) control (S5400).

The order between the above-described steps S5100 to S5400 is merely an example, and the present invention is not limited thereto. That is, the order between the above-described steps (S5100 to S5400) may be mutually changed, and some of the steps may be executed or deleted at the same time.

The matters not described with respect to the lighting effect control method using the gesture-based NUI of FIG. 5 are the same as or have been described above with respect to the lighting effect control method using the gesture-based NUI through FIGS. 1 to 4. Since it can be easily inferred, a description thereof will be omitted.

The matters that are not described with respect to the lighting effect control method using the gesture-based NUI of FIG. 5 are the same as or described above with respect to the lighting effect control method using the gesture-based NUI through FIGS. 1 to 4. Since it can be easily inferred, a description thereof will be omitted.

The lighting effect control method using the gesture-based NUI according to an embodiment described with reference to FIG. 5 may also be implemented in the form of a recording medium including instructions executable by a computer, such as an application or a program module executed by a computer. Can be. Computer readable media can be any available media that can be accessed by a computer and includes both volatile and nonvolatile media, removable and non-removable media. In addition, computer readable media may include all computer storage media. Computer storage media includes both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data.

Lighting effect control method using a gesture-based NUI according to an embodiment of the present invention described above, by the application (which may include a program included in the platform, operating system, etc. basically installed in the terminal) It may be executed, or may be executed by an application (ie, a program) installed by the user directly on the master terminal through an application providing server such as an application store server, an application, or a web server associated with the corresponding service. In this sense, the lighting effect control method using a gesture-based NUI according to an embodiment of the present invention described above is implemented as an application (that is, a program) that is basically installed on the terminal or directly installed by the user and to a computer such as a terminal. It can be recorded on a readable recording medium.

The above description of the present invention is intended for illustration, and it will be understood by those skilled in the art that the present invention may be easily modified in other specific forms without changing the technical spirit or essential features of the present invention. will be. Therefore, it should be understood that the embodiments described above are exemplary in all respects and not restrictive. For example, each component described as a single type may be implemented in a distributed manner, and similarly, components described as distributed may be implemented in a combined form.

The scope of the present invention is shown by the following claims rather than the above description, and all changes or modifications derived from the meaning and scope of the claims and their equivalents should be construed as being included in the scope of the present invention. do.

Claims (8)

At least one light source installed inside or outside the VR booth for a realistic experience to irradiate light to correspond to a predetermined event in order to implement special effects according to game data or motion data including a motion or gesture of a game subject;
A dimmer controller that increases or decreases the light output of the at least one light source by using a digital addressable lighting interface (DALI) protocol, and individually assigns an address to each of the at least one light source to perform individual control. ;
At least one infrared camera for capturing motion data of the game subject;
A control device for controlling the at least one light source when the motion data or a predetermined event occurs through the dimmer based on the motion data input from the at least one infrared camera;
Lighting effect control system using a gesture-based NUI comprising a.
The method of claim 1,
The gesture of the game subject in the motion or gesture is,
Any one or at least one of a figure drawing operation using a finger, a left and right shake, an up and down touch, a horizontal touch, a clenching fist, a pinch of a finger, a finger spread, a palm touch, a finger spread, a bending, and a palm flip,
The hand gesture is a lighting effect control system using a gesture-based NUI, characterized in that the data to control the driving and output of the at least one light source and is connected to the NUI mapping (Natural User Interface mapping).
The method of claim 1,
The control device is configured to detect and analyze motion data of the game subject using location information, a coordinate range, and a detection level of the game subject received from the at least one infrared camera. Effect control system.
The method of claim 1,
At least one controller positioned on at least one human body part of the game subject to detect motion data of the game subject;
Lighting effect control system using a gesture-based NUI, characterized in that it further comprises.
The method of claim 1,
The at least one light source is a lighting effect control system using a gesture-based NUI, characterized in that consisting of any one or at least one of Flexible Neon, LED, OLED, and LCD.
In the lighting effect control method performed in the control device,
Receiving movement data including a motion or a gesture of a game subject, installed in a VR booth for a realistic experience captured from at least one infrared camera;
Comparing the movement data with previously stored NUI mapping data to determine whether to implement a special effect;
Transmitting control signals to a dimmer controller using a digital addressable lighting interface (DALI) protocol to independently control the at least one light source when the special effect is motion data to be implemented;
Receiving a result of outputting a signal controlling the at least one light source from the dimmer as feedback data to perform PID (Proportional Integral Derivative) control;
Lighting effect control method using a gesture-based NUI comprising a.
The method of claim 6,
At least one infrared sensor for collecting three-axis data including X, Y, and Z of the motion data of the game subject;
More,
The control device generates a protocol by analyzing three-axis data including X, Y, and Z of the shooting data captured by the at least one infrared camera and the movement data of the game subject collected from the at least one infrared sensor. Lighting effect control method using a gesture-based NUI, characterized in that.
A computer-readable recording medium having recorded thereon a program for performing the lighting effect control method using the gesture-based NUI according to claim 6.

Images