KR20230068635A - Apparatus and method for recognizing motion in video - Google Patents

Abstract

The present invention relates to a game apparatus for recognizing a motion which enables a user to execute a mission by using a character selected by the user through a motion of the user as the character selected by the user imitates the motion. According to an embodiment of the present invention, the game apparatus for recognizing a motion is realized by sensing the motion of the user and enabling the character selected by the user to perform the same motion as that of the user in real time in video data. The game apparatus for recognizing a motion includes: a character data storing unit storing image data or the video data about multiple characters; a character data receiving unit receiving character data including information about the character selected by the user from a user terminal used by the user; a motion data receiving unit receiving motion data that a picture of the user taking the motion is taken by a camera module of the user terminal; a human body data generating unit generating human body data by extracting the body of the user from the motion data; a candidate area generating unit generating multiple candidate areas by dividing the human body data by each human body part; a subject recognition unit recognizing the human body part included in the candidate area by extracting a characteristic of each candidate area by using a deep learning algorithm; and a matching data extracting unit classifying the human body part recognized by the subject recognition unit by using a machine learning algorithm and extracting matching data corresponding to a unit cell of each image data of the classified human body part from the image data or the video data of the character selected by the user.

Description

Motion Recognition Game Device {APPARATUS AND METHOD FOR RECOGNIZING MOTION IN VIDEO}

The present invention relates to a motion recognition game device, and more particularly, recognizes a user's motion and allows a character selected by the user to perform the corresponding motion as it is, so that the user can perform a mission through the user's motion. It relates to a motion recognition game device and method.

Research is being conducted on various methods for discriminating the characteristics of object shapes in images and how to design and detect such information. This object recognition image processing directly or indirectly affects all industries such as education, shopping, and transportation, and can be a clue to solving recognition-related problems needed in modern society.

Existing object recognition image processing and motion recognition technologies require a physical and temporal synchronization process of data according to the performance capture environment production process, and it is impossible to process multiple motion recognition simultaneously due to limitations in hardware specifications.

In the present specification, a motion recognition game device that can be free from these limitations, does not require a separate hardware device, and recognizes a user's motion only with motion data received through a camera, and from which the user can play a game while performing a mission. I would like to introduce about

[Prior literature]

Registered Patent 10-1990123

The present invention relates to a motion recognition game device, and more particularly, recognizes a user's motion and allows a character selected by the user to perform the corresponding motion as it is, so that the user can perform a mission through the user's motion. It relates to a motion recognition game device and method.

According to an embodiment of the present invention, a motion recognition game device implemented by detecting a user's motion and allowing a character selected by the user to perform the same motion as the user's motion in real time in image data is a motion recognition game device for a plurality of characters. A character data storage unit for storing image data or video data, a character information receiver for receiving character information including information on a character selected by the user from a user terminal used by a user, and a camera module of the user terminal taking motion. A motion data receiving unit receiving motion data of a user, a body data generating unit generating body data by extracting the user's body from the motion data, and generating a plurality of candidate regions by dividing the body data by body part A candidate region generation unit, an object recognition unit that recognizes a body part included in the candidate region by extracting features for each of the candidate regions using a deep learning algorithm, and machine learning of the body part recognized by the object recognition unit A matching data extractor that classifies using an algorithm and extracts matching data corresponding to each unit cell of the image data of the classified body part from the image data or video data of the character selected by the user; After conversion into matching data, a matching data output unit outputting the matching data to the user terminal, a background data extraction unit extracting background data excluding the body data extracted from image data captured by the camera module, and the background data. and an object assigning unit for assigning an object of a mission to be performed by the user through his/her own motion to the data according to time, and the matching data output unit merges the extracted background data with the matching data when an object is assigned to the extracted background data. It is characterized by outputting to a terminal.

A motion recognition game device according to an embodiment of the present invention uses image data of space as learning data and generates moving space data obtained by calculating the user's movable space as coordinate data from background data extracted through a deep learning algorithm The deep learning algorithm is an algorithm through a Region Proposal-Convolutional Neural Network (R-CNN) neural network, and the machine learning algorithm uses a Support Vector Machine.

When the matching data merged with the background data to which the object is assigned is output to the user terminal, the user moves his or her body part so that it overlaps the part corresponding to the position of the object in the matching data output from the user terminal. Do it.

The motion recognition game device characterized in that the object assigning unit assigns an object so as to be overlapped on a path on which the user can move based on the generated movement space data.

As such, through the motion recognition game device of the present invention, the user can play a game in which the user dances to the rhythm of music through a character that follows his or her motion in real time in a virtual space implemented with matching data, and furthermore, in a real space, the user By giving an object to a virtual space in consideration of a space in which the user can move, it is possible to allow the user to perform a mission without any obstacles to movement while playing the game.

1 is a schematic diagram of a motion recognition game system according to an embodiment of the present invention.
2 is a block diagram of a motion recognition game device according to an embodiment of the present invention.
3 is a block diagram of a data receiver according to an embodiment of the present invention.
4 is a diagram showing line data connecting body parts recognized by an object recognition unit according to an embodiment of the present invention by way of example.
5 is a diagram showing a character performing a user's motion in the present invention by way of example.
FIG. 6 is a diagram showing a scene in which a character performing a user's motion in the present invention performs a mission by moving a body part to overlap an object assigned to background data, by way of example.

Hereinafter, embodiments of the present invention will be described in detail so that those skilled in the art can easily practice the present invention with reference to the accompanying drawings. However, the present invention may be embodied in many different forms and is not limited to the embodiments described herein. And in order to clearly explain the present invention in the drawings, parts irrelevant to the description are omitted, and similar reference numerals are attached to similar parts throughout the specification.

Throughout the specification, when a part is said to be "connected" to another part, this includes not only the case where it is "directly connected" but also the case where it is "electrically connected" with another element interposed therebetween. . In addition, when a certain component is said to "include", it means that it may further include other components, not excluding other components, unless otherwise stated. Refer to the accompanying drawings below. So, the present invention will be described in detail.

1 is a schematic diagram of a motion recognition game system 1000 according to one embodiment of the present invention.

Referring to FIG. 1 , a motion recognition game system 1000 according to an embodiment of the present invention includes a user terminal 100 and a motion recognition game device 200 linked to the user terminal 100 through a network 400. can do.

The user terminal 100 may be a terminal used by a person using the motion recognition game device 200 . More specifically, the user terminal 100 may be a terminal used by a person who installs and uses an application implemented from the motion recognition game device 200 . The application may be a game application, and when the user executes the game application and takes a motion, the game may proceed as image data of a specific character performing the motion in real time is output to the user terminal 100 through the application. there is. This process will be described in more detail in FIGS. 2 to 6 .

The user terminal 100 may be a smart phone. However, it is not limited thereto, and the user terminal 100 may include electronic devices such as general desktop computers, navigation devices, laptop computers, digital broadcasting terminals, personal digital assistants (PDAs), portable multimedia players (PMPs), and tablet PCs. An electronic device may have one or more general or special purpose processors, memory, storage, and/or networking components (wired or wireless).

The motion recognition game device 200 may receive data from the user terminal 100 through the network 400 and provide data resulting from processing the data to the user terminal 100 . The motion recognition game device 200 may be a server and may be implemented in the form of an application in the user terminal 100 .

The communication method of the network 400 is not limited, and an example of a communication network that the network 400 may include is a communication method utilizing a mobile communication network, wired online, wireless online, and broadcasting networks, as well as short-range wireless communication between devices. can be included For example, the network 400 includes a personal area network (PAN), a local area network (LAN), a campus area network (CAN), a metropolitan area network (MAN), a wide area network (WAN), and a broadband network (BBN). It may include one or more arbitrary networks among networks such as , online, and the like.

Figure 2 is a block diagram of a motion recognition game device 200 according to an embodiment of the present invention, Figure 3 is a block diagram of a data receiving unit 201 according to an embodiment of the present invention, Figure 4 is a block diagram of the present invention It is a diagram exemplarily showing line data connecting body parts recognized by the object recognition unit 205 according to an embodiment, and FIG. 5 is a diagram exemplarily showing a character performing a motion of a user in the present invention. , FIG. 6 is a diagram showing a scene in which a character performing a user's motion in the present invention performs a mission by moving a body part so that it overlaps an object assigned to background data.

2 to 6, the motion recognition game device 200 according to an embodiment of the present invention includes a data receiver 201, a character data storage unit 202, a body data generator 203, and a candidate area generator. 204, object recognition unit 205, matching data extraction unit 206, matching data output unit 207, background data extraction unit 208, object granting unit 209, and movement space data generation unit 210 ) may be included. The data receiving unit may include a character information receiving unit 211 and a motion data receiving unit 212 .

The character data storage unit 202 may store image data or video data for a plurality of characters. More specifically, the character data storage unit 202 receives image data or video data for a plurality of characters from the user terminal 100 or a terminal used by an administrator who manages the motion recognition game device 200. can be saved The plurality of characters may be famous characters, celebrities, great people, and the like. Image data or video data for a plurality of stored characters can be converted into big data and used in the process of extracting matching information through an AI algorithm to be described later.

The character information receiving unit 211 may receive character information including information about a character selected by the user from the user terminal 100 .

The user can check the classification information classified by the character data storage unit 202 through an application and select one of the characters, and the character information receiving unit 211 transmits the character information selected by the user to the user terminal 100. ) can be received from

The motion data receiving unit 212 may receive motion data obtained by photographing a user making a motion by a camera module of the user terminal 100 . Motion data may be video. As mentioned above, the application implemented by the video motion recognition device may be a game application, and the user may take motions according to motion missions given in the application, and these motions may be performed by a shooting mode driven by the application. can be filmed with

The body data generation unit 203 may generate body data by extracting the user's body from motion data. The body data generation unit 203 detects whether there is a human body in the received motion data, and if there is no human body, the motion recognition process does not proceed because the body data is not generated.

As an example of the present invention, a known object detection algorithm may be used as a method for the body data generator 203 to sense and extract the user's body from motion data. The corresponding algorithm can detect the body using visible ray image information or thermal image information, and can improve the accuracy of body detection by simultaneously utilizing the two image information.

The candidate region generation unit 204 may generate a plurality of candidate regions by dividing body data according to body parts.

The candidate region generation unit 204 may generate candidate regions by detecting the user's body part included in the motion data and segmenting the detected body parts according to the detected body part. A body part may include a face part.

The candidate region generation unit 204 may generate regions where body parts are likely to be located in the motion data through a selective search algorithm.

The object recognition unit 205 may recognize a body part included in the candidate region by extracting features for each candidate region using a deep learning algorithm. As shown in FIG. 4 , the object recognition unit 205 may perform a process of extracting recognized body parts as dots and connecting them.

The deep learning algorithm may be an algorithm through a Region Proposal-Convolutional Neural Network (R-CNN) neural network. R-CNN is a CNN technique based on Region Proposal, which uses the features of the Region Proposal extracted from an image to classify objects in the candidate region and correct the location of the object. The object recognition unit 205 uses the cell labeling information assigned to each cell included in the image data and image data of the character stored in the character data storage unit 202 as learning data, and converts each of the candidate regions to a standard size. A body part included in the candidate region can be recognized through an algorithm using an R-CNN neural network that determines cells.

The algorithm through the R-CNN neural network can proceed like the selective search algorithm described above. That is, candidate regions where objects may exist are extracted from the input image using a selective search algorithm, and then each candidate region is input to an individual CNN to extract features of the candidate region to classify the type of object existing in each candidate region. , the detection position of the object can be precisely corrected by narrowing the bounding box for the object.

The matching data extraction unit 206 classifies the body part recognized by the object recognition unit 205 using a machine learning algorithm, and matches the image data of the classified body part corresponding to each unit cell. Data may be extracted from image data or video data of a character selected by a user.

The machine learning algorithm may be an algorithm using a support vector machine.

The matching data output unit 207 may output the matching data to the user terminal 100 after converting the classified image data into matching data. The process of generating matching data may be performed through a regression learning process of machine learning.

When the matching data is output to the user terminal 100 in this way, as shown in FIG. 5 , the user may check through the user terminal 100 that the character selected by the user takes the same motion as the user's motion.

When the motion data is received in real time, the matching data output unit 207 may output matching data corresponding to unit frames of the motion data to the user terminal 100 in real time. This can be applied to a dance game application implemented from the motion recognition device of the present invention mentioned in the description of FIG. 1 above. More specifically, it may be a game application in which, when a user dances (motions) according to a specific pattern and takes an accurate motion, a score is given accordingly and the user can go to a higher level. In such a game application, the user can check the motion through the user terminal 100 while taking a motion according to a specific pattern. You can get more interest in the game while checking in real time that you follow the same through the algorithm.

While playing such a dance game, the user may perform missions through motions of himself or herself. It will be explained in detail below.

The background data extractor 208 may extract background data excluding body data extracted from image data photographed by the camera module. The background data extractor 208 may recognize the remaining image data after excluding the aforementioned body data as background data and extract the image data.

The object assigning unit 209 may assign an object of a mission to be performed by the user through his or her motion to the background data according to time. When an object is given to the extracted background data, the matching data output unit 207 may merge the extracted background data with matching data and output the result to the user terminal 100 . For example, as shown in FIG. 6 , the matching data output unit 207 may merge background data and matching data to which objects (red circle, yellow circle, blue circle) are assigned and output the merged data to the user terminal 100. In addition, the user can perform the mission by having the character in the matching data touch the object through his or her motion. In other words, when the matching data merged with the background data to which the object is assigned is output to the user terminal 100, the user's body part overlaps with the part corresponding to the position of the object in the matching data output from the user terminal 100. You can move as much as possible to complete the mission.

Hereinafter, a process of assigning an object by the object assigning unit 209 will be described in detail.

The movement space data generation unit 210 may generate movement space data obtained by calculating a user's movable space as coordinate data. Specifically, the movement space data generation unit 210 may generate movement space data obtained by calculating a user's movable space as coordinate data from background data extracted through a deep learning algorithm using image data of the space as learning data. .

The object assigning unit 209 may assign an object to be overlapped on a path through which a user can move based on the generated movement space data. That is, the object assigning unit 209 does not assign an object to a space where the user cannot move, but assigns an object to a space where the user can move, so that the user can freely touch the object through his/her motion, thereby playing a game. can proceed.

As such, through the motion recognition game device 200 of the present invention, the user can play a game in which he dances to the rhythm of music through a character that follows his motion in real time in a virtual space implemented as matching data, By giving an object to a virtual space in consideration of a space in which the user can move in the space, it is possible to enable the user to perform the mission without any obstacles to movement while playing the game.

The above-described embodiments are for illustrative purposes, and those skilled in the art to which the above-described embodiments belong can easily transform into other specific forms without changing the technical spirit or essential features of the above-described embodiments. You will understand. Therefore, it should be understood that the above-described embodiments are illustrative in all respects and not restrictive. For example, each component described as a single type may be implemented in a distributed manner, and similarly, components described as distributed may be implemented in a combined form.

The scope to be protected through this specification is indicated by the claims to be described later rather than the detailed description, and should be construed to include all changes or modifications derived from the meaning and scope of the claims and equivalent concepts thereof.

100: user terminal 202: character data storage unit
200: motion recognition game device 203: body data generator
201: data receiving unit 204: candidate area generating unit

Claims (4)

A motion-recognition game device realized by detecting a user's motion and allowing a character selected by the user to perform the same motion as the user's motion in real time in image data,
a character data storage unit that stores image data or image data for a plurality of characters;
a character information receiving unit receiving character information including information on a character selected by the user from a user terminal used by the user;
a motion data receiving unit for receiving motion data of a user taking a motion by a camera module of the user terminal;
a body data generating unit extracting the user's body from the motion data and generating body data;
a candidate region generation unit configured to divide the body data according to body parts to generate a plurality of candidate regions;
an object recognition unit extracting features for each of the candidate regions using a deep learning algorithm and recognizing a body part included in the corresponding candidate region;
The body part recognized by the object recognition unit is classified using a machine learning algorithm, and matching data corresponding to each unit cell of the image data of the classified body part is extracted from the image data or video data of the character selected by the user. Matching data extraction unit to do;
a matching data output unit that converts the classified image data into the matching data and outputs the corresponding matching data to the user terminal;
a background data extraction unit extracting background data excluding the body data extracted from image data photographed by the camera module; and
An object granting unit for assigning an object of a mission to be performed by the user through his/her motion to the background data according to time;
The matching data output unit, when an object is given to the extracted background data, merges it with the matching data and outputs it to the user terminal. According to claim 1,
Further comprising a movement space data generation unit for generating movement space data obtained by calculating the user's movable space as coordinate data from background data extracted through a deep learning algorithm using image data of space as learning data;
The deep learning algorithm is an algorithm through a R-CNN (Region Proposal-Convolutional Neural Network) neural network,
The image motion recognition device, characterized in that the machine learning algorithm is an algorithm using a support vector machine. According to claim 2,
When the matching data merged with the background data to which the object is assigned is output to the user terminal, the user moves his or her body part so that it overlaps the part corresponding to the position of the object in the matching data output from the user terminal. A motion recognition game device characterized in that for performing. According to claim 3,
The motion recognition game device according to claim 1 , wherein the object assigning unit assigns an object to be overlapped on a path through which a user can move based on the generated movement space data.

Images