KR200239844Y1 - Simulation game system using machine vision and pattern-recognition - Google Patents

Abstract

The present invention simulates the motion picture character set as the default dance in front of one or more cameras 15 recording the motion of the human body in the chroma key screen (outside shadow extraction background) (18). The present invention relates to a dance game apparatus of a new method of scoring a score in comparison with an image.

The basic dance movement provided by the monitor 13 can be used to dynamically produce each part of the human body in accordance with music of a predetermined rhythm, and is a non-contact game method that relies solely on the artificial visual algorithm of the camera, which is different from the conventional scaffold detection method. A background tent (Green Screen) 18, which is installed to block the view of the outside background behind the user, obtains the conversion result according to the movement of the subject (user) 19 in front of one or more digital wide-angle cameras 15. Image input unit 21, the control unit 22 for pattern matching, deviation analysis and game mode processing between the three-dimensional standard data and the camera input data DB by the motion capture in the storage device, the user's instruction information through the external input image Audio output unit 231 for outputting the selected music, the image output unit 232 for displaying the position information and the movement of the three-dimensional character on the monitor in real time There is sex.

The present invention has a structure that can be developed while easily learning the dance of the presented 3D character model by adapting the repeating units of the rhythm divided into each measure unit to the user according to the music and the dance according to the selected difficulty level.

Description

Sensory game device using artificial vision and pattern recognition. {Simulation game system using machine vision and pattern-recognition}

The present invention relates to a haptic game device. Previously, a dedicated tool such as a keyboard, a joystick, and a mouse used extremely limited input means, but there were many problems in freely transmitting all behavior patterns of a user.

The present invention is capable of developing a variety of games modified from the traditional input method, but in particular, it has the effect of replacing the scaffolding-type dance game input device that is widely used in the commercial entertainment room, according to the shape displayed on the monitor according to the selected music (rhythm). It is a game device that extracts user image using artificial visual technique and scores after analyzing 3D character silhouette motion on the display, skeletal line agreement, and deviation of external area.

Most games on the market are being developed or developed, using the foot pad method to step on the direction data of the diagonal, vertical and horizontal arrows, reading the monitor scan line from the sensor attached to the instrument, and infrared sensor signal at a certain point by arm or leg. There is a method of checking the motion capture method to track the markers attached to the body with each camera. All of the above techniques were difficult to grasp the detailed movement of the whole body by detecting only a specific part of the human body, and it was difficult to learn the trick of gradually learning the dance movement throughout the body as a uniform progress in the direction of the instruction data.

Therefore, the present invention presents the detailed standard model (pattern prototype) 63 on the screen according to the body size and rhythm input by the user in the modeling data of the three-dimensional dance stored in the memory and internally the artificial visual and pattern recognition Using the principle, user-specific patterns are extracted from the video and then recognized and evaluated.

The machine vision necessary for this is to execute all processes that humans perform on visual information by computer, and the ultimate goal is to substitute and expand human visual functions. Various applications can be made, and the application of processing in terms of application and the advantages of accuracy, objectivity, repeatability and rapidity are increasing awareness of the importance. Artificial vision combines image processing technology and pattern recognition technology with computer technology, and can provide two-dimensional information that can be processed by a computer to provide information about measurement or control of an object.

The process of processing an image of a letter or a figure to determine what it is is called pattern recognition. In order to recognize the pattern, the input pattern image is separated and extracted through various image processing steps. In order to determine what it is, a pattern prototype stored in computer memory is basically required. Existing pattern recognition system is sensitive to pattern size, direction, position, lighting, etc., so the real world application is limited.

The present invention separates the area of the pattern from the background image, extracts the feature, and then determines the outline of the external recognition. It is a method of contrasting the skeletal lines of the human body. Analyze the user's posture by calculating the optimal match while reducing the error range generated from wrinkles.

By applying the above-mentioned principle, it is to provide a tangible game device in which the user actively expresses the dance movement in real time by escaping from the conventional passive dance game system in which the body is moved by limited input means.

1 is a schematic side and front view of a device to which the present invention is applied

2 is a block diagram of a device to which the present invention is applied

Figure 3 shows the operation of choreography

4 is a flow chart of the dance game apparatus

FIG. 5 is a flowchart for explaining the 'pattern recognition processing' subroutine of FIG.

FIG. 6 is a diagram illustrating a progress screen as an example of a display. FIG. 7 is a flowchart illustrating a principle of a pattern recognition process. FIG. 8 is an example of pattern recognition and a score evaluation. 9 is a view illustrating an example of pattern recognition and score evaluation, and a block diagram illustrating image processing with small operation of an input image in FIG. 7.

The basic game apparatus according to the present invention is an output means for generating music 14 and a small wide-angle camera 15 necessary for photographing the same, and a monitor for outputting a 3D character standard image 63 for instructing movement of the whole body in accordance with music. It consists of

If necessary, a stable user 19 of the photographing means, the same primary color floor sheet 17 as the luminaire 20 around the front monitor, an aid for extracting shadows (silhouettes), and a pipe frame 11 for the background screen 18. Install).

1 is a side view and a front view of the overall appearance of a dance game apparatus to which the present invention is applied. First, an image output monitor 11 and a sound source output speaker 14 are located on the upper part of the main body 16, and a solid background screen 18 is disposed between a column fixed to the side and a pipe frame 11 connected by a square steel structure. It is connected. A monochromatic bottom sheet 17 of the same color as the background screen 18 is adhered to the bottom, and a small digital camera 15 equipped with a wide-angle lens is fixed behind the center of the main body 16. The image output monitor 13 is installed in consideration of the eye height of the standard key and is designated by pressing a direction key in the center of the main body 16 with a finger or by touching a virtual space screen on which a user's motion is projected.

In the structure associated with the user's activity space, the outer pipe frame 11 of the hexahedron is positioned higher than that of a person of average height, as shown in FIG. 1, and the monochrome background screen 18 is also a pipe frame 11. ) Increase the height to the ground level. The width and width of the pipe frame 11 and the bottom sheet 17 are such that the field of view 12 of the digital camera 15 can be sufficiently secured.

2 is a block diagram of a device to which the present invention is applied. Inside the main body 16, a control unit 22 composed of a computer is contained. When the video signal is received from the input unit 21, the music selector 224 executes the music and keyframe data. The data storage unit 227, which is an auxiliary memory device for storing a program including motion control, also records a selected music compression file and motion key animation data. The central processing unit (CPU) that performs arithmetic logic operations receives the performance data checked by the counter 228 from the music selection unit 224 and synchronizes in real time to output the output data from the sound source processing unit 229 to the speaker 231. Instructs and outputs the 3D animation data to the monitor 232 via the image processing unit. The network processing unit 220 is mounted for two or more participants, and two or more external devices 233 are linked to each other, and many of them can compete in the superiority of the dance or perform a group choreography.

In the detailed structure of the data storage unit 224, as shown in FIG. 3, the rhythm and the choreography (c) data divided by the beat (a) and the measure (b) are stored in a record format. . The dance consists of four beats (A1 to A4), the most basic unit being four beats in one measure. When there are four measures, 16 beats (B1 to B4) are used. In some cases, the motion is divided according to the counter beat. In this way, the choreography points are recorded in the data storage 224 according to the rhythm for each dance genre. The four time signatures (A1 to A4) are selected as one measure, and the operation units (C1 to C4) are generally divided into two measures (B1 to B2), four measures (B1 to B4), and eight measures (B1 to B8). do. You can set various dance types such as Latin, techno, soul, hip hop, and wave. The more difficult the difficulty, the faster the rhythm and the longer the measure (b) unit.

The following is a method of analyzing the shape of the user's motion. In a condition where there is a wall hidden behind the user 19 in front of the device and there is no external obstacle, the primary color floor sheet 17, the background screen 18, and the pipe frame 11 The user's body extraction principle required for measurement is to store the background 71 captured by the digital camera in the memory before the subject appears, and then detect the change in color, brightness, and saturation that occur when the user appears. The basic form is extracted. The essential requirement is that there are no moving objects in front of the camera that cause confusion with the human body of the subject. Alternatively, the background must have a metallic surface reflecting the front object or a material with a high gloss finish, so that it does not reflect the afterimage of the monitor or the back of the user.

If the above conditions are not met, the pipe frame 11 for the primary color floor sheet 17 and the background screen 18 should be additionally installed. In the extraction of the user's feature, the change of brightness which is slightly lower in the same color than the wall and floor stored in the memory is excluded from the extraction object in consideration of the problem of detecting the shadow of the human body.

After that, the pattern / skeletal line inspection unit 222 undergoes a complex process of checking the difference between the shade from which the user feature is extracted (221) and the result of the choreography presented as a standard. The technique of inferring the correlation 504 by calculating the ratio 503 of the deviation of the external region of the standard motion and the position of the human skeletal line between the two images. According to the result of performing the pattern / skeletal line inspection unit 222, the rise and fall of the parameters are evaluated, and the next connection operation is read from the data storage unit 227, and then the process is transferred to the sub-control unit 223.

The method of analyzing the coincidence ratio with the standard pattern by the data of the input image should be a differentiated game progression method that can reduce the calculation error occurring and acquire the input operation gradually. Therefore, the progress level 62 is evaluated based on the average score at the time the basic measure (b) is completed, rather than the calculation based on the accuracy of one frame. If the matching probability of extracted choreography (c) is low, repeat until it reaches a certain level or more and then move on to the next choreography (c). When one action is mastered, the next choreography (c) is automatically presented when the player's operation error persists several times in the situation where the next choreography (c) continues. It is suitable for both beginners and experienced users.

Next, the operation procedure of the game apparatus will be described using the flowcharts of Figs.

4 is a flow chart showing the whole of the game device, when a signal is detected at the coin inlet 211, the user must initially input his height 400 among three options of 160Cm or less, 160-170Cm, 170Cm or more. Since the length of the arm and leg and the height of the waist, the ratio of the head and the height may vary depending on the height of various users, the data of the 3D standard models being compared are modified. After that, when the music selection input 401 is switched, a user's desired song is selected, and performance data is read from the data storage unit 227 and output to the speaker 231. When the electric pole portion 402 proceeds, a preview 403 of the choreographed movement in the time unit is presented, and then the choreography of the selected difficulty is connected and displayed on the monitor. On the screen, the input image 64 is superimposed on the standard image 63, and the pattern recognition processing 404 in units of beats is performed internally with other images required for a comparison operation. When the operation of all beats in the frame unit is checked, when the one measure is completed (405), the operation divided into large units is evaluated (406), and the next data block is set (409) based on the grade of the evaluation (410 to 403). Repeat until. If the game is finished 410 is determined, the added evaluation total point 411 is displayed on the monitor.

5 is a flowchart showing a subroutine of the 'pattern recognition process 404'.

In general, after extracting the basic elements of the input image divided between beats, the image preprocessing process (501) is performed. This process saves the back background captured by the camera in advance in memory and saves the human body shape when the user participates. The input image is extracted using the color and brightness differences on the background obscured by. Thereafter, the result of the approximate processing of the motion matching ratio is converted (505) through a three-step measurement method (502 to 504) that filters and removes noise generated by an error of the camera hardware during image comparison and identifies characteristics of internal particles. To return a value.

FIG. 7 illustrates a process of processing the principle of the pattern recognition process of FIG. 5 into an image. First, the still background 71 image before the user of the digital camera 15 appears is stored. Thereafter, the user 72 of the input image is photographed to measure the change between the two images to generate a silhouette-processed extracted image 73. Noise generated by the characteristics of the digital camera is filtered by comparing the area of the portion adjacent to the pixel. The noise elimination method is likely to be noise if the color value of the pixel to be detected is different from the color values of the upper and lower left and right sides, so that the screen 74 of the filtered input image is deleted after deleting the input image. 74 is performed by two measurement methods, the first of which inputs the standard image 75 composed of only the shadows of the wide 3D characters of the wide thickness produced by the animation keyframe data for the operation instruction to the data storage unit 227. Comparing with the image 74, the area of the region that does not match is calculated (76). Second, by comparing the vertex position of the 3D character human skeletal line (77) corresponding to the standard image with the input image (74) by checking the external area that is not included within the rectangular vertex to measure the degree of agreement. The greater the difference in motion, the greater the deviation from the outside. Before comparing the two images, the standard image shade 75 and the skeletal line 77 are enlarged and reduced in accordance with the position and overall size ratio of the input image 74. , Move it. If the user moves closer to the camera or moves outside the image, an error occurs in the measurement, so a warning message is displayed on the monitor and the deduction process is performed. FIG. 8 The change in posture is clearly a different operation, and FIG. 9 is a subtle change in posture. An example of recognition and score evaluation is divided into an input image (B) similar to the standard motion and an input image (C) of the wrong motion. The algorithm shown here is not a means of accurately measuring the motion error of each part of the body, The motion image may be distinguished from the motion image. The image M4 captured by the camera is converted into the input image silhouette M5, and then overlapped with the standard image shade M2 in which the joint width is widened (M6). As the detected error area ratio (M7) protruding from the outside of the input image is closer to 0%, a perfect matching ratio is shown, and as the error area ratio (M7) increases with respect to the area of the input image, it is evaluated as different from the standard operation. Subsequently, the number of inconsistent vertices (M9) deviating to the outside of the input image is measured by comparing (M8) the skeletal vertices (M3) and the input image silhouette (M5) of the standard model with the quadrangle vertices in parallel with the detection result. The lower the number of mismatched vertices (M9), the more similar it is to standard behavior, and the higher the number of mismatched vertices. The final deviation amount (M10) is detected by adding the area error ratio (M7) and the number of mismatched vertices (M9), and as the value increases, the score is evaluated as low. After the calculation, the average of the deviation amount M10 detected in units of one measure is continuously calculated, and the score set in advance is displayed on the monitor 232. FIG. 6 shows a game progress screen displayed on the monitor 232. FIG. In the upper right portion, the user's score 61 is described, and a preview 66 for the next operation is illustrated in the upper left portion. The level bar increases (62) if the dance motion is similar to the 3D character (63) of the standard video, and decreases if the dance is wrong. If the level does not reach the required position even after one song is completely finished, The game is terminated. At the bottom of the text 65, choreography precautions, advice on the wrong posture, dancing tips, etc. are recorded.

Therefore, as outlined above, users can easily mimic their dance movements by projecting their outlines on the monitor and comparing them with the 3D characters of standard movements, and repeatedly acquiring wrong sections several times. Increase the interest of the game with aerobic exercise effect.

Elements of the above implementation principle can be produced at a low price in terms of cost, and can be applied to game genres of other genres such as aerobics, fighting and sports.

Claims (4)

Output means 14 for generating music, and a small wide-angle camera 15 necessary for capturing the same, data for outputting the monitor 13 for outputting the 3D character standard image 63 to instruct movement of the whole body in accordance with the music on the top of the main body, and an external device A dance game device having a function of allowing a single person or many to enjoy various types of dance movements by mounting a network processor 220 capable of transmission. The method of claim 1, Additional installation of the same primary color floor sheet 17 as the luminaire 20 around the front monitor and the pipe frame 11 for the background screen 18 for extracting the shadows (silhouettes) of the stable user 19 of the photographing means. Dance game device, characterized in that made The method according to claim 1 or 2, And a measuring means for contrast-reading the input image signal photographed by a digital camera of the human body motion on the background screen used for the shadow extraction with the sample motion (pattern circular) data stored in the memory. The method according to claim 1 or 2, The composition of the dance is composed of four beats and one measure unit, and if the probability of matching choreography extracted from the user is low, the choreography of the wrong section is repeated until a score of a certain level or more is achieved, and at the same time, the composition of the production suitable for music Dance game device characterized by the progressive way to learn choreography.