KR102315842B1 - System for providing carom billards guidance service using reinforcement learning of deep learning model - Google Patents

Abstract

Provided is a billiard game guidance service using reinforcement learning of a deep learning model. The billiard game guidance service using reinforcement learning of a deep learning model comprises: a user terminal; and a guidance service providing server. The user terminal is configured to recognize coordinates of a target and at least one ball as position vectors and output simulation results obtained by simulating trajectories of the target and the at least one ball in accordance with a point which is a striking part on the surface of the target and the force that strikes the point. The guidance service providing server includes: a building unit which builds a billiard simulation environment using at least one open source for building a physics engine-based simulation environment; an initialization unit which initially sets the coordinates of the target and the at least one ball randomly in the built billiard simulation environment, and sets the coordinates of the target and the at least one ball as position vectors; a training unit which conducts training by repeating the simulation of drawing the trajectories of the target and at least one ball in accordance with a point within the target and the force that strikes the target and then providing a reward in accordance with the result using reinforcement learning with at least one deep learning model; and a transmission unit which transmits, when the user terminal recognizes or specifies the coordinates of the target and at least one ball, simulation results in accordance with the number of at least one case in accordance with the point of the target and the force that strikes the point to the user terminal. In accordance with the present invention, by means of the billiard game guidance service, even a beginner can intuitively predict a trajectory or direction of the ball.

Description

Billiard guide service provision system using reinforcement learning of deep learning model

The present invention relates to a system for providing a billiard guide service using reinforcement learning of a deep learning model, and provides a platform for building artificial intelligence that can guide the trajectory of a ball according to the number of cases through compensation for the result without a dataset do.

Billiards is a sport made up of physical laws that require a lot of training and adaptation to get used to. The path changes depending on how you hit the ball. The forces that act the most when the ball moves include friction and repulsion. The magnitude of each force varies depending on the point of the ball, the striking force, the coefficient of friction of the pool table, and the coefficient of friction of the cushion. Billiards can score points when both path design and cueing and cueing are performed successfully. Even if beginners design their own path and cue it, it is difficult to send the ball along the path itself. Billiards usually starts as a play rather than a systematic training, and psychological rewards are needed to continue learning as a play. In billiards, scoring corresponds to this, and compared to these barriers to entry, a beginner in billiards has to endure long periods of training without clear psychological rewards until they become proficient.

At this time, a method of showing a simulation and a trajectory in advance to guide the billiards has been researched and developed. Issues (published on July 03, 2008) and Korean Patent Registration No. 10-0970924 (published on July 21, 2010) provide information on the expected or recommended path of a billiard ball through image recognition and billiard simulation applications. , a configuration that analyzes the coordinates and recommended path of the billiard ball through image data using a camera that shoots the billiard table, and recommends a path after performing a virtual simulation using the analysis data, and detects the position change of the billiard ball A configuration that recognizes the camera and objects in the image information, compares, judges, and calculates the stored data and the current data to display the processed data, and continuously detects the 3D coordinate information of the billiard table, billiard ball and cue table, and coordinate information It stores the course information of the billiard ball and the cue table's hitting information corresponding to the Each configuration for outputting the other method information to be guided by voice is disclosed.

However, the above-described configuration only concentrates on the trajectory or path, and there is no guide on how to hit the target in which direction and how much force. In addition, in order to create artificial intelligence that can compete with players, it is necessary to learn artificial intelligence through simulation. The time and cost involved are enormous. Accordingly, research and development of a platform that can perform modeling without a dataset while providing simulation results using artificial intelligence when providing a billiards guide is required.

An embodiment of the present invention performs simulation through reinforcement learning that does not require a dataset based on a deep learning model, and currently in an environment where billiards is placed without a dataset for training and testing through reinforcement learning. By recognizing the state of the state and learning in a way that maximizes the reward through the action, training is carried out so that the behavior can be modified in a good direction for the reward, and the user is simply guided along the trajectory. In addition, reinforcement learning of deep learning models that can predict the trajectory or direction of the ball intuitively, even for beginners, and create artificial intelligence that can confront players It is possible to provide a method of providing a billiard guide service using However, the technical task to be achieved by the present embodiment is not limited to the technical task as described above, and other technical tasks may exist.

As a technical means for achieving the above-described technical problem, an embodiment of the present invention recognizes the coordinates of a target and at least one ball as a position vector, and then the force to hit the hitting part on the surface of the target and the point A building unit that builds a billiard simulation environment using at least one open source for constructing a simulation environment based on a physics engine and a user terminal that outputs a result of simulating the trajectory of a target and at least one ball according to After initially setting the coordinates of the target and at least one ball at random in the environment, an initialization unit that sets the coordinates of the target and at least one ball as a position vector, the target and at least After repeating the simulation of drawing the trajectory of one ball, at least one deep learning model uses reinforcement learning to provide a reward according to the result, thereby recognizing the coordinates of the target and at least one ball in the training unit and user terminal Or, when designated, includes a guide service providing server including a transmission unit for transmitting a simulation result according to the number of at least one case according to the force hitting the point of the target and the point to the user terminal.

According to any one of the above-described problem solving means of the present invention, simulation is performed through reinforcement learning that does not require a dataset based on a deep learning model, and an environment in which billiards is placed without a dataset for training and testing through reinforcement learning By recognizing the current state within (Enironment) and learning in a way that maximizes the reward through the action, training is carried out so that the behavior can be modified in a good direction for the reward, and the user In addition to simply guiding the trajectory to the player, it also tells the location of the point and how to adjust the force and direction that pushes the point, so that even a beginner can intuitively predict the ball's trajectory or direction, and create an artificial intelligence that can compete with the player.

1 is a view for explaining a billiard guide service providing system using reinforcement learning of a deep learning model according to an embodiment of the present invention.
FIG. 2 is a block diagram illustrating a guide service providing server included in the system of FIG. 1 .
3 and 4 are diagrams for explaining an embodiment in which a billiard guide service using reinforcement learning of a deep learning model is implemented according to an embodiment of the present invention.
5 is an operation flowchart illustrating a billiard guide service providing method using reinforcement learning of a deep learning model according to an embodiment of the present invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings so that those of ordinary skill in the art can easily implement them. However, the present invention may be embodied in many different forms and is not limited to the embodiments described herein. And in order to clearly explain the present invention in the drawings, parts irrelevant to the description are omitted, and similar reference numerals are attached to similar parts throughout the specification.

Throughout the specification, when a part is "connected" with another part, this includes not only the case of being "directly connected" but also the case of being "electrically connected" with another element interposed therebetween. . Also, when a part "includes" a component, it means that other components may be further included, rather than excluding other components, unless otherwise stated, and one or more other features However, it is to be understood that the existence or addition of numbers, steps, operations, components, parts, or combinations thereof is not precluded in advance.

The terms "about", "substantially", etc. to the extent used throughout the specification are used in a sense at or close to the numerical value when the manufacturing and material tolerances inherent in the stated meaning are presented, and serve to enhance the understanding of the present invention. To help, precise or absolute figures are used to prevent unfair use by unscrupulous infringers of the stated disclosure. As used throughout the specification of the present invention, the term “step for (to)” or “step for” does not mean “step for”.

In this specification, a "part" includes a unit realized by hardware, a unit realized by software, and a unit realized using both. In addition, one unit may be implemented using two or more hardware, and two or more units may be implemented by one hardware. Meanwhile, '~ unit' is not limited to software or hardware, and '~ unit' may be configured to be in an addressable storage medium or may be configured to reproduce one or more processors. Thus, as an example, '~' refers to components such as software components, object-oriented software components, class components, and task components, and processes, functions, properties, and procedures. , subroutines, segments of program code, drivers, firmware, microcode, circuitry, data, databases, data structures, tables, arrays and variables. The functions provided in the components and '~ units' may be combined into a smaller number of components and '~ units' or further separated into additional components and '~ units'. In addition, components and '~ units' may be implemented to play one or more CPUs in a device or secure multimedia card.

In this specification, some of the operations or functions described as being performed by the terminal, apparatus, or device may be performed instead of in a server connected to the terminal, apparatus, or device. Similarly, some of the operations or functions described as being performed by the server may also be performed in a terminal, apparatus, or device connected to the server.

In this specification, some of the operations or functions described as mapping or matching with the terminal means mapping or matching the terminal's unique number or personal identification information, which is the identification data of the terminal. can be interpreted as

Hereinafter, the present invention will be described in detail with reference to the accompanying drawings.

1 is a view for explaining a billiard guide service providing system using reinforcement learning of a deep learning model according to an embodiment of the present invention. Referring to FIG. 1 , a billiard guide service providing system 1 using reinforcement learning of a deep learning model includes at least one user terminal 100 , a guide service providing server 300 , at least one camera 400 , at least One display 500 may be included. However, since the billiard guide service providing system 1 using reinforcement learning of the deep learning model of FIG. 1 is only an embodiment of the present invention, the present invention is not limitedly interpreted through FIG. 1 .

At this time, each component of FIG. 1 is generally connected through a network (Network, 200). For example, as shown in FIG. 1 , at least one user terminal 100 may be connected to the guide service providing server 300 through the network 200 . In addition, the guide service providing server 300 may be connected to at least one user terminal 100 , at least one camera 400 , at least one display 500 , and the manager terminal 600 through the network 200 . have. Also, the at least one camera 400 may be connected to the guide service providing server 300 through the network 200 . In addition, the at least one display 500 may be connected to the at least one user terminal 100 , the guide service providing server 300 , and the at least one camera 400 through the network 200 .

Here, the network refers to a connection structure in which information exchange is possible between each node, such as a plurality of terminals and servers, and an example of such a network includes a local area network (LAN), a wide area network (WAN: Wide Area Network), the Internet (WWW: World Wide Web), wired and wireless data communication networks, telephone networks, wired and wireless television networks, and the like. Examples of wireless data communication networks include 3G, 4G, 5G, 3rd Generation Partnership Project (3GPP), 5th Generation Partnership Project (5GPP), Long Term Evolution (LTE), World Interoperability for Microwave Access (WIMAX), Wi-Fi (Wi-Fi) , Internet, LAN (Local Area Network), Wireless LAN (Wireless Local Area Network), WAN (Wide Area Network), PAN (Personal Area Network), RF (Radio Frequency), Bluetooth (Bluetooth) network, NFC ( Near-Field Communication) networks, satellite broadcast networks, analog broadcast networks, Digital Multimedia Broadcasting (DMB) networks, and the like are included, but are not limited thereto.

In the following, the term at least one is defined as a term including the singular and the plural, and even if at least one term does not exist, each element may exist in the singular or plural, and may mean the singular or plural. it will be self-evident In addition, that each component is provided in singular or plural may be changed according to embodiments.

The at least one user terminal 100 uses a web page, an app page, a program or an application related to a billiard guide service using reinforcement learning of a deep learning model to shoot a target and a ball in the pool table, or to set the position of the target and the ball After doing so, it may be a terminal that outputs a simulation result of a trajectory including the location or path of the target and the ball according to the target's point, force, and direction.

Here, the at least one user terminal 100 may be implemented as a computer that can access a remote server or terminal through a network. Here, the computer may include, for example, a navigation device, a laptop computer equipped with a web browser, a desktop computer, and a laptop computer. In this case, the at least one user terminal 100 may be implemented as a terminal capable of accessing a remote server or terminal through a network. At least one user terminal 100, for example, as a wireless communication device that guarantees portability and mobility, navigation, PCS (Personal Communication System), GSM (Global System for Mobile communications), PDC (Personal Digital Cellular), PHS(Personal Handyphone System), PDA(Personal Digital Assistant), IMT(International Mobile Telecommunication)-2000, CDMA(Code Division Multiple Access)-2000, W-CDMA(W-Code Division Multiple Access), Wibro(Wireless Broadband Internet) ) terminal, a smart phone, a smart pad, a tablet PC, etc. may include all kinds of handheld-based wireless communication devices.

The guide service providing server 300 may be a server that provides a billiard guide service web page, an app page, a program, or an application using reinforcement learning of a deep learning model. In addition, the guide service providing server 300 may be a server that builds an environment for reproducing a simulation of a pool table ball using at least one type of open source. In addition, the guide service providing server 300 initially sets the target and the ball at random based on the deep learning model for training, and then expresses the position coordinates of the target and the ball as a vector, the coordinates of our point in the target, our point It may be a server that conducts training using reinforcement learning without a dataset by simulating the trajectory of the ball according to the force and direction that strikes it and giving a reward according to the result. In addition, the guide service providing server 300 identifies the target and the ball on the captured screen when the location of the target and the ball is designated or photographed in the user terminal 100, and according to the target's point, force and direction After simulating the predicted path of each ball may be a server that transmits to the user terminal (100). At this time, the guide service providing server 300, when the camera 400 is shooting the billiard table in the vertical direction, since the billiard table can be photographed in the same shape as a plan view, using this to extract the positions of the target and the ball It may be a server that outputs a simulation result according to the number of , to the display 500 . If the AR glasses are interlocked with the user terminal 100 or the like, the guide service providing server 300 may be a server that guides by overlaying the trajectories or coordinates of the shop according to the number of each case on the AR glasses. In addition, the guide service providing server 300, even if the size of the force is physically calculated, the user has to apply the force to the cue table by the amount of the force, but at least one device (not shown) time), comparing the magnitude of the force to be applied to the shop with the magnitude of the force applied to the device, and then providing an increase/decrease indication to the user so that the magnitude of the force can be estimated.

Here, the guide service providing server 300 may be implemented as a computer capable of accessing a remote server or terminal through a network. Here, the computer may include, for example, a navigation device, a laptop computer equipped with a web browser, a desktop computer, and a laptop computer.

The at least one camera 400 is a device for transmitting, in real time, an image captured using a web page, an app page, a program or an application related to a billiard guide service using reinforcement learning of a deep learning model to the guide service providing server 300 can be At this time, the camera 400 may be provided as an intelligent CCTV, and in this case, after identifying the target and the ball as the subject, the trajectory may be tracked, and the result may be provided to the guide service providing server 300 .

Here, the at least one camera 400 may be implemented as a computer that can connect to a remote server or terminal through a network. Here, the computer may include, for example, a navigation device, a laptop computer equipped with a web browser, a desktop computer, and a laptop computer. In this case, the at least one camera 400 may be implemented as a terminal capable of accessing a remote server or terminal through a network. The at least one camera 400 is, for example, a wireless communication device that ensures portability and mobility, such as navigation, Personal Communication System (PCS), Global System for Mobile communications (GSM), Personal Digital Cellular (PDC), and PHS. (Personal Handyphone System), PDA (Personal Digital Assistant), IMT (International Mobile Telecommunication)-2000, CDMA (Code Division Multiple Access)-2000, W-CDMA (W-Code Division Multiple Access), Wibro (Wireless Broadband Internet) It may include all kinds of handheld-based wireless communication devices such as terminals, smartphones, smartpads, and tablet PCs.

The at least one display 500 outputs a screen containing the trajectory transmitted from the guide service providing server 300 using a billiard guide service related web page, app page, program or application using reinforcement learning of the deep learning model. It may be a device.

Here, the at least one display 500 may be implemented as a computer that can access a remote server or terminal through a network. Here, the computer may include, for example, a navigation device, a laptop computer equipped with a web browser, a desktop computer, and a laptop computer. In this case, the at least one display 500 may be implemented as a terminal capable of accessing a remote server or terminal through a network. The at least one display 500 is, for example, as a wireless communication device that ensures portability and mobility, navigation, Personal Communication System (PCS), Global System for Mobile communications (GSM), Personal Digital Cellular (PDC), PHS (Personal Handyphone System), PDA (Personal Digital Assistant), IMT (International Mobile Telecommunication)-2000, CDMA (Code Division Multiple Access)-2000, W-CDMA (W-Code Division Multiple Access), Wibro (Wireless Broadband Internet) It may include all kinds of handheld-based wireless communication devices such as terminals, smartphones, smartpads, and tablet PCs.

2 is a block diagram for explaining a guide service providing server included in the system of FIG. 1, and FIGS. 3 and 4 are billiard guide services using reinforcement learning of a deep learning model according to an embodiment of the present invention. It is a diagram for explaining an embodiment.

Referring to FIG. 2 , the guide service providing server 300 may include a construction unit 310 , an initialization unit 320 , a training unit 330 , a transmission unit 340 , and a determination unit 350 .

The guide service providing server 300 according to an embodiment of the present invention or another server (not shown) operating in conjunction with at least one user terminal 100 , at least one camera 400 , and at least one display 500 . ) to transmit a billiard guide service application, program, app page, web page, etc. using reinforcement learning of a deep learning model, at least one user terminal 100, at least one camera 400, and at least one display ( 500) may install or open a billiard guide service application, program, app page, web page, etc. using reinforcement learning of a deep learning model. In addition, a service program may be driven in at least one user terminal 100 , at least one camera 400 , and at least one display 500 using a script executed in a web browser. Here, the web browser is a program that enables the use of a web (WWW: World Wide Web) service, and refers to a program that receives and displays hypertext written in HTML (Hyper Text Mark-up Language), for example, Netscape. , Explorer, Chrome, and the like. In addition, the application means an application on the terminal, for example, includes an app (App) executed in a mobile terminal (smartphone).

Referring to FIG. 2 , the building unit 310 may construct a billiards simulation environment using at least one open source for constructing a physics engine-based simulation environment. The collision of an object may appear in various ways depending on the shape or strength of the object. According to the change in kinetic energy of an object after collision, it can be divided into elastic collision and inelastic collision. At this time, in the case of a billiard ball, since it is an elastic collision of an unbreakable circular object, this is taken as a reference. Here, three types of collision operation can be used. First, when two objects collide during a collision between objects, a force acts on the colliding surface. Therefore, by calculating the force acting on the object, the velocity after the collision can be found. When two objects collide elastically in a two-dimensional space, a new speed can be obtained after setting a new reference axis perpendicular to the collision surface, calculating a one-dimensional elastic collision with respect to the reference axis, and this is shown in Equation 1. Equation 1 is a speed calculation formula at the time of a one-dimensional elastic collision.

The second is the collision of an object and a line. When an elastically collided object collides with a wall, the direction of velocity with respect to the colliding surface is reversed. That is, the angle of incidence and the angle of reflection with respect to the collision surface are the same. The third is gravity simulation. In billiards, when an object crosses a railing, it receives a force in the direction of the railing due to gravity, and this force can be calculated. To calculate the new velocity, we need to calculate the velocity caused by gravity. When extending from a two-dimensional space to a three-dimensional space to consider gravity, the xy-axis becomes a two-dimensional plane and the z-axis becomes the depth. The force acting tangentially does not affect the speed of the object and is therefore neglected. Therefore, the force parallel to the tangent must be considered. Since an object moves in a three-dimensional space, the force must be converted into a three-dimensional force, and a new velocity can be calculated by calculating the acceleration from the calculated force.

An environment is required for reinforcement learning to proceed. Simulators such as Gym, MuJoCo, Pygame, Unity, and Unreal Engine can be used to build this environment. At this time, among the simulators, Unity currently supports the Unity ML-Agents Toolkit, so it is easy to apply the reinforcement learning algorithm to the environment, and because it provides parallel processing learning by default, the acceleration and stabilization of learning can be guaranteed. Alternatively, Unreal Engine can be used, which is based on C++, all source codes are open to GitHub, and Nvidia's PhysX 3.3 is used as a physics engine. If you use it, you can experiment with using image recognition algorithms using Photorealistic and UnrealCV. The latest Unreal Engine also comes with real-time ray tracing capabilities. Second, interfaces for various devices such as VR devices and joysticks, as well as keyboards and mice, are built-in, making it easy to map and control complex inputs. Third, it is possible to implement selective collision by setting the collision level, and it has a built-in function to automatically generate a collision model as a k-DOP (Discrete Oriented Polytope) suitable for a physics engine. In addition to this, various settings such as simple collision and complex collision are possible. Fourth, it can be linked with ROS through a plug-in. However, since code integration based on Unreal Engine uses C++ with many Unreal macros, programming with Unreal-style coding standards should take precedence over standard C++ style. Otherwise, it may cause unpredictable function behavior or memory leak problems.

The initialization unit 320 may initially set the coordinates of the target and the at least one ball at random in the established billiard simulation environment, and then set the coordinates of the target and the at least one ball as a position vector. As shown in (a) and (d) of Figure 3, the initialization unit 320, the randomly set position of the target (xt, yt, zt), the position of the first ball (xball1, yball1, zball1), the second The position of the ball can be expressed as a vector as (xball2, yball2, zball2) and used as observation information. Here, in the case of three balls, the number of at least one ball may be two.

The training unit 330 repeats the simulation of drawing the trajectory of the target and at least one ball according to the target within the target and the force that hits the point, and then uses reinforcement learning with at least one deep learning model to provide a reward according to the result This will allow you to proceed with the training. In this case, reinforcement learning may use the MODEL-BASED method or the MODEL-FREE method. Referring to Figure 3b, the MODEL-BASED algorithms learn the dynamics of the environment itself and select the optimal behavior, that is, indirectly acquire a policy or a value function. , it is a method that directly approximates a policy or value function and selects the optimal behavior through it. At this time, in the case of PPO, which will be described later, the MODEL-FREE method is used, and the MCTS is one of the methods for obtaining policy functions indirectly in the MODEL-BASED method. However, MCTS itself is not a MODEL-BASED algorithm. Reinforcement learning according to an embodiment of the present invention may be the MuZero algorithm, which is a method that takes only the advantages of the MODEL-BASED and MODEL-FREE methods. In this way, while learning the dynamics of the environment itself, the policy function and the value function are also directly approximated internally. At this time, the MuZero algorithm is the next version of the AlphaZero algorithm and is known as a MODEL-BASED method, but in fact, it cannot be defined as belonging to any one classification. Here, the PPO will be described later in (c) of FIG. 3 .

The training unit 330 recognizes the current state in the environment in which the billiard is placed without a dataset for training and testing through reinforcement learning and maximizes the reward through action. learning can proceed. The training unit 330, as shown in (a) of FIG. 4 , may express the point, which is the striking part on the surface of the target, as a vector as (x1, y1, z1) based on the center point of the target. The training unit 330 may express the force to hit the point and the direction to hit the point as f(Force) and (x2, y2, z2) as vectors, respectively. Training unit 330, (xt, yt, zt), (xball1, yball1, zball1) and (xball2, yball2, zball2) included in nine consecutive values as input, (x1, y1, z1), A deep learning model that outputs seven consecutive values included in (x2, y2, z2) and f can be trained by reinforcement learning through repetition of the simulation.

The transmission unit 340, when recognizing or designating the coordinates of the target and at least one ball in the user terminal 100, the simulation result according to the number of at least one case according to the target point and the force to hit the point It can be transmitted to the user terminal 100 . After recognizing the coordinates of the target and at least one ball as a position vector, the user terminal 100 simulates the trajectory of the target and at least one ball according to the force hitting the point and the point that is the hitting part on the surface of the target. You can print the results.

The billiard guide service providing system 1 using reinforcement learning of a deep learning model according to an embodiment of the present invention is installed in a direction opposite to the billiard table so that the user of the user terminal 100 illuminates the billiard table to play billiard. It may further include one camera 400 and at least one display 500 that overlays the simulation result output from the transmitter in the image of the billiard table photographed by the at least one camera 400 and outputs it. The user's head posture obtained by analyzing the image input through the camera 400 to calculate the relative posture of the billiard table and the billiard ball with respect to the camera, and augmented reality equipment, for example, AR glasses such as Google Glass (camera position and rotation) to convert it to an absolute pose. When the positions of the billiard table and billiard balls are identified, a physics simulation can be performed based on them. This can be done in such a way that the player conducts a hitting simulation at various speeds in all 360-degree directions of the seven ball and then grasps all the scoring paths. The visualization function can visualize the most reasonable path according to the user's head position and path weight using the Unity engine's various particle systems. The above three processes can be performed in real time while the system is running.

<Image processing>

It is assumed that the felt of the pool table is a color that is distinctly different from other objects on the image. If this is converted into HSV color space, Hue, Saturation, and Value, and evaluated, the felt surface of a pool table with a uniform color will have stable color and saturation value. Just by picking out all the pixels of brightness, you can find the area of the pool table quite accurately. In this case, you can use inRange(), a function provided by the OpenCV library to find all pixels with a value in each channel within a certain range, to set all pixels with a color value and saturation value to be True.

From the boundary image calculated by applying the erosion operation to the filtering result and subtracting it from the original, all figures existing in the image can be found. Each figure may be composed of a list of coordinates of vertices constituting the corresponding figure. A function called findContours() is provided in the OpenCV library. Since the algorithm of the OpenCV function evaluates whether a straight line between vertices is very strict, a rectangular shape is evaluated as a much larger number of vertices than four. Therefore, it is necessary to approximate the number of vertices constituting the figure to the four vertices with the most distinct characteristics. Here, by applying the approxPolyDP() function of OpenCV, which implements the Ramer-Douglas-Peucker linear approximation algorithm, as an argument, countless vertices can be approximated as four vertices. After that, even if the pool table area is invaded by other objects such as billiard cue tables, OpenCV's convexHull() function is applied so that the pool table area can be detected as long as the four vertices of the billiard table are in the screen to make all the pool table areas into convex polygons.

Through the above process, a set of figure vertices for all areas having the same color as the pool table on the image is obtained. At this time, it is self-evident that the number of vertices of the pool table is 4, and if the user is looking at the pool table, the size of the pool table area also occupies a certain amount on the screen. Therefore, it traverses each figure to find a figure that meets the condition, and assumes that it is the four vertices of the pool table. Since the billiard table is a rigid body that does not deform, and the dimensions of the billiard table are already known, it is assumed that an object with the same dimensions as the billiard table is at the origin (0, 0, 0). In this case, the origin means the position of the camera. The algorithm that finds the posture (position and rotation) of an object by comparing each vertex of the 3D model located in the model space with the coordinates projected on the screen is called the PNP (Perspective N-Point Algorithm) algorithm. In OpenCV, a function called solvePnP() implement the algorithm in However, the OpenCV implementation assumes that the order of the 2D vertices detected on the screen matches the order of the 3D vertices in the model space. The order may be different. Therefore, the PNP algorithm is first applied, the index list is rotated once regardless of the direction, and the PNP algorithm is applied again, and the candidate with less error is selected as the final posture of the pool table.

Since the posture of the billiard table obtained through this is relative to the camera, the absolute posture of the camera returned by the position tracking function of AR glasses is applied to this to calculate the absolute posture of the billiard table. Even if the posture of the pool table cannot be estimated in the video, the camera's current posture is still updated through the position tracking function of the AR glasses, so that tracking can be maintained based on the stored absolute posture of the pool table. For this reason, additional recognition of the pool table after the initial recognition is made in the dimension of error correction of the AR glasses position tracker. Once the posture of the billiard table is obtained, the 3D midpoint of the billiard ball can be calculated by projecting a ray from the camera in the direction of the 2D midpoint of the billiard ball detected on the screen to collide with the billiard table plane. Since all the billiard balls are located on one plane, if only the UV position of the center point on the image of the billiard ball can be accurately obtained, the center position of the billiard ball can be obtained. The real problem is to find the central position of the billiard ball, and unlike the easy way to find the billiard ball with the naked eye, it becomes difficult to mechanically identify it due to the influence of the location of lighting and ambient light.

Accordingly, a balance can be found between the accuracy of the central search and the performance optimization through the modified template matching. In the estimated center 2D coordinates of the ball, template matching is performed for color with a kernel having a pixel radius on the image of the ball at the corresponding position, and only the color and luminance are compared with the template except for brightness. However, it is not possible to perform an operation with the same kernel over the entire image like normal template matching, because the radius of the kernel must change dynamically according to the actual distance to the ball. As the kernel size changes dynamically, each kernel operation must be applied to all candidates that can be the center of the billiard ball, which causes performance loss even if the power of GPU acceleration provided by OpenCV is borrowed.

Therefore, performance optimization can be achieved by introducing the concept of a sparse template kernel and performing template matching by sampling only about 5 to 10% of the total pixels. Since the sparse template kernel performs a simple weight sum operation, it is necessary to create a weight field to which the kernel is applied. This is to obtain the error of each channel by subtracting the representative value of the color-luminance of each ball from the color-luminance color space image. A distributional fit is obtained. The above fitness calculation for each pixel is expressed as Equations 2 to 4 below.

In this case, hn and sn in Equation 2 are the color and luminance values of each pixel, hr and sr represent the representative color and luminance values of the ball to be detected, and wh and ws in Equation 3 are the colors when calculating the Euclidean distance of error. and weight to be applied to each of the luminance component errors. β in Equation 4 is a coefficient that determines the influence of an error, and the higher the value, the more sensitive the error becomes, and the higher the accuracy, but the weaker it is to environmental influences such as ambient light or illuminance. Each channel is calculated in 32-bit floating point form, and for this purpose, it is normalized to a value between 0 and 1, so the distance d has a very small value in practice. To overcome this, β can be assigned a sufficiently large value. After obtaining the fitness field for each color, the midpoint of the ball is estimated by multiplying the above field by the sparse template kernel. However, since the area where the ball can exist corresponds to a very small area even in the ROI of the table, it is inefficient to apply the expensive template operation to the entire image. Therefore, the template operation can be applied only to the index by first binarizing the image using the fixed fitness value as a threshold and extracting the index.

In this case, if the fitness of the ball is simply binarized, the field is highly likely to be formed only at the boundary line, not the actual center of the ball, depending on the light source situation. Therefore, after performing binarization first, the expansion-erosion operation is repeatedly applied so that the center of the billiard ball is included in the matching field. For each point in the matching field, the kernel is applied to the goodness-of-fit field to compute the sum of goodness-of-fits for each point. Conceptually, the more pixels of the target color within the radius and the fewer pixels of the target color outside the radius, the better the fit. After that, if the point with the maximum value is selected in the fitness sum field, this becomes the 2D estimated midpoint of each ball. At this time, if there are two balls of the same color, for example, if there are two red balls, the above process is performed once more. Since the already detected fit sum field is meaningless, it is erased with the radius of the ball. . By projecting a ray from the origin to the center position of the ball calculated in this way, the 3D center coordinates of the ball with respect to the camera can be obtained, which is multiplied by the camera attitude matrix and converted into an absolute position.

Alternatively, in order to recognize a billiard ball, the Hough Transform is a method of extracting geometric components from a pixel-based image after undergoing a Gaussian blurring process to remove noise generated by normal and probability distributions. ) is used to detect a circle, and since the Hough transform function in OpenCV receives a gray scale image itself as an input, it internally performs a transformation through a Sobel operation that detects the amount of change based on the center point after matrix operation. In addition, cue recognition converts the image data received from the camera into a black-and-white image and then converts it into a binary image through the image binarization process. Through the labeling process of grouping adjacent areas with the same value from the processed image, the position of the cue table is recognized and the center point is detected. Then, connect the end point and the center point of the cue bar to make a straight line, and draw a straight trajectory through the connected straight line. When the trajectory touches the outline of the set pool table area, it is also possible to use the method of reflecting the size of the incident angle using the law of reflection.

<path simulation>

The detected billiard table and the location of each billiard ball are transmitted to the AR billiards application implemented with Unity Engine. If you know the absolute positions of the pool table and all four balls, you can simulate the path of the ball hit by the player through physics simulation and calculate the case where a goal is scored. In general, physics simulations are controlled by a discrete method called time division, or time step. For example, when the acceleration of the object at time n is a → n and the simulation interval is Δt, the next velocity v → n+1 and the position s → n+1 in each simulation step can be calculated as in Equation 5 below. .

Equation 5 is calculated by simplifying the integral operation for calculating the speed change of an object for a unit time in a system that performs a physical simulation discretely for each unit time Δt as a product of the unit time for the acceleration, and accumulate Equation 6 simplifies the integral operation for the velocity for calculating the position change in the same context.

The above method used in discrete time division physics simulation has the advantage of being free to introduce complex physics equations and ignition equations by simplifying the integration. Also, by adjusting the simulation interval Δt, the accuracy and optimization level of the simulation can be controlled. However, in the time division simulation, the amount of computation increases linearly according to the length of the entire simulation and the level of accuracy. This is somewhat unsuitable for reinforcement learning, which requires simulating the hitting of a ball in a 360-degree direction and over thousands of times at various speeds to find a path.

Accordingly, in an embodiment of the present invention, an event-based physics simulation function, not time division, may be used. The event-based model is a model that omits the entire simulation for a collision-free period, which takes up most of the computation in the time-division-based simulation. Unlike a typical time-division simulation that calculates the distance traveled by all objects for a time of Δt at every simulation step, and then examines the collision of all objects, the event-based model calculates all objects over a very long time interval based on the spherical trajectory intersection. After examining the collision of an object, first, the movement of all objects is simulated for the time at which the collision occurs, and then the collision physics of the object that collided at that point is calculated. That is, the event-based collision operation executes collision detection with a complexity of O(n2) only as many times as the number of collisions occurs during the simulation period, which has many advantages in optimization compared to the time division method.

Of course, if the velocity is not a constant or natural damping (v(t)=V0e-αt), the complexity of the trajectory inspection is greatly increased. Since no external force acts except for the initial velocity, the introduction of event-based simulation may be appropriate. After first checking for collision through spherical trajectory inspection, the actual collision physics reaction must be calculated. Instead of precisely realizing the physics acting on the billiard ball and the cushion, the model directly specifies the elastic modulus between the ball and the ball and between the ball and the cushion, and implements the rotation applied to the ball when the ball and cushion collide through an empirical heuristic. . In order to calculate the collision applied between the ball and the ball and between the ball and the cushion, the following Equations 7 to 9 may be applied.

Pct in Equation 7 is a collision point between two objects, and V1' in Equation 9 is the velocity of object 1 after collision. The same formula applies to the other collided object, V2. If the ball collides with the cushion, m2 is set to infinity and V1p'=-εV1p is calculated. ε is 0.77 for the ball-to-ball collision and 0.73 for the ball-to-cushion collision. The effect of rotation is also considered to some extent. In general, the minimum rolling start time troll is set by simulating that the billiard ball travels and the rolling direction coincides after a certain period of time after the collision. Linear interpolation is performed between the current rotational speed w→n and the maximum rotational speed w→n(max) to match the moving direction as shown in Equations 10 to 13 below.

α in Equation 10 indicates the degree of coincidence between the linear speed and the rotation speed of the ball object, and is simply expressed as the ratio of Δt to the constant troll. In this case, the maximum value of α is limited to 1. If the billiard ball rotates without slipping in the direction of travel, the velocity of the contact point between the ball and the ground is in the opposite direction to the linear velocity of the ball. At this time, the angular velocity vector is calculated by dividing the cross product of the opposite direction of the velocity and the normal (upward direction vector) of the contact plane by the radius r, which is shown in Equation 11. (hat)uup in Equation 12 is the upward direction vector, and r is the radius of the ball. w→n+1 in Equation 13 becomes the rotational speed at the collision time tn+1. When a collision occurs between a ball and a ball, the velocity v0 calculated by the collision equation based on the above elastic modulus is primarily manipulated in consideration of rotation. In this case, the reflection ratio of rotation is directly specified as the friction constant fs(ball) =0.23. This process is the same as Equation 14 below and is applied to the two colliding balls, respectively.

By multiplying the angular velocity w→n+1 with the normal vector (hat)uup of the ground and multiplying by the radius r, the angular velocity of the ball can be expressed as the linear velocity acting on the surface of the pool table. In Equation 14, only a portion of this is reflected in the final linear velocity of the billiard ball object through the friction constant fs(ball) and accumulated. The collision between the ball and the cushion is calculated more complexly. Similarly to Equation 14, v0 is calculated, and the frictional force generated by the rotation of the ball at the contact point between the cushion plane and the ball is simulated. It is equal to the velocity in the external direction of the normal (hat)n of the plane of the cushion in contact with the ball's rotation velocity vector w→. However, since gravity is not taken into account, the vertical velocity is removed, and the normal velocity can also be removed. This is equivalent to scaling the velocity vector of the junction as a result of the cross product of the up vector and the normal vector. In this case, a friction constant fs(cushion)=0.67 may be applied to determine how much rotation is reflected in the speed, which is expressed in Equations 15 and 16.

Product by element (◎ sign) of Equation 15 The equation on the left calculates the linear velocity expression of the angular velocity acting on the ground as in Equation 14, and adjusts the application rate with the friction constant fs (cushion) of the billiard table cushion . For the linear velocity component calculated in this way, only the velocity acting in the tangential direction remains by performing the element-by-element product of the equation (hat)uup×(hat)n on the right, which is the horizontal vector of the billiard table cushion. In Equation 16, the tangential velocity calculated in Equation 15 is accumulated in the final linear velocity of the billiard ball after the collision. So that the collision also affects the rotation of the ball, the velocity vector v→0 in the direction of friction is calculated by subtracting the velocity vector v→0 from the velocity vector v→p in the direction of the cushion, and the angular velocity is obtained by cross product with the normal of the cushion. After calculating the vector, it is accumulated in the angular velocity by the ratio of the constant c=0.2. Equation 17 below expresses this as an equation.

A simulation engine is implemented based on the above heuristic, and each time the arrangement of the billiard ball is changed, the 360-degree angle of the ball is divided by 0.5°, and then for starting speeds 0.15, 0.3, 0.6, and 0.9 [m/s], respectively. simulation can be performed. Among the simulation results for all paths, the scoring path is primarily acquired, and the list of optimal paths is filtered by considering various factors, such as the collision angle with the scoring construction and movement due to external factors before the collision. From this list, the path closest to the direction of the user's head is selected and visualized.

In this way, using AR glasses, you can feel interested in the visual effects applied to the billiard ball itself, and even if you have never played billiards, you can score points based on the route presented by the application and get great psychological rewards and motivation. As collision calculation is based on heuristics, it does not show ideal accuracy in all situations, but in many cases, this is an error of a level that is difficult to recognize by a billiard beginner who is difficult to meet the conditions suggested by the algorithm. It can be implemented to an acceptable level. Above all, when AR glasses are used, the scope of visualization is expanded to three-dimensional, so it has a significant advantage in terms of immersion, and can be expected to have a superior effect than simply learning video media through flat media. Therefore, it can be used as a new learning platform.

The determination unit 350 is, when the user terminal 100 selects any one simulation result from among the simulation results according to the number of at least one case transmitted from the transmission unit 340 , any one selected by the user terminal 100 . After receiving the pressure from the user of the user terminal 100 to the device for measuring the force for realizing the simulation result of

In addition, an embodiment of the present invention may develop a deep learning model trained by reinforcement learning as an artificial intelligence to face the player. Various problems in the reality of playing billiards require the ability of planning to select actions one after another in a continuous action space. Monte-Carlo Tree Search (MCTS) is an efficient online planning algorithm for solving problems with large search spaces, and has been very successfully applied to cases where the action space is discrete, such as Go and real-time games . However, unlike in the discrete action space, MCTS was not a top-priority planning algorithm for problems in which the action space was continuous. This is because the process of coarsely discretizing the action space for tree search is necessary because the number of selectable actions is infinite, and as a result, it can lead to ineffectiveness in real problems requiring very precise manipulation.

MCTS algorithms that deal with the existing continuous action space take two kinds of approaches. In progressive widening, the number of selectable actions of each node in the tree is gradually increased based on the number of visits to the node. Another approach, Hierarchical Optimistic Optimization, progressively partitions the action space to increase the precision of the action to be selected. Although these approaches can show sufficient performance for selecting moderately good behaviors, a huge amount of simulation is required to obtain very precise control as a planned result. On the other hand, gradient-based optimization is very effective in finding an accurate local optimum, but it is very difficult to reach the global optimum. Accordingly, in an embodiment of the present invention, a method of combining MCTS and gradient-based behavior fine-tuning may be used. This algorithm can perform both the global search through MCTS and the local search based on the behavioral value gradient, and is based on the observation that various real-world problems with continuous action spaces have locally differentiable environmental dynamics with respect to states and behaviors. It is based on the assumption that, in this case, the optimization based on gradient rise can be very effective.

Hereinafter, an operation process according to the configuration of the guide service providing server of FIG. 2 will be described in detail with reference to FIGS. 3 and 4 as an example. However, it will be apparent that the embodiment is only one of various embodiments of the present invention and is not limited thereto.

Referring to FIG. 3 , (a) after setting the vector positions of each ball and target, (b) building a simulation environment to start reinforcement learning. At this time, reinforcement learning, an area of machine learning, is an action sequence or action that maximizes a reward among actions that an agent defined for use in a specific environment can select after recognizing the current state. How to choose (Action). Based on this, it is possible to solve complex problems by simultaneously performing path planning by controlling the actual hardware with reinforcement learning through deep reinforcement learning.

<Value-based deep reinforcement learning>

The purpose of reinforcement learning is to obtain high rewards while interacting with the environment. Here, there are two functions that define the reward. One of these is the value function, which defines the expected value of the expected reward. This value function v(s) is composed of the concept that the agent obtains the state data s, that is, the observed data at time t, and calculates the sum of the series of rewards received at that time. Here, by multiplying the depreciation rate by the past data, which becomes less important, it can be expressed as Equations 18 and 19.

However, the value function is expressed as a state-value function because the behavior directed to the state is not considered, and the expected value obtained by considering the behavior a is called a Q-function (Quality function), so Equation 20 and can be expressed together.

The Q-function is a function that expects a high reward by using it as a state s and an action a as in Equation 20. And the Q-function receives the next state s′ and action a′ and is continuously updated through the Bellman optimization equation to obtain a high reward. An algorithm for replacing such a value function with a neural network based on the neural network parameter θ as shown in Equation 21 is called a Deep-Q-Network (DQN), and as shown in Equation 21, a cost function with a target Q-function ) and update the neural network by combining the off-policy method based on this.

<Policy-based deep reinforcement learning>

In policy-based deep reinforcement learning, a policy is expressed as πθ(a|s), and it is a function that receives a state, performs an operation based on the policy neural network parameter θ, and emits an action. In other words, policy-based reinforcement learning selects actions according to states. Accordingly, the parameter θ is updated based on the change in the value function that occurs, and the update direction is directed toward the gradient ascent to obtain a higher value function. This process is called PG (Policy Gradient) and is expressed as Equation 22 below.

∇θJ(θ) is a target function, and is defined as Equation 23 in PG.

Here, in order to reduce the bias, it is replaced with Equation 24 by changing the Advantage Function by subtracting the baseline V(s), that is, A(s,a) = Q(s,a)-V(s), and this is ) is called PG.

The preceding PG is also called SPG (Stochastic Policy Gradient), and the algorithm configured to calculate this value as a continuous action value is called DPG. It becomes DDPG, which is an Off-Policy, Actor-Critic algorithm, and becomes D4PG when the concept of Distributional is additionally reflected here. Actor-Critic helps mutual update when the policy and value functions in the algorithm are called Actor and Critic, respectively. This is because the value of the value function increases as the policy is updated, and the policy is also updated in a better direction based on the increased value of the value function.

<TPRO & PPO>

Trust Region Policy Optimization (TPRO) and Proximal Policy Optimization (PPO) are algorithms that enable stable neural network updates by creating trust regions, that is, limiting the scope, and preventing problems from excessive policy updates occurring in the PG. In TRPO and PPO, like the existing policy-based reinforcement learning, the policy neural network πθ(a|s) interacts with the environment to collect data in memory. In the corresponding algorithm, after collecting compensation values for each timestep, the loss for updating the neural network is expressed as Equation 25 by calculating A (advantage) determined by the algorithm and using the value multiplied by Importance Sampling.

(hat) A is a term related to the predicted reward, and the Q-V value can be used as A. In this case, the Surrogate Function for limiting the range of Loss can be expressed as Equation 26 by using KL-Divergency in TRPO, which is the predecessor of PPO.

Unlike the above formula expressed in TPRO, the PPO algorithm, which simplifies the surrogate function and exhibits similar performance to TRPO, can limit the limiting range in PPO to a constant ε by rt(θ) in Equation 27, and calculate Ldip accordingly It can be expressed as Equation 28.

After that, like updating a general neural network, the data accumulated in the memory is cut by a batch, and the designated data is updated K times in order using Loss, and the process from the top to the present is continuously repeated. To execute the training code based on the above algorithm, it is necessary to specify constants such as the limiting range ε of the PPO. After that, the compensation is optimized by repeating the process of updating the neural network and collecting data using the algorithm and loss using the above constants. In the training code, the Actorcritic-based PPO algorithm as above is used. In this algorithm, when learning, two updates of Actor and Critic neural networks are performed. In general, the computation of a neural network requires high computational power, but in order to lighten and update the neural network, it receives the specified number of state data from the State, and performs light operation through the nodes in order by alternating Linear Regression and Tanh Activation Function. You may. In addition, since there is no Fully Connected Layer used in general neural networks, data operation in the training and testing process can be reduced to two.

Unity provides ML-Agent (Machine Learning-Agent), a service that uses its own software as an environment for reinforcement learning and enables learning of neural networks in Python. Through this, the environment test of the code composed in Python can be performed in the Unity simulator. The reinforcement learning code calculates the observed state data and uses the behavior, that is, the control data, to the simulator. In other words, artificial intelligence replaces human input. These results are shared with the user as shown in (b), and the user can know which strength to hit and in which direction with what amount of force.

The matters that have not been described with respect to the method of providing the billiard guide service using reinforcement learning of the deep learning model of FIGS. 2 to 4 are described above with respect to the method of providing the billiard guide service using the reinforcement learning of the deep learning model through FIG. 1. Since it is the same as the content or can be easily inferred from the described content, the following description will be omitted.

5 is a diagram illustrating a process in which data is transmitted/received between components included in the system for providing a billiard guide service using reinforcement learning of the deep learning model of FIG. 1 according to an embodiment of the present invention. Hereinafter, an example of a process in which data is transmitted and received between each component will be described with reference to FIG. 5, but the present application is not limited to such an embodiment, and the example shown in FIG. 5 according to the various embodiments described above will be described. It is apparent to those skilled in the art that the data transmission/reception process may be changed.

Referring to FIG. 5 , the guide service providing server builds a billiards simulation environment using at least one open source for building a physics engine-based simulation environment ( S5100 ).

Then, the guide service providing server randomly initially sets the coordinates of the target and the at least one ball in the established billiard simulation environment, and then sets the coordinates of the target and the at least one ball as a position vector ( S5200 ).

In addition, the guide service providing server repeats the simulation of drawing the trajectory of the target and at least one ball according to the force hitting the point in the target and the point in the target, and then uses reinforcement learning with at least one deep learning model to compensate according to the result. By providing the training proceeds (S5300).

Finally, the guide service providing server, when recognizing or designating the coordinates of the target and at least one ball in the user terminal, provides a simulation result according to the number of at least one case according to the force hitting the point of the target and the point of the target. It is transmitted to the terminal (S5400).

The order between the above-described steps (S5100 to S5400) is only an example and is not limited thereto. That is, the order between the above-described steps S5100 to S5400 may be mutually changed, and some of these steps may be simultaneously executed or deleted.

Matters that have not been described with respect to the method of providing a billiard guide service using reinforcement learning of the deep learning model of FIG. Since it is the same as the content or can be easily inferred from the described content, the following description will be omitted.

The method of providing a billiard guide service using reinforcement learning of a deep learning model according to an embodiment described with reference to FIG. 5 is in the form of a recording medium including instructions executable by a computer, such as an application or program module executed by a computer can also be implemented. Computer-readable media can be any available media that can be accessed by a computer and includes both volatile and nonvolatile media, removable and non-removable media. Also, computer-readable media may include all computer storage media. Computer storage media includes both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data.

The method for providing a billiard guide service using reinforcement learning of a deep learning model according to an embodiment of the present invention described above includes an application basically installed in a terminal (which may include a program included in a platform or an operating system basically installed in the terminal) ), and may be executed by an application (ie, a program) installed directly on the master terminal by a user through an application providing server such as an application store server, an application, or a web server related to the corresponding service. In this sense, the method for providing a billiard guide service using reinforcement learning of a deep learning model according to an embodiment of the present invention described above is implemented as an application (that is, a program) installed by default in a terminal or directly installed by a user and installed in the terminal. It may be recorded on a computer-readable recording medium, such as.

The above description of the present invention is for illustration, and those of ordinary skill in the art to which the present invention pertains can understand that it can be easily modified into other specific forms without changing the technical spirit or essential features of the present invention. will be. Therefore, it should be understood that the embodiments described above are illustrative in all respects and not restrictive. For example, each component described as a single type may be implemented in a dispersed form, and likewise components described as distributed may be implemented in a combined form.

The scope of the present invention is indicated by the following claims rather than the above detailed description, and all changes or modifications derived from the meaning and scope of the claims and their equivalent concepts should be interpreted as being included in the scope of the present invention. do.

Claims (10)

After recognizing the coordinates of the target and the at least one ball as a position vector, the result of simulating the trajectory of the target and the at least one ball according to the hitting part on the surface of the target and the force hitting the point is output a user terminal to;
A building unit that builds a billiard simulation environment using at least one open source for building a physics engine-based simulation environment, after initially setting the coordinates of a target and at least one ball at random in the built billiard simulation environment, After repeating the simulation of drawing the trajectory of the target and the at least one ball according to the initialization unit setting the coordinates of the target and the at least one ball as a position vector, the point in the target, and the force hitting the point, at least one A training unit that conducts training by providing a reward according to the result using reinforcement learning as a deep learning model, when recognizing or designating the coordinates of the target and at least one ball in the user terminal, the target of the target and the A transmission unit that transmits a simulation result according to the number of at least one case according to the hitting force to the user terminal, and a simulation result of any one of a simulation result according to the number of at least one case transmitted from the transmission unit to the user When selected in the terminal, after receiving a pressure from the user of the user terminal to a device for measuring a force for realizing any one simulation result selected by the user terminal, the force according to the pressure and the force for realizing the simulation result A guide service providing server including a determination unit for determining whether the same or similar; and
AR glasses that work with the user terminal and the guide service providing server;
includes,
The training department
Through the reinforcement learning, learning is carried out in a way that recognizes the current state in the environment in which the billiard is placed and maximizes the reward through action without a dataset for training and testing. and
The guide service providing server,
Overlaying the trajectory or coordinates of our shop according to the number of cases included in the simulation result on the AR glass,
The device for measuring the force is
Reinforcement learning of a deep learning model, characterized in that after comparing the magnitude of the force to be applied to the point and the magnitude of the force applied to the device that measures the force, an increase/decrease indication is provided so that the user can estimate the magnitude of the force Billiard guide service provision system using
The method of claim 1,
The reinforcement learning is a billiard guide service providing system using reinforcement learning of a deep learning model, characterized in that it uses a MODEL-BASED method or a MODEL-FREE method.
The method of claim 1,
the number of said at least one ball is two;
The initialization unit,
Observation information by expressing the randomly set position of the target as (xt, yt, zt), the position of the first ball as (xball1, yball1, zball1), and the position of the second ball as (xball2, yball2, zball2) as vectors Billiard guide service providing system using reinforcement learning of deep learning model, characterized in that it is used as
4. The method of claim 3,
The training department
Billiard guide service providing system using reinforcement learning of a deep learning model, characterized in that the point, which is the hitting part on the surface of the target, is expressed as a vector as (x1, y1, z1) based on the target center point.
5. The method of claim 4,
The training department
Billiard guide service providing system using reinforcement learning of a deep learning model, characterized in that the force to hit the point and the direction to hit the point are expressed as vectors as f and (x2, y2, z2), respectively.
6. The method of claim 5,
The training department
9 consecutive values included in (xt, yt, zt), (xball1, yball1, zball1) and (xball2, yball2, zball2) are input, and (x1, y1, z1), (x2, y2, A billiard guide service providing system using reinforcement learning of a deep learning model, characterized in that the deep learning model that outputs seven consecutive values included in z2) and f is trained by reinforcement learning through repetition of simulation.
delete The method of claim 1,
Billiard guide service providing system using reinforcement learning of the deep learning model,
at least one camera installed in a direction opposite to the billiard table to illuminate the billiard table for the user of the user terminal to play billiard;
Billiard guide service providing system using reinforcement learning of the deep learning model, characterized in that it further comprises.
9. The method of claim 8,
Billiard guide service providing system using reinforcement learning of the deep learning model,
at least one display for overlaying and outputting a simulation result output from the transmitter on the image of the pool table captured by the at least one camera;
Billiard guide service providing system using reinforcement learning of the deep learning model, characterized in that it further comprises.
delete

Images