KR102602589B1 - Method and device for managing metaverse pop-up store based on information related to offline pop-up store using neural network - Google Patents

Abstract

Embodiments present a method and device for managing a metaverse pop-up store based on information related to an offline pop-up store using a neural network. The method according to an embodiment receives image information in real time from a plurality of cameras provided in a plurality of tracking areas of the offline pop-up store, and creates a first neural network based on the image information for the plurality of tracking areas. And determining real-time information of a visitor through a tracking model using a second neural network, generating a plurality of first virtual avatars in the metaverse pop-up store based on the real-time information of the visitor, and generating a plurality of first virtual avatars in the metaverse pop-up store based on the real-time information of the visitor. Each moves along the movement line of the visitor matched to the first virtual avatar, and provides information about non-real visitors through a movement line prediction model using a third neural network based on real-time information of the visitor generated during a preset time period. The preset time section is a section between a first time point, which is the start of business of the offline pop-up store, and a second time point, which is the end point of business of the offline pop-up store, and from the second time point onwards, the non-existent visitor A plurality of second virtual avatars are created in the metaverse pop-up store based on the information about the second virtual avatar, and each of the plurality of second virtual avatars can move along the movement path of the non-real visitor matched with the second virtual avatar.

Description

Method and device for managing a metaverse pop-up store based on information related to an offline pop-up store using a neural network {METHOD AND DEVICE FOR MANAGING METAVERSE POP-UP STORE BASED ON INFORMATION RELATED TO OFFLINE POP-UP STORE USING NEURAL NETWORK}

Embodiments of the present disclosure relate to technology for managing a metaverse pop-up store based on information related to an offline pop-up store, and manage a metaverse pop-up store based on information related to the offline pop-up store using a neural network. It is about management methods and devices.

Meanwhile, interest in metaverse is increasing as a representative service that provides virtual worlds to users. Metaverse is a compound word of meta, meaning processing and abstraction, and universe, meaning the real world, and refers to a three-dimensional virtual world. The core technologies of this metaverse are virtual reality (VR), It is an extended reality (XR) technology that encompasses augmented reality (AR) and mixed reality (MR).

Research and development on new types of content are underway in various fields by combining them with the metaverse. For example, Metaverse is being commercialized in fields such as broadcasting, advertising, exhibitions, education, and promotion. However, it is still at an experimental level, and it is difficult to say that it has spread widely.

In particular, various commercialization models are being proposed by combining the metaverse and the real economy, and metaverse implementation technology is needed to implement commercialization models more effectively.

Meanwhile, companies are opening pop-up stores in various locations to promote their brands, increasing brand awareness and improving sales of products. In the case of pop-up stores, short-term lease of idle space is required, and companies invest a lot of money and time to rent idle space suitable for their brand image. Additionally, in the case of products such as cosmetics that have a long repurchase period, when the short-term lease ends, there is a problem that the pop-up store disappears or has to be opened in a different location during the repurchase period.

At this time, it is necessary to expand the user experience to the offline area by reflecting actual information about the offline pop-up store in the metaverse pop-up store. However, when the business hours of the offline pop-up store have ended, a problem arises in which the actual information of the offline pop-up store cannot be reflected.

Accordingly, real-time information about the visitor is generated through a neural network based on video information received from the camera installed in the offline pop-up store, and the first virtual avatar created based on the visitor's real-time information is reflected in the Metaverse pop-up store. , a method and device for managing the metaverse pop-up store are needed.

In addition, after the offline pop-up store closes, non-real visitors are created through a neural network based on real-time information about visitors generated during business hours, and a second virtual avatar is created according to the information about the non-real visitors. By reflecting this in the metaverse pop-up store, a method and device for managing the metaverse pop-up store is needed.

Embodiments of the present disclosure may provide a method and device for managing a metaverse pop-up store based on information related to an offline pop-up store using a neural network.

The technical challenges to be achieved in the embodiments are not limited to the matters mentioned above, and other technical challenges not mentioned may be considered by those skilled in the art from the various embodiments described below. You can.

In one embodiment, a method in which a server manages a metaverse pop-up store based on information related to an offline pop-up store using a neural network includes using a plurality of cameras provided in a plurality of tracking areas of the offline pop-up store. Receive video information in real time, determine real-time information about the visitor through a tracking model using a first neural network and a second neural network based on the video information for the plurality of tracking areas, and determine real-time information about the visitor. Based on this, a plurality of first virtual avatars are created in the metaverse pop-up store, each of the plurality of first virtual avatars moves along the movement line of the visitor matched with the first virtual avatar, and the plurality of first virtual avatars are generated during a preset time period. Based on the visitor's real-time information, information on non-actual visitors is generated through a movement prediction model using a third neural network, and the preset time period starts from the first point in time when the offline pop-up store opens for business. It is a section between a second point in time, which is the end of business of the store, and from after the second point in time, a plurality of second virtual avatars in the metaverse pop-up store are created based on information about the non-actual visitor, and the plurality of second virtual avatars are created based on the information about the non-actual visitor. Each of the two virtual avatars can move along the movement path of the non-real visitor matched with the second virtual avatar.

According to embodiments, the server determines the real-time information of the visitor through a tracking model using a first neural network and a second neural network based on image information about a plurality of tracking areas of the offline pop-up store, Management of visitors can be easy.

According to embodiments, the server generates a plurality of first virtual avatars in the metaverse pop-up store based on real-time information of visitors, thereby operating the metaverse pop-up store by reflecting information about visitors to the offline pop-up store. there is.

According to embodiments, the server generates information about non-real visitors through a movement prediction model using a third neural network based on real-time information of visitors, and provides information about non-real visitors even after the offline pop-up store has closed. By creating a plurality of second virtual avatars in the Metaverse pop-up store based on information, the Metaverse pop-up store can provide online visitors with a continuous experience of the offline pop-up store.

delete

The effects that can be obtained from the examples are not limited to the effects mentioned above, and other effects not mentioned can be clearly derived and understood by those skilled in the art based on the detailed description below. It can be.

The accompanying drawings, which are included as part of the detailed description to aid understanding of the embodiments, provide various embodiments and together with the detailed description describe technical features of the various embodiments.
1 is a diagram showing the configuration of an electronic device according to an embodiment.
Figure 2 is a diagram showing the configuration of a program according to one embodiment.
Figure 3 shows a method in which a server manages a metaverse pop-up store based on information related to an offline pop-up store using a neural network, according to an embodiment.
Figure 4 shows an example of a movement line prediction model according to an embodiment.
Figure 5 is a flowchart showing a method in which a server manages a metaverse pop-up store based on information related to an offline pop-up store using a neural network according to an embodiment.
Figure 6 is a block diagram showing the configuration of a server according to one embodiment.

The following embodiments combine elements and features of the embodiments in a predetermined form. Each component or feature may be considered optional unless explicitly stated otherwise. Each component or feature may be implemented in a form that is not combined with other components or features. Additionally, various embodiments may be configured by combining some components and/or features. The order of operations described in various embodiments may change. Some features or features of one embodiment may be included in other embodiments or may be replaced with corresponding features or features of other embodiments.

In the description of the drawings, procedures or steps that may obscure the gist of the various embodiments are not described, and procedures or steps that can be understood at the level of a person with ordinary knowledge in the relevant technical field are not described. did.

Throughout the specification, when a part is said to “comprise or include” a certain element, this means that it does not exclude other elements but may further include other elements, unless specifically stated to the contrary. do. In addition, terms such as "... unit", "... unit", and "module" used in the specification refer to a unit that processes at least one function or operation, which refers to hardware, software, or a combination of hardware and software. It can be implemented as: Additionally, the terms “a or an,” “one,” “the,” and similar related terms are used herein in the context of describing various embodiments (particularly in the context of the claims below). Unless otherwise indicated or clearly contradicted by context, it may be used in both singular and plural terms.

Hereinafter, embodiments according to various embodiments will be described in detail with reference to the attached drawings. The detailed description set forth below in conjunction with the accompanying drawings is intended to describe exemplary embodiments of various embodiments and is not intended to represent the only embodiment.

In addition, specific terms used in various embodiments are provided to aid understanding of the various embodiments, and the use of such specific terms may be changed to other forms without departing from the technical spirit of the various embodiments. .

1 is a diagram showing the configuration of an electronic device according to an embodiment.

1 is a block diagram of an electronic device 101 in a network environment 100, according to various embodiments. Referring to FIG. 1, in the network environment 100, the electronic device 101 communicates with the electronic device 102 through a first network 198 (e.g., a short-range wireless communication network) or a second network 199. It is possible to communicate with at least one of the electronic device 104 or the server 108 through (e.g., a long-distance wireless communication network). According to one embodiment, the electronic device 101 may communicate with the electronic device 104 through the server 108. According to one embodiment, the electronic device 101 includes a processor 120, a memory 130, an input module 150, an audio output module 155, a display module 160, an audio module 170, and a sensor module ( 176), interface 177, connection terminal 178, haptic module 179, camera module 180, power management module 188, battery 189, communication module 190, subscriber identification module 196 , or may include an antenna module 197. In some embodiments, at least one of these components (eg, the connection terminal 178) may be omitted or one or more other components may be added to the electronic device 101. In some embodiments, some of these components (e.g., sensor module 176, camera module 180, or antenna module 197) are integrated into one component (e.g., display module 160). It can be. The electronic device 101 may also be referred to as a client, terminal, or peer.

The processor 120, for example, executes software (e.g., program 140) to operate at least one other component (e.g., hardware or software component) of the electronic device 101 connected to the processor 120. It can be controlled and various data processing or calculations can be performed. According to one embodiment, as at least part of data processing or computation, the processor 120 stores commands or data received from another component (e.g., sensor module 176 or communication module 190) in volatile memory 132. The commands or data stored in the volatile memory 132 can be processed, and the resulting data can be stored in the non-volatile memory 134. According to one embodiment, the processor 120 includes a main processor 121 (e.g., a central processing unit or an application processor) or an auxiliary processor 123 that can operate independently or together (e.g., a graphics processing unit, a neural network processing unit ( It may include a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor). For example, if the electronic device 101 includes a main processor 121 and a secondary processor 123, the secondary processor 123 may be set to use lower power than the main processor 121 or be specialized for a designated function. You can. The auxiliary processor 123 may be implemented separately from the main processor 121 or as part of it.

The auxiliary processor 123 may, for example, act on behalf of the main processor 121 while the main processor 121 is in an inactive (e.g., sleep) state, or while the main processor 121 is in an active (e.g., application execution) state. ), together with the main processor 121, at least one of the components of the electronic device 101 (e.g., the display module 160, the sensor module 176, or the communication module 190) At least some of the functions or states related to can be controlled. According to one embodiment, co-processor 123 (e.g., image signal processor or communication processor) may be implemented as part of another functionally related component (e.g., camera module 180 or communication module 190). there is. According to one embodiment, the auxiliary processor 123 (eg, neural network processing unit) may include a hardware structure specialized for processing artificial intelligence models.

Artificial intelligence models can be created through machine learning. For example, such learning may be performed in the electronic device 101 itself on which the artificial intelligence model is performed, or may be performed through a separate server (e.g., server 108). Learning algorithms may include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but It is not limited. An artificial intelligence model may include multiple artificial neural network layers. Artificial neural networks include deep neural network (DNN), convolutional neural network (CNN), recurrent neural network (RNN), restricted boltzmann machine (RBM), belief deep network (DBN), bidirectional recurrent deep neural network (BRDNN), It may be one of deep Q-networks or a combination of two or more of the above, but is not limited to the examples described above. In addition to hardware structures, artificial intelligence models may additionally or alternatively include software structures.

The memory 130 may store various data used by at least one component (eg, the processor 120 or the sensor module 176) of the electronic device 101. Data may include, for example, input data or output data for software (e.g., program 140) and instructions related thereto. Memory 130 may include volatile memory 132 or non-volatile memory 134.

The program 140 may be stored as software in the memory 130 and may include, for example, an operating system 142, middleware 144, or application 146.

The input module 150 may receive commands or data to be used in a component of the electronic device 101 (e.g., the processor 120) from outside the electronic device 101 (e.g., a user). The input module 150 may include, for example, a microphone, mouse, keyboard, keys (eg, buttons), or digital pen (eg, stylus pen).

The sound output module 155 may output sound signals to the outside of the electronic device 101. The sound output module 155 may include, for example, a speaker or a receiver. Speakers can be used for general purposes such as multimedia playback or recording playback. The receiver can be used to receive incoming calls. According to one embodiment, the receiver may be implemented separately from the speaker or as part of it.

The display module 160 can visually provide information to the outside of the electronic device 101 (eg, a user). The display module 160 may include, for example, a display, a hologram device, or a projector, and a control circuit for controlling the device. According to one embodiment, the display module 160 may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of force generated by the touch.

The audio module 170 can convert sound into an electrical signal or, conversely, convert an electrical signal into sound. According to one embodiment, the audio module 170 acquires sound through the input module 150, the sound output module 155, or an external electronic device (e.g., directly or wirelessly connected to the electronic device 101). Sound may be output through the electronic device 102 (e.g., speaker or headphone).

The sensor module 176 detects the operating state (e.g., power or temperature) of the electronic device 101 or the external environmental state (e.g., user state) and generates an electrical signal or data value corresponding to the detected state. can do. According to one embodiment, the sensor module 176 includes, for example, a gesture sensor, a gyro sensor, an air pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biometric sensor, It may include a temperature sensor, humidity sensor, or light sensor.

The interface 177 may support one or more designated protocols that can be used to connect the electronic device 101 directly or wirelessly with an external electronic device (eg, the electronic device 102). According to one embodiment, the interface 177 may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

The connection terminal 178 may include a connector through which the electronic device 101 can be physically connected to an external electronic device (eg, the electronic device 102). According to one embodiment, the connection terminal 178 may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (eg, a headphone connector).

The haptic module 179 can convert electrical signals into mechanical stimulation (e.g., vibration or movement) or electrical stimulation that the user can perceive through tactile or kinesthetic senses. According to one embodiment, the haptic module 179 may include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

The camera module 180 can capture still images and videos. According to one embodiment, the camera module 180 may include one or more lenses, image sensors, image signal processors, or flashes.

The power management module 188 can manage power supplied to the electronic device 101. According to one embodiment, the power management module 188 may be implemented as at least a part of, for example, a power management integrated circuit (PMIC).

The battery 189 may supply power to at least one component of the electronic device 101. According to one embodiment, the battery 189 may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

Communication module 190 is configured to provide a direct (e.g., wired) communication channel or wireless communication channel between electronic device 101 and an external electronic device (e.g., electronic device 102, electronic device 104, or server 108). It can support establishment and communication through established communication channels. Communication module 190 operates independently of processor 120 (e.g., an application processor) and may include one or more communication processors that support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module 190 is a wireless communication module 192 (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module 194 (e.g., : LAN (local area network) communication module, or power line communication module) may be included. Among these communication modules, the corresponding communication module is a first network 198 (e.g., a short-range communication network such as Bluetooth, wireless fidelity (WiFi) direct, or infrared data association (IrDA)) or a second network 199 (e.g., legacy It may communicate with an external electronic device 104 through a telecommunication network such as a cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., LAN or WAN). These various types of communication modules may be integrated into one component (e.g., a single chip) or may be implemented as a plurality of separate components (e.g., multiple chips). The wireless communication module 192 uses subscriber information (e.g., International Mobile Subscriber Identifier (IMSI)) stored in the subscriber identification module 196 within a communication network such as the first network 198 or the second network 199. The electronic device 101 can be confirmed or authenticated.

The wireless communication module 192 may support 5G networks after 4G networks and next-generation communication technologies, for example, NR access technology (new radio access technology). NR access technology provides high-speed transmission of high-capacity data (eMBB (enhanced mobile broadband)), minimization of terminal power and access to multiple terminals (mMTC (massive machine type communications)), or high reliability and low latency (URLLC (ultra-reliable and low latency). -latency communications)) can be supported. The wireless communication module 192 may support high frequency bands (eg, mmWave bands), for example, to achieve high data rates. The wireless communication module 192 uses various technologies to secure performance in high frequency bands, for example, beamforming, massive array multiple-input and multiple-output (MIMO), and full-dimensional multiplexing. It can support technologies such as input/output (FD-MIMO: full dimensional MIMO), array antenna, analog beam-forming, or large scale antenna. The wireless communication module 192 may support various requirements specified in the electronic device 101, an external electronic device (e.g., electronic device 104), or a network system (e.g., second network 199). According to one embodiment, the wireless communication module 192 supports Peak data rate (e.g., 20 Gbps or more) for realizing eMBB, loss coverage (e.g., 164 dB or less) for realizing mmTC, or U-plane latency (e.g., 164 dB or less) for realizing URLLC. Example: Downlink (DL) and uplink (UL) each of 0.5 ms or less, or round trip 1 ms or less) can be supported.

The antenna module 197 may transmit or receive signals or power to or from the outside (eg, an external electronic device). According to one embodiment, the antenna module 197 may include an antenna including a radiator made of a conductor or a conductive pattern formed on a substrate (eg, PCB). According to one embodiment, the antenna module 197 may include a plurality of antennas (eg, an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network such as the first network 198 or the second network 199 is connected to the plurality of antennas by, for example, the communication module 190. can be selected Signals or power may be transmitted or received between the communication module 190 and an external electronic device through the at least one selected antenna. According to some embodiments, in addition to the radiator, other components (eg, radio frequency integrated circuit (RFIC)) may be additionally formed as part of the antenna module 197.

According to various embodiments, the antenna module 197 may form a mmWave antenna module. According to one embodiment, a mmWave antenna module includes: a printed circuit board, an RFIC disposed on or adjacent to a first side (e.g., bottom side) of the printed circuit board and capable of supporting a designated high frequency band (e.g., mmWave band); And a plurality of antennas (e.g., array antennas) disposed on or adjacent to the second side (e.g., top or side) of the printed circuit board and capable of transmitting or receiving signals in the designated high frequency band. can do.

At least some of the components are connected to each other through a communication method between peripheral devices (e.g., bus, general purpose input and output (GPIO), serial peripheral interface (SPI), or mobile industry processor interface (MIPI)) and signal ( (e.g. commands or data) can be exchanged with each other.

According to one embodiment, commands or data may be transmitted or received between the electronic device 101 and the external electronic device 104 through the server 108 connected to the second network 199. Each of the external electronic devices 102 or 104 may be of the same or different type as the electronic device 101. According to one embodiment, all or part of the operations performed in the electronic device 101 may be executed in one or more of the external electronic devices 102, 104, or 108. For example, when the electronic device 101 needs to perform a certain function or service automatically or in response to a request from a user or another device, the electronic device 101 may perform the function or service instead of executing the function or service on its own. Alternatively, or additionally, one or more external electronic devices may be requested to perform at least part of the function or service. One or more external electronic devices that have received the request may execute at least part of the requested function or service, or an additional function or service related to the request, and transmit the result of the execution to the electronic device 101. The electronic device 101 may process the result as is or additionally and provide it as at least part of a response to the request. For this purpose, for example, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology can be used. The electronic device 101 may provide an ultra-low latency service using, for example, distributed computing or mobile edge computing. In another embodiment, the external electronic device 104 may include an Internet of Things (IoT) device. Server 108 may be an intelligent server using machine learning and/or neural networks. According to one embodiment, the external electronic device 104 or server 108 may be included in the second network 199. The electronic device 101 may be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

The server 108 is connected to the electronic device 101 and can provide services to the connected electronic device 101. In addition, the server 108 may perform a membership registration process, store and manage various information of users who have registered as members, and provide various purchase and payment functions related to the service. Additionally, the server 108 may share execution data of service applications running on each of the plurality of electronic devices 101 in real time so that services can be shared between users. This server 108 may have the same hardware configuration as a typical web server or service server. However, in terms of software, it may be implemented through any language such as C, C++, Java, Python, Golang, and Kotlin and may include program modules that perform various functions. In addition, the server 108 is generally connected to an unspecified number of clients and/or other servers through an open computer network such as the Internet, and receives work performance requests from clients or other servers and derives and provides work results in response. It refers to a computer system and the computer software (server program) installed for it. In addition, in addition to the server program described above, the server 108 includes a series of application programs running on the server 108 and, in some cases, various databases (DBs) built internally or externally, hereinafter " It should be understood as a broad concept including “DB”). Accordingly, the server 108 classifies membership registration information and various information and data about games, stores them in a DB, and manages this DB, which may be implemented inside or outside the server 108. In addition, the server 108 can be implemented using a variety of server programs provided on general server hardware and operating systems such as Windows, Linux, UNIX, and Macintosh. , Representative examples include IIS (Internet Information Server) used in a Windows environment and CERN, NCSA, APPACH, and TOMCAT used in a Unix environment, etc., to implement web services. Additionally, the server 108 may be linked with an authentication system and payment system for user authentication of the service or payment for purchases related to the service.

The first network 198 and the second network 199 are a connection structure that allows information exchange between each node, such as terminals and servers, or a network connecting the server 108 and the electronic devices 101 and 104. It means (Network). The first network 198 and the second network 199 are the Internet, LAN (Local Area Network), Wireless LAN (Wireless Local Area Network), WAN (Wide Area Network), PAN (Personal Area Network), and 3G. , 4G, LTE, 5G, Wi-Fi, etc., but are not limited to these. The first network 198 and the second network 199 may be closed, such as a LAN or WAN, but are preferably open, such as the Internet. The Internet includes protocols such as TCP/IP protocol, TCP, and UDP (user datagram protocol), as well as various services that exist at the upper layer, such as HTTP (HyperText Transfer Protocol), Telnet, FTP (File Transfer Protocol), and DNS (Domain Name System). ), a worldwide open computer primary network (198) and secondary network (199) that provides Simple Mail Transfer Protocol (SMTP), Simple Network Management Protocol (SNMP), Network File Service (NFS), and Network Information Service (NIS). ) refers to the structure.

A database can have a general data structure implemented in the storage space (hard disk or memory) of a computer system using a database management program (DBMS). A database may have a data storage format that allows for free search (extraction), deletion, editing, addition, etc. of data. Databases are relational database management systems (RDBMS) such as Oracle, Infomix, Sybase, and DB2, or object-oriented database management such as Gemston, Orion, and O2. It can be implemented according to the purpose of an embodiment of the present disclosure using a system (OODBMS) and an XML native database such as Excelon, Tamino, and Sekaiju, and has its own functions. To achieve this, you can have appropriate fields or elements.

Figure 2 is a diagram showing the configuration of a program according to one embodiment.

Figure 2 is a block diagram 200 illustrating program 140 according to various embodiments. According to one embodiment, the program 140 includes an operating system 142, middleware 144, or an application 146 executable on the operating system 142 for controlling one or more resources of the electronic device 101. It can be included. Operating system 142 may include, for example, AndroidTM, iOSTM, WindowsTM, SymbianTM, TizenTM, or BadaTM. At least some of the programs 140 are preloaded into the electronic device 101, for example, at the time of manufacture, or are stored in an external electronic device (e.g., the electronic device 102 or 104, or a server) when used by a user. It can be downloaded or updated from 108)). All or part of the program 140 may include a neural network.

The operating system 142 may control management (eg, allocation or retrieval) of one or more system resources (eg, process, memory, or power) of the electronic device 101 . Operating system 142 may additionally or alternatively operate on other hardware devices of electronic device 101, such as input module 150, audio output module 155, display module 160, and audio module 170. , sensor module 176, interface 177, haptic module 179, camera module 180, power management module 188, battery 189, communication module 190, subscriber identification module 196, or It may include one or more driver programs for driving the antenna module 197.

The middleware 144 may provide various functions to the application 146 so that functions or information provided from one or more resources of the electronic device 101 can be used by the application 146. The middleware 144 includes, for example, an application manager 201, a window manager 203, a multimedia manager 205, a resource manager 207, a power manager 209, a database manager 211, and a package manager 213. ), connectivity manager (215), notification manager (217), location manager (219), graphics manager (221), security manager (223), call manager (225), or voice recognition manager (227). You can.

The application manager 201 may, for example, manage the life cycle of the application 146. The window manager 203 may, for example, manage one or more GUI resources used on the screen. For example, the multimedia manager 205 identifies one or more formats required for playing media files, and encodes or decodes the corresponding media file using a codec suitable for the selected format. It can be done. The resource manager 207 may, for example, manage the source code of the application 146 or the memory space of the memory 130. The power manager 209 manages, for example, the capacity, temperature, or power of the battery 189, and may use this information to determine or provide related information necessary for the operation of the electronic device 101. . According to one embodiment, the power manager 209 may interface with a basic input/output system (BIOS) (not shown) of the electronic device 101.

Database manager 211 may create, search, or change a database to be used by application 146, for example. The package manager 213 may, for example, manage the installation or update of applications distributed in the form of package files. The connectivity manager 215 may manage, for example, a wireless connection or direct connection between the electronic device 101 and an external electronic device. For example, the notification manager 217 may provide a function for notifying the user of the occurrence of a designated event (eg, an incoming call, message, or alarm). The location manager 219 may, for example, manage location information of the electronic device 101. The graphics manager 221 may, for example, manage one or more graphic effects to be provided to the user or a user interface related thereto.

Security manager 223 may provide, for example, system security or user authentication. The telephony manager 225 may manage, for example, a voice call function or a video call function provided by the electronic device 101. For example, the voice recognition manager 227 transmits the user's voice data to the server 108 and provides a command corresponding to a function to be performed in the electronic device 101 based at least in part on the voice data, Alternatively, text data converted based at least in part on the voice data may be received from the server 108. According to one embodiment, the middleware 244 may dynamically delete some existing components or add new components. According to one embodiment, at least a portion of the middleware 144 may be included as part of the operating system 142 or may be implemented as separate software different from the operating system 142.

The application 146 includes, for example, home 251, dialer 253, SMS/MMS (255), instant message (IM) 257, browser 259, camera 261, and alarm 263. , Contacts (265), Voice Recognition (267), Email (269), Calendar (271), Media Player (273), Album (275), Watch (277), Health (279) (such as exercise amount or blood sugar) It may include applications that measure biometric information) or environmental information 281 (e.g., measure atmospheric pressure, humidity, or temperature information). According to one embodiment, the application 146 may further include an information exchange application (not shown) that can support information exchange between the electronic device 101 and an external electronic device. The information exchange application may include, for example, a notification relay application configured to deliver designated information (e.g., calls, messages, or alarms) to an external electronic device, or a device management application configured to manage the external electronic device. there is. The notification relay application, for example, transmits notification information corresponding to a specified event (e.g., mail reception) generated in another application (e.g., email application 269) of the electronic device 101 to an external electronic device. You can. Additionally or alternatively, the notification relay application may receive notification information from an external electronic device and provide it to the user of the electronic device 101.

The device management application, for example, controls the power (e.g., turn-on or turn-off) of an external electronic device or some component thereof (e.g., a display module or camera module of the external electronic device) that communicates with the electronic device 101. ) or functions (such as brightness, resolution, or focus). A device management application may additionally or alternatively support installation, deletion, or update of applications running on external electronic devices.

Throughout this specification, neural network, neural network, and network function may be used with the same meaning. A neural network may consist of a set of interconnected computational units, which may generally be referred to as “nodes.” These “nodes” may also be referred to as “neurons.” A neural network is composed of at least two or more nodes. Nodes (or neurons) that make up neural networks may be interconnected by one or more “links.”

Within a neural network, two or more nodes connected through a link can relatively form a relationship as an input node and an output node. The concepts of input node and output node are relative, and any node in an output node relationship with one node may be in an input node relationship with another node, and vice versa. As described above, input node to output node relationships can be created around links. One or more output nodes can be connected to one input node through a link, and vice versa.

In a relationship between an input node and an output node connected through one link, the value of the output node may be determined based on data input to the input node. Here, nodes connecting the input node and the output node may have weights. Weights may be variable and may be varied by a user or algorithm in order for the neural network to perform a desired function. Here, the edges or links that interconnect the input nodes and output nodes have weights that can be variably applied by the user or algorithm to perform the function desired by the neural network. For example, when one or more input nodes are connected to one output node by respective links, the output node is set to the values input to the input nodes connected to the output node and the links corresponding to each input node. The output node value can be determined based on the weight.

As described above, in a neural network, two or more nodes are interconnected through one or more links to form an input node and output node relationship within the neural network. The characteristics of the neural network may be determined according to the number of nodes and links in the neural network, the correlation between the nodes and links, and the value of the weight assigned to each link. For example, if there are two neural networks with the same number of nodes and links and different weight values between the links, the two neural networks may be recognized as different from each other.

Figure 3 shows a method in which a server manages a metaverse pop-up store based on information related to an offline pop-up store using a neural network, according to an embodiment. The embodiment of FIG. 3 can be combined with various embodiments of the present disclosure.

Referring to FIG. 3, in step S310, a server (e.g., server 108 of FIG. 1) may receive video information in real time from a plurality of cameras provided in a plurality of tracking areas of an offline pop-up store.

The server is a server that provides services for managing the Metaverse pop-up store built in virtual space based on information about offline pop-up stores. At this time, offline pop-up stores can be installed in idle spaces such as hotels, shopping malls, and tourist attractions. For example, the server can manage information about offline pop-up stores and information about Metaverse Pop-up 310 stores in virtual space. For example, even if the lease contract for the offline pop-up store has expired, the server can provide operating services for the metaverse pop-up store in virtual space by reflecting information acquired during the business period of the offline pop-up store.

An offline pop-up store may be a pop-up store installed in an idle space where a corporate terminal rents the idle space. A corporate terminal may be a terminal used by a company that rents an idle space to install a pop-up store and wants to build a metaverse pop-up store in a virtual space linked to the idle space. For example, a corporate terminal may include the electronic device 101 of FIG. 1 .

The Metaverse pop-up store may be a pop-up store that is an offline pop-up store built in a virtual space. For example, a metaverse pop-up store may be a pop-up store built in a virtual space through a virtual pop-up store creation model using a neural network based on information about pop-up stores related to idle space.

The tracking area may be an area where images are taken using a camera to track visitors to an offline pop-up store. For example, the plurality of tracking areas may include a visitor entrance area, at least one movement passage area, at least one product display area, and a product purchase area. Hereinafter, the product may refer to a product sold in at least one of an offline pop-up store or an online pop-up store, and may be referred to as a sales product instead. The visitor entrance area is an area that includes the entrance to the offline pop-up store, and video information of the visitor entrance area can be obtained through a camera installed at the entrance. The moving passage area is an area that includes the moving passage of visitors located in an offline pop-up store, and image information of the moving passage area can be obtained through a camera installed in the moving passage. The product display area is an area that includes a space where products are displayed in an offline pop-up store, and video information about the product display area can be obtained through a camera installed in the space where the product is displayed. The product purchase area is a space where products are purchased at an offline pop-up store, that is, an area that includes a space where a POS terminal is installed, and image information of the product purchase area can be obtained through a camera provided in the space where the POS terminal is installed. Here, the image information may be information about an image captured through a camera.

In step S320, the server may determine real-time information about the visitor through a tracking model using a first neural network and a second neural network based on image information about a plurality of tracking areas.

For example, image information may be classified by multiple tracking areas.

For example, the server may generate a face image vector for each time section through first data preprocessing on image information of the visitor entrance area. For example, the first data preprocessing extracts the visitor's face image from a plurality of images included in the image information of the visitor's entrance area, determines the center coordinates of both eyes in the visitor's face image, and determines the center coordinates of both eyes from the center coordinates of both eyes. This may be a process of rotating the visitor's face image so that the value for the y coordinate is the same. A plurality of images included in the video information of the visitor entrance area may include video images per unit time constituting the video. For example, the server may generate a face image vector containing pixel values for a rotated visitor's face image for each time interval. That is, the face image vector for each time section may be a vector expressed as a pixel value by first data preprocessing the face image of a visitor who visited the offline pop-up store for each time section.

Additionally, for example, a faster R-CNN (faster regions with convolutional neural network) model may be used in the process of extracting a visitor's face image from a plurality of images included in the video information of the visitor entrance area. R-CNN (Regions with Convolutional Neural Network) refers to a neural network that performs object detection by using a set region as an input value for CNN. R-CNN can perform two operations: region estimation and region classification. For example, R-CNN uses region proposals to explore the object's region regardless of category by inputting data and labels in the image, and warping/cropping a fixed-sized feature vector from the proposed region. /crop) can be used as input to CNN. Here, a pre-trained network is used in CNN, and classification is performed through SVM (Support Vector Machine), a linear supervised learning model, using feature maps generated through CNN, and bounding box regression through a regressor. box regression) can be performed. At this time, Fast R-CNN can pass the CNN once by extracting the features of the corresponding area from the feature map rather than extracting (crop) the features of the area from the original image. In other words, Fast R-CNN performs pooling on the region of interest (ROI), converts each region into a vector of a fixed size, and passes each vector through a fully connected (FC) layer to speed up the speed. can be improved. However, there was still a problem of slow speed due to the bottleneck of the algorithm used to propose a region in Fast R-CNN, and Faster R-CNN uses a region proposal network (RPN) instead of the algorithm for region proposal. Proposal speed was improved. RPN may include a convolution layer, a region proposal layer, a classification layer, and a regression layer. Specifically, for example, the server can learn an RPN through end-to-end learning to propose a region, and use only the region proposal layer excluding the basic CNN in the learned RPN to perform Fast R-CNN can be trained. Afterwards, the management server can fine-tune the CNN that extracts the first feature map, fix other weights using the learned Fast R-CNN and RPN, and fine-tune only the layers corresponding to the RPN. Afterwards, the server can train Faster R-CNN by fixing the shared convolutional layer and fine-tuning the layer corresponding to Fast R-CNN.

For example, a feature map for an image can be extracted through a convolutional layer. The feature map is delivered to the RPN and Fast R-CNN, and the RPN can search multiple regions through a region proposal layer based on the feature map and pool the multiple regions into multiple regions of interest. The plurality of regions of interest may include areas of the visitor's face. Finally, the server can output values related to multiple regions of interest as a face image vector using the feature map and region proposal bounding box.

At this time, RPN can receive the entire image regardless of size and return a region proposal bounding box. Each bounding box can indicate whether an object exists with a score. Each sliding window can be mapped to a small-dimensional feature, and the features obtained by the sliding window can be passed to the classification layer and regression layer. Several bounding box areas can be predicted for each center position of each sliding window. If the maximum number of bounding box areas that can be predicted for each sliding window position is k, the regression layer can have 4k coordinate values, and the classification layer can have 2k score values.

For example, when the server inputs face image vectors for each time section into the first neural network, values for the gender and age group of visitors who visited the offline pop-up store may be output. The values for the visitor's gender and age range may be values representing the visitor's gender and age range (e.g., age range from teens to 90s). For example, if the visitor is male and a teenager, the values for the gender and age range of the text may include a value indicating a man and a value indicating a teenager.

For example, the first neural network uses a convolutional neural network (CNN) model and may include a first input layer, one or more first hidden layers, and a first output layer. Each learning data consisting of a plurality of face image vectors, a plurality of reference face image vectors, and a plurality of values for the gender and age range of the plurality of correct answer visitors is input to the first input layer of the first neural network and the one or more first It passes through the hidden layer and the first output layer and is output as a 12th output vector, the first output vector is input to the first loss function layer connected to the first output layer, and the first loss function layer is the first loss function layer. A first loss value can be output using a first loss function that compares the output vector with the correct answer vector for each learning data, and the parameters of the first neural network can be learned in a direction where the first loss value decreases. there is.

That is, the first neural network extracts feature points for a plurality of face image vectors and a plurality of face reference image vectors, determines the similarity between the feature points, and determines the gender and age group matched to the reference face image vector with the highest similarity. The value can be compared with the correct answer for gender and age range and learned to reduce the difference.

The plurality of reference face image vectors may be vectors that measure similarity to the face image vector to derive the age group and gender of the face image vector. For example, the plurality of reference image vectors may include vectors generated through first data preprocessing of a plurality of face images for men and women by age group.

The values for the gender and age range of the plurality of correct visitors may be the values for the actual gender and actual age range for each of the plurality of face image vectors.

For example, the plurality of face image vectors and the plurality of reference face image vectors may include feature values related to eyes, feature values related to the nose, feature values related to the skin, and feature values related to the mouth. That is, a plurality of feature points related to the eyes, a plurality of feature points related to the nose, and a feature point related to the mouth may be determined for the plurality of face image vectors and the plurality of reference face image vectors. For example, feature values related to eyes may be determined based on a plurality of feature points for the eyes. Feature values related to the eyes include the value for the horizontal length of the right eye, the value for the vertical length of the eye on the right, the value for the horizontal length of the left eye, the value for the vertical length of the left eye, and the value for the vertical length of the left eye. It can contain a value for length. For example, a feature value related to the nose may be determined based on a plurality of feature points for the nose. Feature values related to the nose may include values for the vertical length of the nose and values for the horizontal width of the nose. For example, a feature value related to skin may be determined based on a plurality of feature points for the skin. Feature values related to skin may include values for skin tone, values for the length of wrinkles, and values for the depth of wrinkles. For example, a feature value related to the mouth may be determined based on a plurality of feature points for the mouth. Feature values related to the mouth may include values for the thickness of the upper lip, values for the thickness of the lower lip, and values for the horizontal length of the lips.

For example, the number of visitors may be counted based on values for the visitor's gender and age range. The server can count the number of visitors for each time section of the offline pop-up store based on the visitor's gender and age group. For example, the server can store the number of visitors by time period and the gender and age group of the visitors. At this time, the number of visitors according to gender and age group may be counted.

For example, the server may generate a plurality of image vectors for each time section for each tracking area through second data preprocessing on image information of the plurality of tracking areas. For each piece of video information in a plurality of tracking areas, a video image per unit time constituting the corresponding video can be determined.

The second data preprocessing may include binarization, image contrast correction, tilt correction, and noise removal for the video image to specify the location of the visitor within the video image. For example, an image vector composed of pixel values for the video images may be generated through second data preprocessing for a plurality of video images for each time section for each tracking area. Binarization is an operation that converts an image into a binarized black and white image using a threshold, and through binarization of a video image, the video image can be converted to a black and white image. Image contrast correction is an operation that sharpens the boundaries of a video image to separate the background from the video image. For example, CLAHE (contrast limited adaptive histogram equalization) technique may be used. The CLAHE technique is a technique that divides a video image into blocks of a certain size and performs histogram equalization for each block to equalize the entire video image. Tilt correction is an operation to adjust the angle of a tilted video image to align it horizontally and vertically in order to improve the recognition rate of the video image. Noise removal can additionally remove noise that has not been removed through binarization and image contrast correction of the video image through various filters such as Gaussian blur, central blur, and bidirectional filtering. Through this, the processing speed of the second neural network can be improved by reducing the capacity for a plurality of video images.

For example, when the server inputs a plurality of image vectors for each time section for each tracking area to the second neural network, the location coordinates for each time section of a plurality of visitors included in each tracking area may be output.

For example, the second neural network may use the YOLO (you only look once) model. The server can recognize visitors in each image vector by inputting multiple image vectors by time section for each tracking area into the YOLO model, and obtain the start coordinates, height, and width information of the border box for the recognized visitor. . Here, the YOLO model is an artificial intelligence algorithm that implements integrated recognition by simultaneously executing bounding box coordinates and classification through a neural network. The server can determine the visitor's location coordinates within the image based on the starting coordinates, height, and width information of the border box. For example, YOLO divides an image vector into a grid of a preset size, and each grid cell can predict the probability that an object, that is, a visitor, exists in that area. YOLO can predict the location of visitors based on the divided grid and set a bounding box at that location. At this time, because YOLO searches for objects while processing the image vector only once through the network, it can identify the visitor's location coordinates faster than other neural network models. YOLO can use feature extraction layers at preset levels. In order to simultaneously predict the location and class of an object, YOLO uses a loss function to measure the difference between the detected object and the actual object, a confidence loss function to determine whether the object exists, and a confidence loss function to adjust the position of the bounding box. It can be composed of a localization loss function and a class loss function to predict the class of the object, and can be learned to minimize it.

For example, real-time information on visitors includes the entry time of multiple visitors, the number of multiple visitors, values for gender and age range for multiple visitors, and the location of multiple visitors included in each tracking area by time section. Can include coordinates.

In step S33O, the server may create a plurality of first virtual avatars in the metaverse pop-up store based on the visitor's real-time information.

For example, a plurality of first virtual avatars may be created by one-to-one matching with a plurality of visitors included in the visitor's real-time information. For example, a plurality of first virtual avatars may be created at the time a plurality of visitors enter. For example, the shape of the first virtual avatar may be randomly determined to be one of a plurality of shapes of the first virtual avatar that matches the gender and age values of the visitor corresponding to the first virtual avatar. For example, the form of a plurality of first virtual avatars may be pre-stored on the server for each value for gender and age group for the visitor.

For example, each of the plurality of first virtual avatars may move along the movement path of the visitor matched to the first virtual avatar. At this time, the location coordinates for each time section of the plurality of visitors included in each tracking area may be linked to the location coordinates within the metaverse pop-up store. That is, a plurality of location coordinates included in each tracking area can be converted into location coordinates within the metaverse pop-up store. For example, the matching relationship between the location coordinates included in each tracking area and the location coordinates within the metaverse pop-up store may be preset on the server. For example, the first virtual avatar may automatically move by time section according to the location coordinates in the metaverse pop-up store linked to the location coordinates included in each tracking area for the visitor matched with the first virtual avatar.

In step S340, the server may generate information about non-existent visitors through a movement prediction model using a third neural network based on real-time information about visitors generated during a preset time period.

The preset time section may be a section between a first point in time when the offline pop-up store starts operating and a second point in time when the offline pop-up store ends.

For example, the server may generate a time vector, a visitor vector, and a movement vector for each tracking area through third-party data preprocessing of real-time visitor information generated during a preset time period. For example, a time vector may include a value for a first time point and a value for a second time point. For example, a visitor vector may include values related to the number of visitors by time interval and the gender and age group of the visitors by time interval. For example, the movement vector may include location coordinates for each of a plurality of visitors for each time section.

For example, when the server inputs a time vector, a visitor vector, and a movement vector for each tracking area into the third neural network, the values for the gender and age range of a plurality of non-real visitors and the movement vector for each tracking area are included in each tracking area. The location coordinates of each time section of the non-actual visitor may be output.

Here, the non-real visitor may be a virtual visitor formed based on real-time information of real visitors who visited during business hours.

For example, the server may determine information about non-real visitors based on values for the gender and age range of a plurality of non-real visitors and the location coordinates of each time section of the non-real visitors included in each tracking area.

For example, information about non-real visitors includes the scheduled entry times of multiple non-real visitors, the number of multiple non-real visitors, the gender and age range of each of the multiple non-real visitors, and the plurality of non-real visitors included in each tracking area. It may include location coordinates for each time section of the non-actual visitor.

In step S350, the server may generate a plurality of second virtual avatars in the metaverse pop-up store based on information about the non-actual visitor from the second point in time.

For example, a plurality of second virtual avatars may be created by one-to-one matching with a plurality of non-real visitors according to information about the non-real visitors. For example, a plurality of second virtual avatars may be created at a time when a plurality of non-actual visitors are scheduled to enter. For example, the shape of the second virtual avatar may be randomly determined to be one of the shapes of a plurality of second virtual avatars that match the gender and age range values for the non-real visitor corresponding to the second virtual avatar. there is. For example, the form of a plurality of second virtual avatars may be pre-stored in the server for each gender and age group value for the non-real visitor. At this time, the shape of the plurality of second virtual avatars may be different from the shape of the first plurality of virtual avatars.

For example, each of the plurality of second virtual avatars may move along the movement path of the non-real visitor matched with the second virtual avatar. At this time, the location coordinates for each time section of the plurality of non-actual visitors included in each tracking area may be linked to the location coordinates within the metaverse pop-up store. For example, the second virtual avatar may automatically move by time section according to the location coordinates in the metaverse pop-up store linked to the location coordinates included in each tracking area for the non-real visitor matched with the second virtual avatar. .

According to one embodiment, the server may receive sales information from a POS terminal installed in an offline pop-up store.

Here, a point of sale (POS) terminal may be an electronic device that processes transactions for products and makes payments within an offline pop-up store. For example, the POS terminal may include the electronic device 102 of FIG. 1 .

For example, the server can receive sales information in real time from a POS terminal installed in an offline pop-up store and store the sales information by reception time.

The server can rearrange the positions of multiple product objects placed in the metaverse pop-up store based on sales information.

Here, the sales information may include information about offline products sold in offline pop-up stores and sales volume for each offline product.

A product object is an object placed in a metaverse pop-up store, and may be an object built in virtual space based on scanned information of sales products placed in an offline pop-up store through a 3D scanning interface.

The 3D scanning interface can scan and model real-world objects or environments in 3D. For example, a 3D scanning interface can typically use various sensors and cameras through various scanning technologies to determine the shape of an object, the surface details of an object, and the color and texture of an object. At this time, the scanned 3D information may be expressed as a point cloud or surface. Here, a point cloud is a set of three-dimensional coordinates sampled from various locations in space, and each point may represent the surface of an object. The surface estimated through the point cloud can be represented as a shape such as a triangle.

For example, a 3D scanning interface can determine 3D information about an object by scanning items placed in an actual pop-up store using a stereo camera and photogrammetry. Here, the stereo camera can estimate depth information about the object using two cameras. The 3D scanning interface acquires images taken of objects from different angles with a stereo camera, analyzes the differences between pixels, and calculates the distance to the corresponding pixel to determine 3D information about the object. Photogrammetry can reconstruct an object into a three-dimensional shape using multiple images. For example, a 3D scanning interface can extract feature points from each image through photogrammetry and calculate the relative positions of the camera and object by comparing the positions of the feature points. A 3D scanning interface can generate 3D information and determine detailed information about the surface of an object.

The server can determine the sales ranking based on sales information for all product objects placed in the metaverse pop-up store. For example, a plurality of product objects may include product objects up to a preset sales rank among all product objects. For example, if the preset sales ranking is 5th, the server determines the sales volume of offline products matching that product object for each of all product objects placed in the metaverse pop-up store, and ranks the offline products in order of highest sales volume. As such, the top five product objects can be determined as multiple product objects.

The server may determine a plurality of locations in order of high density of visitors based on the location coordinates of each time section of the plurality of visitors to at least one product display area. For example, the density of visitors may be the number of visitors who stayed at that location coordinate. For example, a plurality of positions may be determined as many as the number of preset ranks. That is, if the preset ranking is 5th, the plurality of positions may be 5.

The server may sequentially rearrange multiple product objects to locations within the metaverse pop-up store corresponding to each of the multiple locations. For example, the server may rearrange a plurality of product objects one by one in the order of high density of visitors to locations within the metaverse pop-up store corresponding to each of the plurality of locations, according to the order of high sales volume of offline products. For example, if there are locations a and b in the metaverse pop-up store corresponding to exhibition location A with the highest density of visitors and exhibition location B with the next highest density of visitors, the server will select the product with the highest sales volume. Product object x corresponding to

For example, the above-described relocation operation may be performed after a second point in time, which is when the business hours of the offline pop-up store end, and the next relocation operation may be performed after the second point in time, when the business hours of the pop-up store end on the next day offline. It can be done. That is, the above-described relocation operation may be performed periodically after the second time point.

Or, for example, the relocation operation described above may be performed every 6 hours.

According to one embodiment, the server may collect rating information for offline sales products based on sales information through web crawling. For example, rating information for offline sales products may include review information and ratings related to offline sales products posted on a plurality of SNS (social network service) channels and a plurality of online shopping mall web pages.

The server can obtain rating information for online products sold in the Metaverse pop-up store and real-time information on online visitors. For example, rating information for online sales products may include review information and ratings related to online products sold by online visitors who purchased online from the Metaverse pop-up store.

Here, an online visitor may be a user who accesses the Metaverse pop-up store and visits the Metaverse pop-up store online. For example, online visitors can visit the Metaverse pop-up store through a user terminal. The first virtual avatar and the second virtual avatar may be virtual non-player characters (NPCs) that are distinct from online visitors.

The user terminal may include a terminal capable of providing at least one technology/service among virtual reality, augmented reality, mixed reality, and extended reality including these. For example, the user terminal may include the electronic device 101 of FIG. 1, and may include a head mounted display (head mounted display), which is a display device that is connected to the electronic device 101 of FIG. 1 and outputs a VR/AR image. HMD), Google Glass, etc. may be included, but are not limited thereto. Various devices capable of implementing realistic images such as VR/AR/MR/XR, which are already known or developed in the future, can be applied to these user terminals, which are devices that support users to experience realistic images. For example, the user terminal may be the electronic device 101 of FIG. 1 worn by the user and connected to a device that outputs a metaverse pop-up store. The user terminal can be connected to the Metaverse pop-up store through a virtual experience program that provides a shopping mall service based on virtual reality (VR)-based virtual space surfing installed on the user terminal.

Real-time information on online visitors includes the entry time of multiple online visitors, the number of multiple online visitors, values for gender and age group of multiple online visitors, and location coordinates for each time section of multiple online visitors within the Metaverse pop-up store. may include.

The server may determine, among a plurality of sold products, a sold product whose sum of the first score and the second score is equal to or greater than a preset score as a recommended product. The plurality of products for sale may be all products sold in offline pop-up stores and online pop-up stores.

The first score may be determined based on real-time information of the visitor and real-time information of the online visitor. For example, the first score may be a sum of interest in offline sales products and interest in product objects.

The level of interest in offline sales products can be determined based on the density of visitors to offline sales products. For example, the server may determine the density of visitors within a preset distance from the location of the offline product in the product display area in an offline pop-up store as the degree of interest in the offline product.

The level of interest in a product object may be determined based on the density of online visitors to the product object corresponding to an online sold product. For example, the server may determine the density of online visitors within a preset distance from the location of the product object placed in the metaverse pop-up store as the degree of interest in the product object.

The second score may be determined based on rating information for offline-sold products and rating information for online-sold products. For example, the second score may be a sum of the average rating of products sold offline and the average rating of products sold online.

The average rating of offline-sold products is the average of the ratings of offline-sold products posted on multiple SNS channels and multiple online shopping mall web pages, and is weighted by analyzing positive and negative keywords included in review information related to offline-sold products. It may be a value applied. The average rating of an online sold product may be a weighted value obtained by analyzing the positive and negative keywords included in review information related to the online sold product to the average rating of the online sold product.

The server can transmit information about recommended products to a display terminal placed in an offline pop-up store. For example, a display terminal may be an electronic device that displays information or images on a screen, and may include a monitor, digital signage, a display panel, etc. For example, the server may be pre-connected with the display terminal.

For example, the display terminal can display in real time the number of visitors to the offline pop-up store, the number of visitors to the metaverse pop-up store, user reviews, ratings, number of recommendations, sales amount for each product, and sales quantity for each product.

For example, information about recommended products may be displayed on the screen of the display terminal and may be displayed in the display area arranged in the metaverse pop-up store. Information about the recommended product may include the name of the recommended product, an image of the recommended product, and an advertisement image for the recommended product. The display area placed in the metaverse pop-up store may be set at a location corresponding to the location of the display terminal placed in the offline pop-up store.

For example, information about recommended products may be displayed on the screen of a display terminal at a preset period and may be displayed in a display area arranged in a metaverse pop-up store.

For example, a preset effect may be applied to a product object corresponding to a recommended product. Multiple effects for product objects may be preset on the server. The server may apply preset effects for the recommended product to the product object corresponding to the recommended product. For example, the preset effect may be determined as the effect most used by online visitors using the metaverse pop-up store.

Additionally, for example, a plurality of tracking areas are determined according to the size and shape of the offline pop-up store, and the number and location of cameras provided in each tracking area may be determined differently. At this time, the movement lines of visitors can be predicted in advance for each zone included based on the drawing information about the offline pop-up store. For example, the server can determine the layout of the offline pop-up store based on drawing information about the offline pop-up store and predict the number of visitors for each zone in advance. For example, the server can determine the priority for installing cameras in areas predicted to be crowded with visitors, such as entrances, main passages, POS, and product shelves included in an offline pop-up store. Cameras may be installed in each of a plurality of tracking areas according to these priorities. Drawing information about the pop-up store may include at least one of a floor plan, layout plan, elevation view, or cross-sectional view.

Figure 4 shows an example of a movement line prediction model according to an embodiment. The embodiment of FIG. 4 can be combined with various embodiments of the present disclosure.

Referring to FIG. 4, the movement prediction model may use a neural network based on a gated recurrent unit (GRU). That is, the movement line prediction model may include a GRU-based third neural network. Here, GRU may be a modified model of RNN (recurrent neural network).

For example, the server may generate a time vector, a visitor vector, and a movement vector for each tracking area through third-party data preprocessing of real-time visitor information generated during a preset time period. For example, a time vector may include a value for a first time point and a value for a second time point. For example, a visitor vector may include values related to the number of visitors by time interval and the gender and age group of the visitors by time interval. For example, the movement vector may include location coordinates for each of a plurality of visitors for each time section.

The third neural network may include a third input layer 410, one or more third hidden layers 420, and a third output layer 430. For example, each learning data consisting of a plurality of time vectors, a plurality of visitor vectors, a plurality of movement vectors, a plurality of correct answer non-real visitor vectors, and a plurality of correct answer non-real visitor movement vectors are transmitted in the third neural network. 3 It is input to the input layer 410, passes through the one or more third hidden layers 420 and the third output layer 430, and is output as a third output vector, and the third output vector is the third output layer ( 430), and the third loss function layer uses a third loss function that compares the third output vector with the third correct vector for each training data to obtain a third loss value. is output, and the parameters of the third neural network can be learned in a direction in which the third loss value decreases.

That is, the server may learn a movement line prediction model to predict the virtual visitor and the movement line for the virtual visitor for a time other than the time vector based on the correlation between the plurality of visitor vectors and the plurality of movement line vectors.

Specifically, for example, one or more third hidden layers may include one or more GRU blocks, and one GRU block may include a reset gate and an update gate. Here, the reset gate and update gate may include a sigmoid layer. For example, a sigmoid layer uses a sigmoid function ( ) may be a layer that is an activation function. For example, a hidden state may be controlled through a reset gate and an update gate, and weights may exist for each gate and input.

The reset gate resets past information, and the weight r(t) derived through the previous hidden layer can be determined by Equation 1.

For example, through third data preprocessing of real-time information on visitors generated during a preset time period, time vectors, visitor vectors, and movement vectors for each tracking area are input to the third input layer, and the reset gate is When the input value (x _t ) of the current time created based on the time vector, visitor vector, and movement vector is entered, it is dot producted with the weight W _r of the current time, and the previous time vector generated based on the time vector, visitor vector, and movement vector is The hidden state (h _(t-1) ) at the time point is dot producted with the weight U _r at the previous time point, and finally, the two values are added together and input into the sigmoid function, and the result can be output as a value between 0 and 1. Through these values between 0 and 1, it can be determined how much of the hidden state value from the previous point will be utilized.

The update gate determines the update rate for past and present information, and z(t) is the amount of information at the current time and can be determined by Equation 2.

For example, when the input value (x _t ) at the current time is input, it is dot producted with the weight W _z at the current time point, and the hidden state (h _(t-1) ) at the previous time point is the dot product with the weight U _z at the previous time point. And finally, the two values are added together and input into the sigmoid function, so the result can be output as a value between 0 and 1. And 1-z(t) can be multiplied by the information (h _(t-1) ) of the hidden layer at the previous time.

Through this, z(t) can reflect how much current information will be used and 1-z(t) can reflect how much past information will be used.

By multiplying the result of the reset gate, the information candidate group at the current time t can be determined by Equation 3.

For example, when the input value (x _t ) at the current time is input, the weight W _h at the current time and the dot product value, and the hidden state at the previous time (h _(t-1) ) is the weight U _h at the previous time. It can be input to the tanh function by taking the dot product and adding the value multiplied by r(t). For example, tanh stands for nonlinear activation function (hyperbolic tangent function).

By combining the results of the update gate and the candidate group, the weight of the hidden layer at the current time can be determined by Equation 4.

For example, the output value z(t) of the update gate multiplied by the current hidden state (h(t)), the value discarded from the update gate 1-z(t) and the hidden state at the previous time (h( The weight of the hidden layer at the current time can be determined by the sum of the values multiplied by t-1)).

For example, the weight initialization for the third neural network used in the movement prediction model is, for each layer, divided by the sum of the number of input values input to the layer and the number of output values output from the layer. It can be performed based on weights. Therefore, the starting point of the weight can be set to a value within an appropriate range.

Therefore, the server can use the parameters of the third neural network learned through the movement prediction model, and the server considers the past visitor vector and the past movement vector to determine the non-real visitor vector and the non-real visitor vector for times other than the time vector. The movement vector can be determined.

Figure 5 is a flowchart showing a method in which a server manages a metaverse pop-up store based on information related to an offline pop-up store using a neural network according to an embodiment. The embodiment of FIG. 5 can be combined with various embodiments of the present disclosure.

Referring to FIG. 5, in step S501, the server may receive video information in real time from a plurality of cameras provided in a plurality of tracking areas of the offline pop-up store.

For example, image information may be classified by multiple tracking areas. The plurality of tracking areas may include a visitor entrance area, at least one movement passage area, at least one product display area, and a product purchase area.

In step S502, the server may receive sales information from a POS terminal installed in an offline pop-up store.

For example, the server can receive sales information in real time from a POS terminal installed in an offline pop-up store and store the sales information by reception time.

In step 503, the server may collect rating information for offline sales products based on sales information through web crawling.

In step S504, the server may obtain rating information of online sales products sold in the metaverse pop-up store and real-time information of online visitors.

In step S505, the server may determine real-time information about the visitor through a tracking model using a first neural network and a second neural network based on image information about a plurality of tracking areas.

For example, the server may generate a face image vector for each time section through first data preprocessing on image information of the visitor entrance area.

For example, when the server inputs face image vectors for each time section into the above-described first neural network, values for the gender and age group of visitors who visited the offline pop-up store may be output.

For example, the server may generate a plurality of image vectors for each time section for each tracking area through second data preprocessing on image information of the plurality of tracking areas.

For example, when the server inputs a plurality of image vectors for each time section for each tracking area into the above-described second neural network, the location coordinates for each time section of a plurality of visitors included in each tracking area can be output. there is.

In step S506, the server may create a plurality of first virtual avatars in the metaverse pop-up store based on the visitor's real-time information.

For example, each of the plurality of first virtual avatars may move along the movement path of the visitor matched to the first virtual avatar.

In step S507, the server may determine, among a plurality of sold products, a sold product whose sum of the first score and the second score is equal to or greater than a preset score as a recommended product.

For example, the first score may be determined based on real-time information of the visitor and real-time information of the online visitor.

For example, the second score may be determined based on rating information for offline-sold products and rating information for online-sold products.

In step S508, the server may transmit information about recommended products to a display terminal placed in an offline pop-up store.

For example, information about recommended products may be displayed on the screen of a display terminal placed in an offline pop-up store, and may be displayed in a display area placed in a metaverse pop-up store.

For example, a preset effect may be applied to a product object corresponding to a recommended product.

In step S509, the server may determine whether the second time point has passed.

In step S510, when the second time point has elapsed, the server may generate information about the non-actual visitor through a movement prediction model using a third neural network based on real-time information about the visitor generated during a preset time period. .

The preset time section may be a section between a first point in time when the offline pop-up store starts operating and a second point in time when the offline pop-up store ends.

In step S511, the server may generate a plurality of second virtual avatars in the metaverse pop-up store based on information about the non-actual visitor from the second point in time.

For example, each of the plurality of second virtual avatars may move along the movement path of the non-real visitor matched with the second virtual avatar.

In step S512, the server may rearrange the positions of a plurality of product objects placed in the metaverse pop-up store based on sales information.

According to one embodiment, the server may generate a plurality of first virtual avatars without generating a plurality of second virtual avatars from the first point in time. That is, a plurality of first virtual avatars may be created between the first and second viewpoints, and a plurality of second virtual avatars may be created between the first and second viewpoints.

According to one embodiment, the server may receive information related to the idle space from each of a plurality of manager terminals.

The administrator terminal may be a terminal used by the administrator who manages the idle space. For example, the administrator terminal may provide information related to the idle space to the server and receive information about companies that want to install a pop-up store in the idle space. For example, the manager terminal may include the electronic device 101 of FIG. 1 .

Information related to idle space may include information about the location containing the idle space, information about the surrounding floating population, information about visitors, information about the surrounding traffic environment, information related to rental, and information related to the size of the idle space. You can.

Information about a place containing an idle space may include the type of place, the location of the place, the location of the idle space within the place, and the size of the parking space. Here, the type of place may include a hotel, a shopping mall, a tourist attraction, and a site. The information on the surrounding floating population is information showing the average monthly floating population by age group for an area within a preset distance from the location of the place containing the idle space, and may include information uploaded to a web page related to the National Statistical Office. Information about visitors may include the average daily number of visitors by age group and gender ratio of visitors to a location containing an idle space. The information about the surrounding traffic environment may be information indicating the traffic environment around a place containing an idle space. For example, information about the surrounding traffic environment may include the distance from the location containing the idle space to bus stops, subway stations, and train stations, and the distance from the location containing the idle space to major roads. Here, the main road may be a road with an average daily traffic volume greater than or equal to a preset value. Information related to the lease may include the rent of the idle space and the rental period of the idle space. Information related to the size of the idle space may include the contracted area of the idle space and the number of people that can be accommodated in the idle space.

The server may receive information related to the pop-up store from each of a plurality of corporate terminals.

For example, a corporate terminal may provide information related to a pop-up store to a server and receive information about an idle space suitable for the pop-up store. Information related to the pop-up store may include the name of the brand, information about the customer base, information about the budget, information about products related to the pop-up store, and information about the pop-up store.

The name of the brand may be the name of the brand of the product to be sold in the pop-up store. The information about the customer base is information about the target customer base of the pop-up store, and the information about the customer base may include information about the age group of the customer base and the gender of the customer base. The information about the budget is information about the budget for renting idle space, and may include desired rent and desired rental period for the idle space. Information about products related to a pop-up store is information about products to be sold at the pop-up store. Information about products related to the pop-up store may include the type of product and the price range of the product. Information about the pop-up store may include the location of the pop-up store, information about the type of desired location, the required area of the pop-up store, and the expected number of visitors to the pop-up store.

The server may determine a preset number of recommended idle spaces for each of a plurality of corporate terminals based on information related to the idle space and information related to the pop-up store.

For example, an idle space vector may be generated for each of a plurality of manager terminals through data preprocessing of information related to the idle space. The idle space vector includes the average number of monthly foot traffic, the average number of visitors per day, values related to the age range of visitors, gender ratio of visitors, values related to the surrounding transportation environment, values for rent, values for rental period, and type of location. It may include a value for , a value related to the size of the idle space, and a value related to a keyword for the location containing the idle space.

The value related to the visitor's age group is a value for the ratio of visitors by age group, and the ratio of visitors by age group can be determined based on the number of visitors of each age group. For example, values related to the visitor's age range may include percentage values for each age group.

A value related to the surrounding traffic environment may be a value representing the surrounding traffic environment. For example, values related to the surrounding traffic environment may include a first distance value and a second distance value. The first distance value may be determined as the closest distance among the distances from the place containing the idle space to the bus stop, subway station, and train station. The second distance value may be determined as the distance from the place containing the idle space to the main road.

The value for the type of place is a value that indicates whether the place containing the idle space is a hotel, shopping mall, tourist attraction, or site. Values related to the size of the idle space may include values for the contracted area of the idle space and the maximum number of people that can be accommodated in the idle space.

Additionally, for example, a value related to a keyword for a place containing an idle space may be a value determined based on keywords related to a place containing an idle space collected through web crawling. For example, the server can collect keywords related to the place containing the idle space by crawling the web for a preset period of time, from the text position containing the name of the place containing the idle space, and keywords located within a preset text range. there is. For example, if the preset period is 6 months, the server may collect keywords located within the preset text range from text positions containing places containing idle space through web crawling for the last 6 months.

For example, the server can determine x and y coordinates that match keywords related to the location containing the idle space. Specifically, the server may preset a circle with a specific radius on x and y coordinates, and divide the circle into a first axis, a second axis, and a third axis, each spaced at 120 degree intervals. Here, the first axis represents luxury, the second axis represents innovation, and the third axis represents tradition. For example, words representing luxury and innovation (e.g., prestige, authority, sophistication, future-oriented, contemporary) may be placed in the first area between the first and second axes, and the second area may be located in the first area. Words representing innovation and tradition (e.g. evolution, newtro, curiosity, fantasy) can be placed in the second area between the axis and the third axis, and in the third area between the third axis and the first axis. Words representing tradition and luxury (e.g., classic, grand, prime, elegance) can be positioned. That is, the server may preset coordinates that match a plurality of words to the first to third areas.

For example, the server determines the most dense area among the x and y coordinates that match keywords related to the place containing the idle space, and calculates the center of gravity coordinates for the x and y coordinates included in the most dense area. The space can be determined by the value related to the keyword for the place containing it.

For example, a pop-up store vector may be generated for each of a plurality of corporate terminals through data preprocessing of information related to the pop-up store. The pop-up store vector contains values related to the age range of customers, gender ratio of customers, values related to type of location, values related to budget, values related to desired rental period, values related to the size of the pop-up store, and values related to keywords for the brand. may include.

The value related to the customer's age group is a value for the ratio of the customer's age group, and the ratio of the customer's age group can be determined based on the number of customers in each age group. For example, values related to the customer's age range may include percentage values for each age group. The value related to the type of place may be a value indicating preference for each of the hotel, shopping mall, tourist attraction, and site. Preference can have values greater than 0 and less than 100.

Values related to the size of the pop-up store may include a value for the required area of the pop-up store and the expected number of visitors to the pop-up store.

Additionally, for example, a value related to a keyword for a brand may be a value determined based on keywords related to the brand collected through web crawling. For example, the server may collect keywords related to the brand, including keywords located within a preset text range from the text position containing the brand name, through web crawling for a preset period. For example, if the preset period is 6 months, the server can collect keywords located within the preset text range from the text position containing the brand name through web crawling for the last 6 months. For example, the server determines the most dense area among the x and y coordinates that match keywords related to the brand, and calculates the center of gravity coordinates for the x and y coordinates included in the most dense area to be related to the keywords for the brand. It can be determined by value.

For example, the server may determine a preset number of recommended idle spaces for each of a plurality of corporate terminals through a pop-up store brokerage model using a neural network based on information related to the idle space and information related to the pop-up store.

For example, based on the idle space vector for each of the plurality of administrator terminals and the pop-up store vector for each of the plurality of enterprise terminals being input to the pop-up store mediation model, the matching rate for each of the plurality of idle spaces is calculated for each of the plurality of administrator terminals. can be decided. The plurality of idle spaces may be idle spaces received from a plurality of manager terminals.

For example, among a plurality of idle spaces, a preset number of recommended idle spaces may be determined in order of high matching rate.

For example, the matching rate may be determined based on a first similarity related to rental, a second similarity related to customer, and a third similarity related to keyword.

The neural network used in the pop-up store brokerage model may include an input layer, one or more hidden layers, and an output layer.

Each training data consisting of a plurality of idle space vectors, a plurality of pop-up store vectors, and a plurality of correct matching rates is input to the input layer, passes through one or more hidden layers and an output layer, and is output as an output vector, and the output vector is output to the output layer. It is input to a loss function layer connected to, and the loss function layer outputs a loss value using a loss function that compares the output vector and the correct answer vector for each learning data, and the parameters of the neural network used in the pop-up store mediation model. can be learned in a way that the loss value becomes smaller.

For example, the plurality of correct answer matching rates may include the correct answer matching rate for each of one idle space vector among the plurality of idle space vectors and one pop-up store vector among the plurality of pop-up store vectors.

For example, the correct answer matching rate may be determined based on a first similarity related to rental, a second similarity related to customer, and a third similarity related to keyword.

For example, the first similarity related to rental may be a value obtained by applying a first weight to the similarity between the first part of the idle space vector and the first part of the pop-up store vector. For example, the first part of the idle space vector may consist of a value for the rent, a value for the lease period, and a value related to the size of the idle space. For example, the first part of the pop-up store vector may consist of values related to the budget, a value for the desired rental period, and a value related to the size of the pop-up store. For example, the first weight may be determined based on the preference corresponding to the type of place containing the idle space among the preferences for each of hotels, shopping malls, tourist attractions, and sites. For example, the first weight may be a value greater than 0 and less than or equal to 1. For example, if the type of place including the idle space is a hotel, and the preference for the hotel included in the value related to the type of place is 80, the first weight may be determined to be 0.8.

Through this, the server can improve the probability of recommending an idle space suitable for the corporate terminal by determining the first similarity by reflecting the preference for the location desired by the corporate terminal.

For example, the second similarity related to the customer may be a value obtained by applying a second weight to the similarity between the second portion of the idle space vector and the second portion of the pop-up store vector. For example, the second part of the idle space vector may consist of values related to the age range of visitors and the gender ratio of visitors. For example, the second part of the pop-up store vector may consist of values related to the age range of customers and the gender ratio of customers. For example, the second weight may be determined based on values for the average number of monthly foot traffic, the average number of visitors per day, and the contracted area of idle space. For example, the second weight may be a value greater than 0 and less than or equal to 1. For example, a first value that is the average number of monthly foot traffic divided by the value for the contracted area of idle space and a second value that is the sum of the average number of visitors per day divided by the value for the contracted area of idle space. 3 The larger the value, the larger the second weight can be determined.

Additionally, for example, the second weight may be determined by Equation 1 below.

In Equation 1, w ₂ is the second weight, n is the number of a plurality of idle spaces, P1 _i is the number of monthly average floating population of the i-th idle space, and P2 _i is the i-th is the average number of visitors per day in the idle space, where a _i is the contracted area of the ith idle space, P1 is the average number of monthly floating population in the idle space, and P2 is the average number of visitors per day in the idle space. , and a may be the contracted area of the corresponding idle space.

For example, the min function is a function that determines the smaller of the two values in parentheses.

Through this, the server can reflect the number of visitors to the idle space and determine the second similarity, which is the preference for the customer base, to improve the probability of recommending a suitable idle space to the corporate terminal.

For example, the third similarity related to the keyword may be the similarity between the value associated with the keyword for a place containing an idle space and the value associated with the keyword for a brand.

For example, similarity can be determined as the distance value between vectors. At this time, the distance value between vectors may be determined as at least one of Euclidean distance, Manhattan distance, cosine distance, and Hamming distance. Euclidean distance represents the straight line distance between two vectors. It can be calculated through . Manhattan distance represents the distance for the i-th dimension of a vector to reach a point on a grid-shaped map. It can be calculated through . Cosine distance is a value that indicates the degree to which the cosine angles between two vectors are similar. It can be calculated through . Here, m may be the number of dimensions constituting the vector, pj may be the value of the jth dimension of the idle space vector, and qj may be the value of the jth dimension of the pop-up store vector.

For example, the correct answer matching rate may be determined as the sum of the first similarity related to rental, the second similarity related to customer, and the third similarity related to keyword.

The server may transmit information about a preset number of recommended idle spaces to each of a plurality of corporate terminals.

Information about the recommended idle space may include information related to the idle space, a matching rate for the idle space, a first similarity to the idle space, a second similarity to the idle space, and a third similarity to the idle space.

Additionally, for example, the preset number may be determined by Equation 6 below.

In Equation 6, the N _space is the preset number, and is a weight according to the budget of the corporate terminal, N _m is the maximum number of corporate terminals that can be connected to the server, N _a is the average number of corporate terminals connected to the server, and t _r is the corporate terminal connected to the server. is the average response time, t _i is the recommended response time, the RSRP is a value related to the communication state of the enterprise terminal, the RSRP _i is a recommended value for the value related to the communication state, and the N _d may be a default value. there is.

For example, recommended values and default values for the maximum number of corporate terminals that can connect to a server, recommended response time, and values related to communication status may be values pre-stored in the server. For example, the default value may be set to a value of 3 or higher. For example, the average response time for corporate terminals connected to the server and the average number of corporate terminals connected to the server may be values measured at preset intervals.

For example, the weight according to the budget of a corporate terminal is a value greater than 0 and less than 1. The weight according to the budget of the enterprise terminal may be determined to be greater as the budget value increases. RSRP (reference signal received power) is a value indicating the communication status of a corporate terminal and may have a negative dB value. For example, the larger the absolute value of RSRP, the poorer the communication status of the corporate terminal may be determined.

For example, the min function is a function that determines the smaller of the two values in parentheses. For example, the ceil function is a function that omits decimal places from the value in parentheses and determines the value as the next integer, and may be a rounding function.

Through this, the server adjusts the preset number by considering not only the budget of the corporate terminal, but also the performance of the server and the communication status with the corporate terminal, allowing the recommendation function to be performed smoothly even in situations where multiple corporate terminals are connected at the same time. .

According to one embodiment, when the lease contract for the offline pop-up store ends, the Metaverse pop-up store may be managed based on real-time visitor information and sales information received during the lease contract period for the offline pop-up store.

Figure 6 is a block diagram showing the configuration of a server according to one embodiment. One embodiment of FIG. 6 may be combined with various embodiments of the present disclosure.

As shown in FIG. 6, the server 600 may include a processor 610, a communication unit 620, and a memory 630. However, not all of the components shown in FIG. 6 are essential components of the server 600. The server 600 may be implemented with more components than those shown in FIG. 6, or the server 600 may be implemented with fewer components than those shown in FIG. 6. For example, the server 600 according to some embodiments may further include a user input interface (not shown), an output unit (not shown), etc. in addition to the processor 610, the communication unit 620, and the memory 630. .

The processor 610 typically controls the overall operation of the server 600. The processor 610 may include one or more processors and control other components included in the server 600. For example, the processor 610 can generally control the communication unit 620 and the memory 630 by executing programs stored in the memory 630. Additionally, the processor 610 may perform the functions of the server 600 shown in FIGS. 3 to 5 by executing programs stored in the memory 630.

The communication unit 620 may include one or more components that allow the server 600 to communicate with other devices (not shown) and servers (not shown). The other device (not shown) may be a computing device such as the server 600 or a sensing device, but is not limited thereto. The communication unit 620 may receive a user input from another electronic device or receive data stored in an external device from an external device through a network.

For example, the communication unit 620 may transmit and receive a message to establish a connection with at least one device. The communication unit 620 may transmit information generated by the processor 610 to at least one device connected to the server. The communication unit 620 may receive information from at least one device connected to the server. The communication unit 620 may transmit information related to the received information in response to information received from at least one device.

The memory 630 may store programs for processing and control of the processor 610. For example, the memory 630 may store information input to a server or information received from another device through a network. Additionally, the memory 630 may store data generated by the processor 610. The memory 630 may store information input to or output from the server 600.

The memory 630 is a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (for example, SD or XD memory, etc.), and RAM. (RAM, Random Access Memory) SRAM (Static Random Access Memory), ROM (Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), PROM (Programmable Read-Only Memory), magnetic memory, magnetic disk , and may include at least one type of storage medium among optical disks.

The embodiments described above may be implemented with hardware components, software components, and/or a combination of hardware components and software components. For example, the devices, methods, and components described in the embodiments may include, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, and a field programmable gate (FPGA). It may be implemented using one or more general-purpose or special-purpose computers, such as an array, programmable logic unit (PLU), microprocessor, or any other device capable of executing and responding to instructions. A processing device may execute an operating system (OS) and one or more software applications that run on the operating system. Additionally, a processing device may access, store, manipulate, process, and generate data in response to the execution of software. For ease of understanding, a single processing device may be described as being used; however, those skilled in the art will understand that a processing device includes multiple processing elements and/or multiple types of processing elements. It can be seen that it may include. For example, a processing device may include a plurality of processors or one processor and one controller. Additionally, other processing configurations, such as parallel processors, are possible.

Software may include a computer program, code, instructions, or a combination of one or more of these, which may configure a processing unit to operate as desired, or may be processed independently or collectively. You can command the device. Software and/or data may be used on any type of machine, component, physical device, virtual equipment, computer storage medium or device to be interpreted by or to provide instructions or data to a processing device. , or may be permanently or temporarily embodied in a transmitted signal wave. Software may be distributed over networked computer systems and stored or executed in a distributed manner. Software and data may be stored on one or more computer-readable recording media.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc., singly or in combination. Program instructions recorded on the medium may be specially designed and configured for the embodiment or may be known and available to those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floptical disks. -Includes optical media (magneto-optical media) and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, flash memory, etc. Examples of program instructions include machine language code, such as that produced by a compiler, as well as high-level language code that can be executed by a computer using an interpreter, etc. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

Although the embodiments have been described with limited drawings as described above, those skilled in the art can apply various technical modifications and variations based on the above. For example, the described techniques are performed in a different order than the described method, and/or components of the described system, structure, device, circuit, etc. are combined or combined in a different form than the described method, or other components are used. Alternatively, appropriate results may be achieved even if substituted or substituted by an equivalent.

Therefore, other implementations, other embodiments, and equivalents of the claims also fall within the scope of the claims described below.

Claims (5)

In a method for a server to manage a metaverse pop-up store based on information related to an offline pop-up store using a neural network,
Receiving video information in real time from a plurality of cameras provided in a plurality of tracking areas of the offline pop-up store;
determining real-time information about a visitor through a tracking model using a first neural network and a second neural network based on the image information for the plurality of tracking areas;
generating a plurality of first virtual avatars in the metaverse pop-up store based on real-time information of the visitor;
Each of the plurality of first virtual avatars moves along the movement line of the visitor matched to the first virtual avatar,
Generating information about non-actual visitors through a movement prediction model using a third neural network based on real-time information about the visitor generated during a preset time period;
The preset time section is a section between a first time point when the offline pop-up store starts business and a second time point when the offline pop-up store ends business,
generating a plurality of second virtual avatars in the metaverse pop-up store based on information about the non-actual visitor from the second time point onward;
Each of the plurality of second virtual avatars moves along the movement line of the non-real visitor matched with the second virtual avatar,
The image information is classified by the plurality of tracking areas,
The plurality of tracking areas include a visitor entrance area, at least one movement passage area, at least one product display area, and a product purchase area,
Face image vectors for each time section are generated through first data preprocessing of the image information of the visitor entrance area,
Based on the face image vector for each time section being input to the first neural network, values for the gender and age group of the visitor who visited the offline pop-up store are output,
The number of visitors is counted based on the values for the visitor's gender and age range,
A plurality of image vectors are generated for each time section for each tracking area through second data preprocessing of the image information of the plurality of tracking areas,
Based on the plurality of image vectors for each time section for each tracking area being input to the second neural network, the location coordinates for each time section of the plurality of visitors included in each tracking area are output,
The real-time information of the visitors includes the entry time of multiple visitors, the number of multiple visitors, values for gender and age group of multiple visitors, and location coordinates for each time section of multiple visitors included in each tracking area. do,
Receiving sales information from a POS terminal installed in the offline pop-up store;
rearranging a plurality of product objects arranged in the metaverse pop-up store based on the sales information;
Collecting rating information for offline sales products based on the sales information through web crawling;
Obtaining rating information of online products sold in the metaverse pop-up store and real-time information of online visitors;
determining, among a plurality of sold products, a sold product whose sum of the first score and the second score is equal to or greater than a preset score as a recommended product; and
The first score is determined based on real-time information of the visitor and real-time information of the online visitor,
The second score is determined based on the rating information of the offline-sold product and the rating information of the online-sold product,
A step of transmitting information about the recommended product to a display terminal placed in the offline pop-up store,
The sales ranking of all product objects placed in the metaverse pop-up store is determined based on the sales information,
The plurality of product objects include product objects up to a preset sales rank among all product objects,
A plurality of locations are determined in order of high density of visitors based on the location coordinates of each time section of the plurality of visitors to the at least one product display area,
The plurality of product objects are sequentially relocated to locations within the metaverse pop-up store corresponding to each of the plurality of locations,
Information about the recommended product is displayed on the screen of the display terminal and in a display area arranged in the metaverse pop-up store,
A preset effect is applied to the product object corresponding to the recommended product,
method.
delete According to clause 1,
A time vector, a visitor vector, and a movement vector for each tracking area are generated through third data preprocessing of the visitor's real-time information generated during the preset time period,
The time vector includes a value for the first time point and a value for the second time point,
The visitor vector includes values related to the number of visitors by time section and the gender and age group of the visitors by time section,
The movement vector includes location coordinates for each of a plurality of visitors for each time section,
Based on the time vector, the visitor vector, and the movement vector for each tracking area being input to the third neural network, values for the gender and age range of a plurality of non-real visitors and included in each tracking area The location coordinates for each time section of the non-real visitor are output,
The information on the non-real visitors includes the scheduled entry time of the plurality of non-real visitors, the number of the plurality of non-real visitors, the gender and age range of each of the plurality of non-real visitors, and the plurality of non-real visitors included in each tracking area. Containing location coordinates for each time interval,
method. delete delete

Images