KR20230110450A - Method and apparatus for transmitting information on recommended accessories to a user terminal using a neural network - Google Patents

Abstract

Embodiments suggest a method and apparatus for transmitting information about a recommended accessory to a user terminal using a neural network. According to an embodiment, the method receives user appearance information including a first image of the user wearing clothes and a second image of the user's face and basic information of the user from the user terminal, determines characteristic information of the user through a data extraction model using a first neural network based on the user's appearance information, determines an accessory candidate list from accessory information pre-stored in the server based on the user characteristic information and the user basic information, and sends a collection request message based on the determined accessory candidate list. based on receiving a collection permission message from the user terminal, collecting the user's purchase history information and the user's social network service (SNS) activity information, determining the user's preference for each accessory product through a preference analysis model using a second neural network based on the user's purchase history information and the user's SNS activity information, determining a recommended accessory from the accessory candidate list based on the user's preference for each accessory product and a trend score for each accessory product, and transmitting information on the recommended accessory to the user terminal. can include For example, the trend score for each accessory product may be determined according to the number of posts including keywords related to accessories among a plurality of posts generated on a plurality of SNSs during a preset period.

Description

Method and apparatus for transmitting information on recommended accessories to a user terminal using a neural network

Embodiments of the present disclosure relate to a technology for transmitting information about a recommended accessory to a user terminal, and to a method and apparatus for transmitting information about a recommended accessory to a user terminal using a neural network.

Meanwhile, with the development of the Internet and wireless communication technology, electronic commerce in which products are bought and sold in online shopping malls using the Internet has become commonplace. In conventional e-commerce, product recommendation algorithms predict customer tastes and provide recommended products according to customer tastes, and the process of predicting customer tastes is important.

In particular, as big data and AI (artificial intelligence) are introduced into recommendation algorithms, it can be easier to predict customer preferences if data on customers is secured. However, since the customer's data is personal information and can be used only when the customer permits the collection of personal information, it may be difficult to predict the customer's taste when the customer does not permit the collection of personal information. In addition, even if the customer permits the collection of personal information, it may be difficult to predict the customer's taste if the customer's online activity data is insufficient.

Meanwhile, in the case of accessory products, accessories suitable for each customer may be different depending on the customer's body shape, face shape, fashion style, hair style, and the like. Since most customers purchase and wear the accessory product they prefer without considering their overall appearance or face shape, customer satisfaction may be low when purchasing an accessory product.

Accordingly, by receiving an image of the user wearing clothing and an image of the user's face from the user terminal, an accessory candidate suitable for the user is determined through a neural network. If the user does not allow the collection of personal information, information on a recommended accessory determined from among accessory candidates based on a trend according to social network service (SNS) information is transmitted to the user terminal. I need a way to send it to In addition, even when the user allows the collection of personal information, if personal information for determining recommended accessories is insufficient, a method of transmitting information on recommended accessories to the user terminal by receiving information for identifying the user's taste is required.

Embodiments of the present disclosure may provide a method and apparatus for transmitting information about a recommended accessory to a user terminal using a neural network.

Technical tasks to be achieved in the embodiments are not limited to those mentioned above, and other technical tasks not mentioned may be considered by those skilled in the art from various embodiments to be described below.

A method in which a server transmits information on recommended accessories to a user terminal using a neural network according to an embodiment includes receiving external appearance information of a user including a first image of the user wearing clothes and a second image of the user's face and basic information of the user from the user terminal, determining characteristic information of the user through a data extraction model using a first neural network based on the external appearance information of the user, and selecting an accessory from accessory information pre-stored in the server based on the characteristic information of the user and the basic information of the user. Determine a list, based on the determined accessory candidate list, transmit a collection request message to the user terminal, collect user purchase history information and user social network service (SNS) activity information based on receiving a collection permission message from the user terminal, determine the user's preference for each accessory product through a preference analysis model using a second neural network based on the user's purchase history information and the user's SNS activity information, and determine the user's preference for each accessory product and the accessory product based on the user's preference for each accessory product and trend scores for each accessory product. The method may include determining a recommended accessory from a candidate list and transmitting information on the recommended accessory to the user terminal. For example, the trend score for each accessory product may be determined according to the number of posts including keywords related to accessories among a plurality of posts generated on a plurality of SNSs during a preset period.

According to an embodiment, the data extraction model may be a faster regions with convolutional neural network (R-CNN) model. A first input vector composed of a plurality of first regions of interest may be generated through data preprocessing of the first image. The plurality of first regions of interest may include a region of interest related to the user's hair color, a region of interest related to the user's hair style, a region of interest related to the user's body shape, a region of interest related to the user's neck, and a region of interest related to the user's fashion style. A second input vector composed of a plurality of second regions of interest may be generated through data preprocessing of the second image. The plurality of second regions of interest may include a region of interest related to the user's skin tone, a region of interest related to the user's ear, and a region of interest related to the user's face shape. Characteristic information of the user may be determined based on input of the first input vector and the second input vector to the first neural network. The user's characteristic information may include a value related to the user's skin tone, a value related to the user's hair color, a value related to the user's hair style, a value related to the user's body type, a value related to the user's fashion style, a value related to the user's face shape, a value related to the user's ears, and a value related to the user's neck.

According to an embodiment, the preference analysis model may use a multivariate long short term memory networks (LSTM) model. The user's purchase history information may include information on the user's purchased products, rating information on the user's purchased products, the number of clicks on the user's purchased products, whether or not the user has wished for the purchased products, the number of sharing links to the user's purchased products, and the user's stay time on the purchased product page. The user's SNS activity information may include the number of times the user uploads a post related to a purchased product to the SNS. A purchase activity vector reflecting time weights and action weights may be generated through data preprocessing of the user's purchase history information and the user's SNS activity information. The time weight may be set to a larger value for a product recently purchased by the user. The action weight may be set to a larger value based on the user's normalized action value as the ratio of actions related to purchase increases. The purchase activity vector may include a value related to ratings for each purchased product, a value related to the number of clicks for each purchased product, a value related to whether or not to wish for each purchased product, a value related to the number of link sharing times for each purchased product, a value related to staying time for each purchased product, and a value related to the number of SNS uploads for each purchased product. The user's preference for each accessory product may be determined based on the purchase activity vector being input to the second neural network.

For example, the user's preference for each accessory product may be determined based on a fact that the capacity of the user's purchase history information and the capacity of the user's SNS activity information are greater than or equal to a preset reference value. The preset reference value may be determined by the following equation.

In the above equation, B _th is the preset reference value, n _allow is the number of times the collection permission message is received, nr _ef is the reference number set in the server, n _def is the basic number of purchases for determining the user's preference information, k is the number of accessory types, l is the number of purchase-related behavior types, and b _ij is the capacity for storing in the server an action related to the j purchase of the ith accessory type. there is

For example, discount information may be added to information on the recommended accessory based on receiving the collection permission message. The discount information may include a discount rate and a discount period for the recommended accessories. The discount rate for the recommended accessory may be determined differently based on the capacity of the user's purchase history information and the capacity of the user's SNS activity information. The discount period for the recommended accessory may be determined differently according to the user's preference for each accessory product.

Additionally, for example, the trend score may be determined by the following equation.

In the equation, T is a trend score, m is the number of a plurality of SNSs, is the weight for the s-th SNS, n _t is the number of posts containing keywords related to accessories during a preset period in the s-th SNS, and n _t-1 may be the number of posts containing keywords related to accessories during a period prior to the preset period in the s-th SNS.

According to embodiments, the server may determine a recommended accessory through a faster regions with convolutional neural network (R-CNN) model by reflecting the user's appearance information including a first image of the user wearing clothes and a second image of the user's face and basic information of the user. Accordingly, the server may more accurately extract the user's physical characteristics and fashion style and recommend accessories suitable for the user, thereby increasing the user's purchase satisfaction.

According to embodiments, the server collects the user's purchase history information and the user's social network service (SNS) activity information based on receiving the collection permission message from the user terminal, and based on the user's purchase history information and the user's SNS activity information, the user's preference for each accessory product that reflects the time weight and action weight may be determined through a preference analysis model using a neural network. Due to this, the server can safely and accurately determine the user's preference for accessory products under the user's permission without using personal information without permission.

Effects obtainable from the embodiments are not limited to the effects mentioned above, and other effects not mentioned can be clearly derived and understood by those skilled in the art based on the detailed description below.

BRIEF DESCRIPTION OF THE DRAWINGS Included as part of the detailed description to aid understanding of the embodiments, the accompanying drawings provide various embodiments and, together with the detailed description, describe technical features of the various embodiments.
1 is a diagram illustrating a configuration of an electronic device according to an exemplary embodiment.
2 is a diagram showing the configuration of a program according to an embodiment.
3 illustrates a method in which a server transmits information about a recommended accessory to a user terminal using a neural network according to an embodiment.
4 is a diagram illustrating first feature points of a first image according to an exemplary embodiment.
5 is a diagram illustrating second feature points of a second image according to an exemplary embodiment.
6 is an example of a data extraction model according to an embodiment.
7 is an example of a preference analysis model according to an embodiment.
8 is a flowchart illustrating a method for a server to transmit information about a recommended accessory to a user terminal using a neural network according to an embodiment.
9 is a signal exchange diagram illustrating a method in which a server transmits information on recommended accessories to a plurality of user terminals using a neural network according to an embodiment.
10 is a block diagram illustrating a configuration of a server according to an exemplary embodiment.

The following embodiments combine elements and features of the embodiments in a predetermined form. Each component or feature may be considered optional unless explicitly stated otherwise. Each component or feature may be implemented in a form not combined with other components or features. In addition, various embodiments may be configured by combining some components and/or features. The order of operations described in various embodiments may be changed. Some components or features of one embodiment may be included in another embodiment, or may be replaced with corresponding components or features of another embodiment.

In the description of the drawings, procedures or steps that may obscure the gist of various embodiments have not been described, and procedures or steps that can be understood by those skilled in the art have not been described either.

Throughout the specification, when a part is said to "comprising" or "including" a certain element, it means that it may further include other elements, not excluding other elements unless otherwise stated. In addition, terms such as “… unit”, “… unit”, and “module” described in the specification refer to a unit that processes at least one function or operation, which may be implemented by hardware, software, or a combination of hardware and software. Also, "a or an", "one", "the" and similar related words in the context of describing various embodiments (particularly in the context of the claims below) may be used in the meaning of both the singular and the plural, unless otherwise indicated herein or clearly contradicted by context.

Hereinafter, embodiments according to various embodiments will be described in detail with reference to the accompanying drawings. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS The detailed description set forth below in conjunction with the accompanying drawings is intended to describe exemplary embodiments of various embodiments, and is not intended to represent a single embodiment.

In addition, specific terms used in various embodiments are provided to help understanding of various embodiments, and the use of these specific terms may be changed into other forms without departing from the technical spirit of various embodiments.

1 is a diagram illustrating a configuration of an electronic device according to an exemplary embodiment.

1 is a block diagram of an electronic device 101 within a network environment 100, according to various embodiments. Referring to FIG. 1 , in a network environment 100, an electronic device 101 may communicate with the electronic device 102 through a first network 198 (eg, a short-distance wireless communication network), or may communicate with at least one of the electronic device 104 and the server 108 through a second network 199 (eg, a long-distance wireless communication network). According to one embodiment, the electronic device 101 may communicate with the electronic device 104 through the server 108 . According to an embodiment, the electronic device 101 includes a processor 120, a memory 130, an input module 150, a sound output module 155, a display module 160, an audio module 170, a sensor module 176, an interface 177, a connection terminal 178, a haptic module 179, a camera module 180, a power management module 188, a battery 189, a communication module ( 190), a subscriber identification module 196, or an antenna module 197. In some embodiments, in the electronic device 101, at least one of these components (eg, the connection terminal 178) may be omitted or one or more other components may be added. In some embodiments, some of these components (eg, sensor module 176, camera module 180, or antenna module 197) may be integrated into one component (eg, display module 160). The electronic device 101 may also be referred to as a client, terminal, or peer.

The processor 120 may, for example, execute software (eg, program 140) to control at least one other component (eg, hardware or software component) of the electronic device 101 connected to the processor 120, and may perform various data processing or calculations. According to one embodiment, as at least part of data processing or operation, processor 120 may store commands or data received from other components (e.g., sensor module 176 or communication module 190) in volatile memory 132, process the commands or data stored in volatile memory 132, and store resultant data in non-volatile memory 134. According to one embodiment, the processor 120 may include a main processor 121 (eg, a central processing unit or an application processor) or a secondary processor 123 (eg, a graphics processing unit, a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor) that may operate independently or together with the main processor 121. For example, when the electronic device 101 includes the main processor 121 and the auxiliary processor 123, the auxiliary processor 123 may use less power than the main processor 121 or may be set to be specialized for a designated function. The secondary processor 123 may be implemented separately from or as part of the main processor 121 .

The auxiliary processor 123 functions related to at least one of the components of the electronic device 101 (eg, the display module 160, the sensor module 176, or the communication module 190) along with the main processor 121 while the main processor 121 is in an active (eg, application execution) state or instead of the main processor 121 while the main processor 121 is in an inactive (eg, sleep) state. Alternatively, at least some of the states may be controlled. According to one embodiment, the auxiliary processor 123 (eg, an image signal processor or a communication processor) may be implemented as a part of other functionally related components (eg, the camera module 180 or the communication module 190). According to an embodiment, the auxiliary processor 123 (eg, a neural network processing device) may include a hardware structure specialized for processing an artificial intelligence model.

AI models can be created through machine learning. Such learning may be performed, for example, in the electronic device 101 itself where the artificial intelligence model is performed, or may be performed through a separate server (eg, the server 108). The learning algorithm may include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the above examples. The artificial intelligence model may include a plurality of artificial neural network layers. The artificial neural network may be one of a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), deep Q-networks, or a combination of two or more of the above, but is not limited to the above examples. The artificial intelligence model may include, in addition or alternatively, software structures in addition to hardware structures.

The memory 130 may store various data used by at least one component (eg, the processor 120 or the sensor module 176) of the electronic device 101 . The data may include, for example, input data or output data for software (eg, program 140) and commands related thereto. The memory 130 may include volatile memory 132 or non-volatile memory 134 .

The program 140 may be stored as software in the memory 130 and may include, for example, an operating system 142 , middleware 144 , or an application 146 .

The input module 150 may receive a command or data to be used by a component (eg, the processor 120) of the electronic device 101 from the outside of the electronic device 101 (eg, a user). The input module 150 may include, for example, a microphone, a mouse, a keyboard, a key (eg, a button), or a digital pen (eg, a stylus pen).

The sound output module 155 may output sound signals to the outside of the electronic device 101 . The sound output module 155 may include, for example, a speaker or a receiver. The speaker can be used for general purposes such as multimedia playback or recording playback. A receiver may be used to receive an incoming call. According to one embodiment, the receiver may be implemented separately from the speaker or as part of it.

The display module 160 may visually provide information to the outside of the electronic device 101 (eg, a user). The display module 160 may include, for example, a display, a hologram device, or a projector and a control circuit for controlling the device. According to one embodiment, the display module 160 may include a touch sensor set to detect a touch or a pressure sensor set to measure the intensity of force generated by the touch.

The audio module 170 may convert sound into an electrical signal or vice versa. According to an embodiment, the audio module 170 may obtain sound through the input module 150, output sound through the sound output module 155, or an external electronic device (e.g., electronic device 102) (e.g., speaker or headphone) connected directly or wirelessly to the electronic device 101.

The sensor module 176 may detect an operating state (eg, power or temperature) of the electronic device 101 or an external environmental state (eg, a user state), and generate an electrical signal or data value corresponding to the detected state. According to one embodiment, the sensor module 176 may include, for example, a gesture sensor, a gyro sensor, an air pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an infrared (IR) sensor, a bio sensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

The interface 177 may support one or more designated protocols that may be used to directly or wirelessly connect the electronic device 101 to an external electronic device (eg, the electronic device 102). According to one embodiment, the interface 177 may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

The connection terminal 178 may include a connector through which the electronic device 101 may be physically connected to an external electronic device (eg, the electronic device 102). According to one embodiment, the connection terminal 178 may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (eg, a headphone connector).

The haptic module 179 may convert electrical signals into mechanical stimuli (eg, vibration or motion) or electrical stimuli that a user may perceive through tactile or kinesthetic senses. According to one embodiment, the haptic module 179 may include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

The camera module 180 may capture still images and moving images. According to one embodiment, the camera module 180 may include one or more lenses, image sensors, image signal processors, or flashes.

The power management module 188 may manage power supplied to the electronic device 101 . According to one embodiment, the power management module 188 may be implemented as at least part of a power management integrated circuit (PMIC), for example.

The battery 189 may supply power to at least one component of the electronic device 101 . According to one embodiment, the battery 189 may include, for example, a non-rechargeable primary cell, a rechargeable secondary cell, or a fuel cell.

The communication module 190 may support establishment of a direct (e.g., wired) communication channel or wireless communication channel between the electronic device 101 and an external electronic device (e.g., the electronic device 102, the electronic device 104, or the server 108), and communication through the established communication channel. The communication module 190 may include one or more communication processors that operate independently of the processor 120 (eg, an application processor) and support direct (eg, wired) communication or wireless communication. According to one embodiment, the communication module 190 may include a wireless communication module 192 (eg, a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module 194 (eg, a local area network (LAN) communication module or a power line communication module). A corresponding communication module among these communication modules may communicate with the external electronic device 104 through a first network 198 (eg, a short-range communication network such as Bluetooth, wireless fidelity (WiFi) direct, or an infrared data association (IrDA)) or a second network 199 (eg, a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a long-distance communication network such as a computer network (eg, a LAN or a WAN)). These various types of communication modules may be integrated as one component (eg, a single chip) or implemented as a plurality of separate components (eg, multiple chips). The wireless communication module 192 may identify or authenticate the electronic device 101 within a communication network such as the first network 198 or the second network 199 using subscriber information (eg, International Mobile Subscriber Identifier (IMSI)) stored in the subscriber identification module 196.

The wireless communication module 192 may support a 5G network after a 4G network and a next-generation communication technology, for example, NR access technology (new radio access technology). NR access technology can support high-speed transmission of high-capacity data (enhanced mobile broadband (eMBB)), minimization of terminal power and access to multiple terminals (massive machine type communications (mMTC)), or high reliability and low latency (URLLC (ultra-reliable and low-latency communications)). The wireless communication module 192 may support a high frequency band (eg, mmWave band) to achieve a high data rate, for example. The wireless communication module 192 may support various technologies for securing performance in a high frequency band, such as beamforming, massive multiple-input and multiple-output (MIMO), full dimensional MIMO (FD-MIMO), an array antenna, analog beam-forming, or a large scale antenna. The wireless communication module 192 may support various requirements defined for the electronic device 101, an external electronic device (eg, the electronic device 104), or a network system (eg, the second network 199). According to an embodiment, the wireless communication module 192 may support peak data rate (eg, 20 Gbps or more) for eMBB realization, loss coverage (eg, 164 dB or less) for mMTC realization, or U-plane latency (eg, downlink (DL) and uplink (UL) 0.5 ms or less, or round trip 1 ms or less) for realizing URLLC.

The antenna module 197 may transmit or receive signals or power to the outside (eg, an external electronic device). According to one embodiment, the antenna module 197 may include an antenna including a radiator formed of a conductor or a conductive pattern formed on a substrate (eg, PCB). According to one embodiment, the antenna module 197 may include a plurality of antennas (eg, an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network such as the first network 198 or the second network 199 may be selected from the plurality of antennas by, for example, the communication module 190. A signal or power may be transmitted or received between the communication module 190 and an external electronic device through the selected at least one antenna. According to some embodiments, other components (eg, a radio frequency integrated circuit (RFIC)) may be additionally formed as a part of the antenna module 197 in addition to the radiator.

According to various embodiments, the antenna module 197 may form a mmWave antenna module. According to an embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent to a first surface (eg, bottom surface) of the printed circuit board and capable of supporting a designated high frequency band (eg, mmWave band), and a plurality of antennas (eg, array antennas) disposed on or adjacent to a second surface (eg, top surface or side surface) of the printed circuit board and capable of transmitting or receiving signals in the designated high frequency band.

At least some of the components are connected to each other through a communication method between peripheral devices (e.g., a bus, general purpose input and output (GPIO), serial peripheral interface (SPI), or mobile industry processor interface (MIPI)) and signals (e.g., commands or data) can be exchanged with each other.

According to an embodiment, commands or data may be transmitted or received between the electronic device 101 and the external electronic device 104 through the server 108 connected to the second network 199 . Each of the external electronic devices 102 or 104 may be the same as or different from the electronic device 101 . According to an embodiment, all or part of operations executed in the electronic device 101 may be executed in one or more external electronic devices among the external electronic devices 102 , 104 , or 108 . For example, when the electronic device 101 needs to automatically perform a certain function or service or in response to a request from a user or another device, the electronic device 101 may request one or more external electronic devices to perform the function or at least part of the service, instead of or in addition to executing the function or service by itself. One or more external electronic devices receiving the request may execute at least a part of the requested function or service or an additional function or service related to the request, and deliver the execution result to the electronic device 101 . The electronic device 101 may provide the result as at least part of a response to the request as it is or additionally processed. To this end, for example, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used. The electronic device 101 may provide an ultra-low latency service using, for example, distributed computing or mobile edge computing. In another embodiment, the external electronic device 104 may include an internet of things (IoT) device. Server 108 may be an intelligent server using machine learning and/or neural networks. According to one embodiment, the external electronic device 104 or server 108 may be included in the second network 199 . The electronic device 101 may be applied to intelligent services (eg, smart home, smart city, smart car, or health care) based on 5G communication technology and IoT-related technology.

The server 108 is connected to the electronic device 101 and can provide a service to the connected electronic device 101 . In addition, the server 108 may proceed with a membership sign-up procedure, store and manage various types of information of users subscribed as members, and provide various purchase and payment functions related to services. In addition, the server 108 may share execution data of service applications executed in each of the plurality of electronic devices 101 in real time so that the service can be shared among users. This server 108 may have the same configuration as a conventional web server or service server in terms of hardware. However, in terms of software, it may include a program module that is implemented through any language such as C, C++, Java, Python, Golang, or kotlin and performs various functions. In addition, the server 108 is generally connected to an unspecified number of clients and/or other servers through an open computer network such as the Internet, receives a request for performing a task from a client or other server, derives and provides a result of the task, and refers to a computer system and computer software (server program) installed therefor. In addition, the server 108, in addition to the above-described server program, should be understood as a broad concept including a series of application programs operating on the server 108 and various databases (DB: Database, hereinafter referred to as “DB”) built inside or outside depending on the case. Accordingly, the server 108 classifies member registration information and various information and data about games, stores them in a DB, and manages them. This DB may be implemented inside or outside the server 108 . In addition, the server 108 can be implemented using server programs that are variously provided according to operating systems such as Windows, Linux, UNIX, and Macintosh on general server hardware. As representative examples, web services can be implemented using IIS (Internet Information Server) used in a Windows environment and CERN, NCSA, APPACH, TOMCAT, etc. used in a Unix environment. In addition, the server 108 may interoperate with an authentication system and a payment system for user authentication of services or payment for purchases related to services.

The first network 198 and the second network 199 refer to a connection structure capable of exchanging information between nodes such as terminals and servers or a network connecting the server 108 and the electronic devices 101 and 104. The first network 198 and the second network 199 include the Internet, a local area network (LAN), a wireless local area network (Wireless Local Area Network), a wide area network (WAN), a personal area network (PAN), 3G, 4G, LTE, 5G, Wi-Fi, etc. are included, but are not limited thereto. The first network 198 and the second network 199 may be closed first networks 198 and second networks 199 such as LAN and WAN, but are preferably open such as the Internet. The Internet refers to the structure of a worldwide open computer first network 198 and second network 199 that provides protocols such as the TCP/IP protocol, TCP, and user datagram protocol (UDP) and various services existing in the upper layer, that is, HyperText Transfer Protocol (HTTP), Telnet, File Transfer Protocol (FTP), Domain Name System (DNS), Simple Mail Transfer Protocol (SMTP), Simple Network Management Protocol (SNMP), Network File Service (NFS), and Network Information Service (NIS). do

A database may have a general data structure implemented in a storage space (hard disk or memory) of a computer system using a database management program (DBMS). The database may have a data storage form in which data can be freely searched for (extracted), deleted, edited, added, and the like. The database is a relational database management system (RDBMS) such as Oracle, Informix, Sybase, and DB2, an object-oriented database management system (OODBMS) such as Gemston, Orion, and O2, and an XML Native Database such as Excelon, Tamino, and Sekaiju. and may have appropriate fields or elements to achieve its own function.

2 is a diagram showing the configuration of a program according to an embodiment.

2 is a block diagram 200 illustrating a program 140 according to various embodiments. According to one embodiment, the program 140 may include an operating system 142, middleware 144, or an application 146 executable in the operating system 142 for controlling one or more resources of the electronic device 101. The operating system 142 may include, for example, Android™, iOS™, Windows™, Symbian™, Tizen™, or Bada™. At least some of the programs 140 may be preloaded in the electronic device 101 during manufacture, or may be downloaded or updated from an external electronic device (e.g., the electronic device 102 or 104 or the server 108) when used by a user. All or part of program 140 may include a neural network.

The operating system 142 may control management (eg, allocation or reclamation) of one or more system resources (eg, process, memory, or power) of the electronic device 101 . Operating system 142 may additionally or alternatively include other hardware devices of electronic device 101, such as input module 150, sound output module 155, display module 160, audio module 170, sensor module 176, interface 177, haptic module 179, camera module 180, power management module 188, battery 189, communication module 190, subscriber identification module 196 or one or more driver programs for driving the antenna module 197.

The middleware 144 may provide various functions to the application 146 so that the function or information provided from one or more resources of the electronic device 101 may be used by the application 146 . Middleware 144 includes, for example, application manager 201, window manager 203, multimedia manager 205, resource manager 207, power manager 209, database manager 211, package manager 213, connectivity manager 215, notification manager 217, location manager 219, graphic manager 221, security manager 223, call manager ( 225), or a voice recognition manager 227.

The application manager 201 may manage the life cycle of the application 146 , for example. The window manager 203 may manage one or more GUI resources used in a screen, for example. The multimedia manager 205, for example, identifies one or more formats necessary for reproducing media files, and encodes or decodes a corresponding media file among the media files using a codec suitable for a selected corresponding format. The resource manager 207 may manage a source code of the application 146 or a memory space of the memory 130 . The power manager 209 may manage, for example, the capacity, temperature, or power of the battery 189, and determine or provide related information necessary for the operation of the electronic device 101 by using corresponding information among them. According to an embodiment, the power manager 209 may interoperate with a basic input/output system (BIOS) (not shown) of the electronic device 101 .

The database manager 211 may create, search, or change a database to be used by the application 146, for example. The package manager 213 may manage installation or update of applications distributed in the form of package files, for example. The connectivity manager 215 may manage, for example, a wireless connection or a direct connection between the electronic device 101 and an external electronic device. The notification manager 217 may provide a function for notifying a user of occurrence of a designated event (eg, an incoming call, message, or alarm), for example. The location manager 219 may manage location information of the electronic device 101, for example. The graphic manager 221 may manage, for example, one or more graphic effects to be provided to a user or a user interface related thereto.

Security manager 223 may provide system security or user authentication, for example. The telephony manager 225 may manage, for example, a voice call function or a video call function provided by the electronic device 101 . The voice recognition manager 227 may transmit, for example, the user's voice data to the server 108, and receive a command corresponding to a function to be performed in the electronic device 101 based at least in part on the voice data, or text data converted based on at least a portion of the voice data from the server 108. According to one embodiment, the middleware 244 may dynamically delete some existing components or add new components. According to one embodiment, at least part of the middleware 144 may be included as part of the operating system 142 or may be implemented as separate software different from the operating system 142 .

Applications 146 include, for example, home 251, dialer 253, SMS/MMS 255, IM (instant message) 257, browser 259, camera 261, alarm 263, contact 265, voice recognition 267, e-mail 269, calendar 271, media player 273, album 275, watch 27 7), health 279 (eg, measurement of vital information such as exercise or blood sugar), or environment information 281 (eg, measurement of atmospheric pressure, humidity, or temperature information). According to an embodiment, the application 146 may further include an information exchange application (not shown) capable of supporting information exchange between the electronic device 101 and an external electronic device. The information exchange application may include, for example, a notification relay application configured to transfer designated information (eg, a call, message, or alarm) to an external electronic device, or a device management application configured to manage the external electronic device. The notification relay application, for example, can deliver notification information corresponding to a designated event (eg, mail reception) generated in another application (eg, the email application 269) of the electronic device 101 to an external electronic device. Additionally or alternatively, the notification relay application may receive notification information from an external electronic device and provide the notification information to the user of the electronic device 101 .

The device management application may control, for example, power (eg, turn-on or turn-off) or function (eg, brightness, resolution, or focus) of an external electronic device that communicates with the electronic device 101 or some component thereof (eg, a display module or a camera module of the external electronic device). The device management application may additionally or alternatively support installation, deletion, or update of an application operating in an external electronic device.

Throughout this specification, a neural network, a neural network, and a network function may be used interchangeably. A neural network may consist of a set of interconnected computational units, which may be generally referred to as “nodes”. These “nodes” may also be referred to as “neurons”. A neural network includes at least two or more nodes. Nodes (or neurons) constituting neural networks may be interconnected by one or more “links”.

In a neural network, two or more nodes connected through a link may form a relative relationship of an input node and an output node. The concept of an input node and an output node is relative, and any node in an output node relationship with one node may have an input node relationship with another node, and vice versa. As described above, the input node to output node relationship can be created around the link. More than one output node can be connected to one input node through a link, and vice versa.

In a relationship between an input node and an output node connected through one link, the value of the output node may be determined based on data input to the input node. Here, a node interconnecting the input node and the output node may have a weight. The weight may be variable, and may be changed by a user or an algorithm in order for the neural network to perform a desired function. For example, when one or more input nodes are connected to one output node by respective links, the output node may determine an output node value based on values input to input nodes connected to the output node and a weight set for a link corresponding to each input node.

As described above, in a neural network, two or more nodes are interconnected through one or more links to form an input node and output node relationship in the neural network. Characteristics of the neural network may be determined according to the number of nodes and links in the neural network, an association between the nodes and links, and a weight value assigned to each link. For example, when there are two neural networks having the same number of nodes and links and different weight values between the links, the two neural networks may be recognized as different from each other.

3 illustrates a method in which a server transmits information about a recommended accessory to a user terminal using a neural network according to an embodiment. 4 is a diagram illustrating first feature points of a first image according to an exemplary embodiment. 5 is a diagram illustrating second feature points of a second image according to an exemplary embodiment. The embodiments of FIGS. 3 to 5 can be combined with various embodiments of the present disclosure.

Referring to FIG. 3, in step S310, a server (e.g., the server 108 of FIG. 1) may receive the user's appearance information including a first image of the user wearing clothes and a second image of the user's face, and basic information of the user from the user terminal (e.g., the electronic device 101 of FIG. 1).

The server may be a server that collects various information related to the user terminal and recommends accessory products suitable for the current trend and the user's body shape and taste by using a neural network based on the collected information. The user terminal may be a terminal that provides various information such as user appearance information and purchase history information to the server through a user interface installed in the user terminal and receives information about recommended accessories from the server.

The user's appearance information is information for determining the user's appearance. The first image in which the user wears clothes is an image from which the user's body shape and fashion style can be identified, and may include a plurality of images. The second image of the user's face is an image from which the user's face shape and hairstyle can be recognized, and may include a plurality of images. For example, the second image of the user's face may be an image of the user's ears.

The user's basic information is basic information about the user, and may include, for example, the user's ID, the user's age, the user's gender, the user's address, and the user's budget. Here, the user's ID may include at least one of an ID for an online shopping mall or a social network service (SNS) ID. The user's budget may include a price range of accessory products that the user wants to purchase.

According to an embodiment, when the first image or the second image received from the user terminal does not satisfy a preset criterion, the server may re-request the first image or the second image from the user terminal. For example, the user terminal may capture a first image or a second image that satisfies a preset standard through a user interface installed in the user terminal, and transmit the captured first image or second image to a server.

In step S320, the server may determine the user's feature information through a data extraction model using the first neural network based on the user's appearance information.

For example, a data extraction model may be a faster regions with convolutional neural network (R-CNN) model.

R-CNN (Regions with Convolutional Neural Network) refers to a neural network that performs object detection by using a set region as an input value of a CNN. R-CNN can perform two operations: estimating a region and classifying the region. For example, R-CNN is a Region Proposal that searches for an object region regardless of category by inputting data and labels in an image, and warping/cropping a feature vector of a fixed size from the proposed region to use as CNN input. Here, the CNN uses a pre-trained network, performs classification through a linear supervised learning model, a Support Vector Machine (SVM), using a feature map obtained through the CNN, and performs bounding box regression through a Regressor. Regression can be performed. At this time, since R-CNN performs 2,000 CNNs per image, it has a very slow disadvantage. To solve this problem, Fast R-CNN does not crop the features of the region from the original image. By extracting from the feature map, one CNN can pass. That is, Fast R-CNN performs pooling on a region of interest (ROI), converts each region into a vector of a fixed size, and passes each vector through a fully connected (FC) layer to improve speed. However, there was still a problem of slow speed due to the bottleneck of the algorithm used to propose a region in Fast R-CNN, and Faster R-CNN improved region proposal speed by using a region proposal network (RPN) instead of an algorithm for region proposal.

For example, the server may generate a first input vector composed of a plurality of first regions of interest through data preprocessing of the first image. The plurality of first regions of interest may include a region of interest related to the user's hair color, a region of interest related to the user's hair style, a region of interest related to the user's body shape, a region of interest related to the user's neck, and a region of interest related to the user's fashion style.

The region of interest related to the user's hair color may be a region where the user's hair is located in the first image. The region of interest related to the user's hair style may be an region excluding the user's face region from among regions where the user's head is located in the first image. The region of interest related to the user's body shape may be a region including the user's body contour in the first image. The region of interest related to the user's neck may be an area from a lower part of the user's face to an upper part of the user's elbow in the first image. The region of interest related to the user's fashion style may be a region excluding the user's head region in the first image. For example, each of the plurality of first regions of interest may include one or more first feature points.

For example, the server may generate a second input vector composed of a plurality of second regions of interest through data preprocessing of the second image. The plurality of second regions of interest may include a region of interest related to the user's skin tone, a region of interest related to the user's ears, and a region of interest related to the user's face shape. For example, each of the plurality of second regions of interest may include one or more second feature points.

The region of interest related to the user's skin tone may be a specific region among regions of the user's face in the second image. The region of interest related to the user's ears may be regions of both outer sides of the facial line based on the user's face in the second image. The region of interest related to the face shape of the user may be a region including the face line of the user in the second image.

For example, the server may determine the feature information of the user based on input of the first input vector and the second input vector to the first neural network.

The user's characteristic information may include a value related to the user's skin tone, a value related to the user's hair color, a value related to the user's hair style, a value related to the user's body type, a value related to the user's fashion style, a value related to the user's face shape, a value related to the user's ear, and a value related to the user's neck.

For example, the value for the user's skin tone is a value indicating one skin tone type among a plurality of skin tone types classified according to red green blue (RGB) values, and may be, for example, a value indicating one skin tone type among 12 skin tone types. For example, the server may convert an RGB value of a region of interest related to the user's skin tone into a hue, saturation, value (HSV) value. For example, the server may determine a range including a converted HSV value among a plurality of pre-classified HSV value ranges through a data extraction model, and determine a value for the user's skin tone as a value indicating a type corresponding to the corresponding range. For example, the 12 types may be classified according to color, saturation, and lightness, and may be classified into pale, light, light grayish, and dark grayish as the lightness increases from low saturation. For example, it can be classified into light, soft, dull, and dark as brightness increases in medium saturation. For example, it can be classified into bright, strong, and deep as brightness increases in high chroma. For example, if the saturation is greater than or equal to a preset saturation value, it may be classified as vivid. Here, the preset saturation value may be greater than a value corresponding to high saturation.

For example, the value for the user's hair color is a value indicating one type among a plurality of hair color types classified according to RGB (red green blue) values, and may indicate one type among six hair color types. For example, six hair color types may include black, brown, gold, red, gray and white. For example, the server may determine a range in which an RGB value for a region of interest related to the user's hair color is included among a plurality of pre-classified RGB value ranges through a data extraction model, and determine a value for the user's hair color as a value indicating a type corresponding to the range.

For example, the value related to the user's hair style is a value for expressing the user's hair style, and may include, for example, a value related to the length of the front hair, a value related to the length of the back hair, and a value indicating the degree of perm. For example, the value related to the length of the front hair may be a ratio of the length of the front hair to the length of the face, the value related to the length of the back hair may be the ratio of the length of the back hair to the length of the face, and the value representing the degree of perm may be a value for the curvature of the hair. For example, the server may determine the ratio of the length of the front hair to the length of the face, the ratio of the length of the back hair to the length of the face, and the curvature of the hair in the region of interest related to the user's hairstyle through the data extraction model.

For example, the value for the user's body type is a value representing one body type among a plurality of body types, and may be, for example, a value representing one body type among five body types. For example, the five body types may be classified based on the width of the user's first region 410, the width of the user's second region 420, and the width of the user's third region 430 in the first image, and may include an hourglass shape, a pear shape, an apple shape, a rectangle, and an inverted triangle. Referring to FIG. 4 , for example, a first area 410 may be a user's chest area, a second area 420 may be a user's waist area, and a third area 430 may be a user's hip area.

For example, the server may determine the widths of the first region 410 to the third region 430 from the first feature points included in the region of interest related to the user's body shape through a data extraction model. For example, when a value obtained by subtracting the width of the second region 420 from the width of the first region 410 is greater than a specific value, and a value obtained by subtracting the width of the second region 420 from the width of the third region 430 is greater than a specific value, the value for the user's body shape may be determined as a value representing an hourglass shape. For example, when the width of the third region 430 is larger than the widths of the first region 410 and the second region 420 by a specific value or more, the value for the user's body shape may be determined as a value representing the pear shape. For example, when the width of the first region 410 is less than a specific value than the width of the second region 420 and the width of the third region 430, the value for the user's body shape may be determined as a value representing an apple shape. For example, when the difference between the width of the first region 410, the width of the second region 420, and the width of the third region 430 is smaller than a specific value, the value for the user's body shape may be determined as a value representing a rectangle. When the width of the first region 410 and the width of the second region 420 are larger than the width of the third region 430 by a specific value or more, the value for the user's body shape may be determined as a value representing an inverted triangle.

For example, the value of the user's fashion style is a value indicating one fashion style among a plurality of fashion styles, and may be, for example, a value indicating one fashion style among five fashion styles. For example, the five fashion styles may be classified as casual style, street style, sporty style, classic style, and semi-classical style. And, a value indicating each style may be preset for the server. For example, when the user wears jeans and a shirt, the server may determine a value for a fashion style (eg, casual style) corresponding to a light attire based on the first image. For example, five fashion styles may be pre-learned through a neural network based on meta data of clothing images. That is, the server may determine the top 8 most frequent characteristics (eg, characteristics such as color, tone, color arrangement, pattern, material, etc.) based on the meta data of the clothing image through the neural network. Here, for example, the colors may be classified into a plurality of first groups such as a natural color group, a red group, a green group, a yellow group, and a blue group, and the tones may be classified into a plurality of second groups such as a light tone group, a light tone group, a gray tone group, and a black tone group. For example, the colors may be classified into a plurality of third groups such as the same color family group, similar color family group, and opposite color group. For example, patterns may be classified into a plurality of fourth groups such as geometric patterns, animal patterns, stripe patterns, check patterns, dot patterns, floral patterns, abstract patterns, and logo patterns. For example, materials may be classified into a plurality of fifth groups such as cotton, wool, polyester, nylon, silk and chiffon. For example, the server may generate the five fashion styles by clustering clothing images having similar values based on the plurality of groups classified according to a plurality of characteristics through a neural network. Here, the clustering operation may be performed through various machine learning techniques for unsupervised learning. For example, the server may determine a fashion style having the highest similarity in the first image through a data extraction model, and may determine a value for the user's fashion style as a value indicating the corresponding fashion style.

For example, the value related to the user's neck is a value for expressing the size and shape of the user's neck, and may include, for example, a value related to the ratio of the length of the shoulder to the thickness of the neck and a value related to the ratio of the height to the length of the neck. For example, the server may determine a value related to the ratio of the length of the shoulder to the thickness of the neck and the ratio of the height to the length of the neck based on the first feature point included in the region of interest 440 related to the user's neck through a data extraction model.

For example, the value for the user's face type is a value indicating one face type among a plurality of face types, and may be, for example, a value indicating one face type among six face types. Referring to FIG. 5 , for example, the server determines the angle between the tip of the chin and the facial outline (based on the second feature point included in the region of interest related to the user's face shape). ), the angle between the forehead starting point and the face outline ( ), a value obtained by dividing the width of the face 502 by the length of the face 501, the vertical distance 503 from the starting point of the forehead to the middle of the forehead, and the horizontal distance 504 between the outermost end points on both sides of the chin can be determined. For example, six face types can be classified based on the angle between the tip of the chin and the facial outline, the angle between the start of the forehead and the outline of the face, the length of the face divided by the width of the face, the vertical distance from the start of the forehead to the glabella, and the horizontal distance between the outermost endpoints on either side of the chin. For example, the six face shapes may include oval, round, square, diamond, inverted triangle, and rectangle. For example, if the angle between the tip of the chin and the facial outline in the second image is equal to or greater than a specific value, it may be classified into a first group including oval, diamond, and inverted triangle shapes, and if the angle between the tip of the chin and the facial outline in the second image is smaller than a specific value, it may be classified into a second group including round, square, and rectangular shapes. Thereafter, in the first group, the angle between the starting point of the forehead and the facial outline may be classified into an egg shape, a diamond shape, and an inverted triangle according to a horizontal distance between a specific value and an outermost end point on both sides of the chin. For example, when the angle between the starting point of the forehead and the facial outline is greater than a specific value and the horizontal distance between the outermost end points on both sides of the chin is smaller than a specific value, an inverted triangle may be determined. In addition, in the second group, it can be classified into round, square, and rectangular shapes according to the vertical distance from the starting point of the forehead to the middle of the forehead, the horizontal distance between the value obtained by dividing the width of the face from the length of the face, and the outermost end points on both sides of the chin. For example, if the vertical distance from the starting point of the forehead to the middle of the forehead is less than or equal to a specific value, the value obtained by dividing the width of the face from the length of the face is less than or equal to a specific value, and the horizontal distance between the outermost end points on both sides of the chin is less than or equal to a specific value, a round shape may be determined. For example, the server extracts the angle between the tip of the chin and the face outline, the angle between the start point of the forehead and the face outline, the value obtained by dividing the length of the face by the width of the face, the vertical distance from the start point of the forehead to the point between the eyebrows, and the horizontal distance between the outermost end points on both sides of the chin, and determines the value for the face shape of the user as a value representing the user's face shape based on the extracted values.

For example, the value related to the user's ear is a value for expressing the size and shape of the user's ear, and may include, for example, a value representing the ratio of the size of the ear to the size of the face, a value representing the degree of roundness of the ear, and a value for the size of the ear lobe. A value representing the degree of roundness of the ear may be a curvature with respect to the outline of the ear. For example, the server may determine a value representing a ratio of the size of the ear 505 and the size of the face in the region of interest associated with the user's ear, a value representing the curvature of the outline of the ear, and a value for the size of the ear lobe 506, based on the second feature point included in the region of interest associated with the user's ear through a data extraction model.

In step S330, the server may determine an accessory candidate list from accessory information pre-stored in the server based on the user's characteristic information and the user's basic information.

Pre-stored accessory information may include information on a plurality of accessories. For example, the pre-stored accessory information may include information about a first accessory, information about a second accessory, information about a third accessory, and information about a fourth accessory. For example, information on accessories may include the type of accessory, a plurality of accessory product names according to the accessory type, a price for each of a plurality of accessory products, and matching information on each of a plurality of accessory products. For example, the first accessory can be earrings, the second accessory can be a necklace, the third accessory can be a bracelet, and the fourth accessory can be a belt. The information on the first accessory may include a type of earring, a plurality of product names according to the type of earring, a price for each of the plurality of earrings products, and matching information for each of the plurality of earrings products. For example, the types of earrings may include various types such as stud earrings, button earrings, drop earrings, unbalanced earrings, ring earrings, tassel earrings, ear cuff earrings, pierced earrings, and chandelier earrings, and new types of earrings may be added. Similarly, information about other accessories may also be added to new types of accessories. For example, the matching information is information indicating the characteristics of a user to whom a corresponding accessory product matches. For example, the matching information may include a value related to the user's skin tone, a value related to the user's hair color, a value related to the user's hair style, a value related to the user's body type, a value related to the user's fashion style, a value related to the user's face type, a value related to the user's ears, a value related to the user's neck, a range related to the user's age, a user's gender, and a user's budget range. Here, the plurality of accessories described above is not limited to the embodiment and may include various types of accessories.

For example, the server may determine an accessory product including matching information corresponding to the user's characteristic information and the user's basic information from pre-stored accessory information, and may include the determined accessory product in the accessory candidate list. For example, the server may generate a search vector composed of the user's skin tone value, the user's hair color value, the user's hair style value, the user's body type value, the user's fashion style value, the user's face shape value, the user's ear value, the user's neck value, the user's age value, the user's gender, and the user's budget value, from the user's characteristic information and the user's basic information. Thereafter, the server may determine matching information that matches the search vector, and may include an accessory product for which matching information is set in the accessory candidate list.

In step S340, the server may transmit a collection request message to the user terminal based on the determination of the accessory candidate list.

The collection request message is a message requesting permission to collect a history of purchasing an accessory product through an online shopping mall by a user terminal and a history of activity in a social network service (SNS) by the user terminal.

In step S350, the server may collect the user's purchase history information and the user's SNS activity information based on receiving the collection permission message from the user terminal.

The collection permission message is a message allowing the user terminal to collect a history of purchasing an accessory product through an online shopping mall and a history of the user terminal's activities in the SNS. For example, the collection permission message may include user terminal identification information and user terminal authentication information. Identification information of the user terminal may be identification information necessary for collecting a purchase history of the user terminal in an online shopping mall and identification information necessary for collecting details of activity of the user terminal in the SNS. For example, the identification information of the user terminal may include a website address of an online shopping mall of the user terminal, an online shopping mall ID of the user terminal, information about an SNS used by the user terminal, and an SNS ID of the user terminal. Authentication information of the user terminal may be authentication information for accessing a purchase history of the user terminal in an online shopping mall and authentication information for collecting activity history of the user terminal in SNS.

The user's purchase history information may include information on the user's purchased products, rating information on the user's purchased products, the number of clicks on the user's purchased products, whether or not the user has wished for the purchased products, the number of times the user shared links to the purchased products, and the user's stay time on the purchased product page. Information about the product purchased by the user may include a product name of the product purchased by the user, a product price, and a purchase time of the product.

The user's SNS activity information may include the number of times the user uploads a post related to a purchased product to the SNS.

In step S360, the server may determine the user's preference for each accessory product through a preference analysis model using a second neural network based on the user's purchase history information and the user's SNS activity information.

For example, a multivariate long short term memory networks (LSTM) model may be used as the preference analysis model.

For example, the server may generate a purchase activity vector reflecting time weights and action weights through data pre-processing of the user's purchase history information and the user's SNS activity information. The time weight may be set to a larger value for a product recently purchased by the user. The action weight may be set to a larger value based on the user's normalized action value as the ratio of actions related to purchase increases. The purchase activity vector may include a value related to ratings for each purchased product, a value related to the number of clicks for each purchased product, a value related to whether or not to wish for each purchased product, a value related to the number of link sharing times for each purchased product, a value related to staying time for each purchased product, and a value related to the number of SNS uploads for each purchased product.

For example, the user's preference for each accessory product may be determined based on the purchase activity vector being input to the second neural network.

According to an embodiment, the user's preference for each accessory product may be determined based on a fact that the capacity of the user's purchase history information and the capacity of the user's SNS activity information are greater than or equal to a preset reference value.

For example, the preset reference value may be determined by Equation 1 below.

In Equation 1, the B _th is the preset reference value, n _allow is the number of times the collection permission message is received, n _ref is the reference number set in the server, n _def is the basic number of purchases for determining the user's preference information, k is the number of accessory types, l is the number of types of purchase-related actions, b _ij is the capacity for storing in the server an action related to the j purchase for the ith accessory type, and the b _ref may be a basic capacity set in the server.

For example, the basic number of purchases and basic capacity for determining the reference number of times and user preference information may be preset in the server. Here, the accessory type is an accessory type included in the pre-stored accessory information, and may include a first accessory type, a second accessory type, a third accessory type, and a fourth accessory type. The type of activity related to the purchase may include providing a rating for the purchased product, clicking on the purchased product, performing a wish function for the purchased product, staying on the purchased product page, sharing a link to the purchased product, and uploading a post about the purchased product to an SNS. A capacity for storing in the server an action related to the j th purchase of the i th accessory type may be preset in the server.

For example, as the number of times the collection permission message is received, the information on the user's purchase activity and SNS activity increases, so the preference analysis model can be sufficiently learned. Therefore, if the number of times the collection permission message is received is high, even if the capacity of the user's purchase history information and the capacity of the user's SNS activity information are small, the server can determine the user's preference for each accessory product, and thus the preset reference value can be determined to be small.

Through this, the server may adaptively adjust the preset reference value in consideration of the number of times the collection permission message is received, the type of accessory, and the type of behavior of the user terminal, instead of always using the preset reference value as a fixed value.

In step S370, the server may determine a recommended accessory from the accessory candidate list based on the user's preference for each accessory product and the trend score for each accessory product.

The trend score for each accessory product may be determined according to the number of posts including keywords related to accessories among a plurality of posts generated on a plurality of SNSs during a preset period.

For example, the plurality of SNSs may include a first SNS and a second SNS. For example, the keyword related to the accessory may include at least one of a product name or a brand name of the accessory.

For example, the server may determine, as the first number, the number of first posts including keywords related to accessories among a plurality of first posts generated in the first SNS during a period prior to the preset period. The server may determine, as the second number, the number of first posts including keywords related to accessories among a plurality of first posts generated in the first SNS during a preset period.

For example, the server may determine the number of second posts including keywords related to accessories among a plurality of second posts generated in the second SNS during a period prior to the preset period as the third number. The server may determine the number of second posts including keywords related to accessories among a plurality of second posts generated in the second SNS during a preset period as the fourth number.

For example, the server may determine the average of the first value obtained by applying the weighted value according to the influence of the first SNS to the ratio of the second number to the first number and the second value obtained by applying the weighted value according to the influence of the second SNS to the multiplier of the fourth number relative to the third number as the trend score for the corresponding accessory product.

Additionally, for example, the trend score may be determined by Equation 2 below.

In Equation 2, the T is a trend score, the m is the number of a plurality of SNS, is the weight for the s-th SNS, n _t is the number of posts containing keywords related to accessories during a preset period in the s-th SNS, and n _t-1 may be the number of posts containing keywords related to accessories during a period prior to the preset period in the s-th SNS.

For example, the weight for an SNS may have a larger value as the number of members subscribed to the corresponding SNS increases, and may be greater than 0 and smaller than 1.

Through this, the server may determine an accessory that is an issue according to a trend that changes periodically and determine a trend score for the accessory.

For example, the server may sum up preference and trend scores for each accessory product included in the accessory candidate list, and may determine as many accessory products as recommended accessories by a preset number in the order of a greater sum of preference and trend scores.

In step S380, the server may transmit information on recommended accessories to the user terminal.

Information on the recommended accessory may include a product name of the recommended accessory, a product price, and a product image of the recommended accessory. For example, a product image of an accessory may be pre-stored in the server by being matched with a product name of the accessory.

According to an embodiment, based on the fact that the server receives the collection permission message from the user terminal, the server may further add discount information to information on recommended accessories. The discount information may include a discount rate and a discount period for the recommended accessories. The discount rate for the recommended accessory may be determined differently based on the capacity of the user's purchase history information and the capacity of the user's SNS activity information. The discount period for the recommended accessory may be determined differently according to the user's preference for each accessory product.

6 is an example of a data extraction model according to an embodiment. The embodiment of FIG. 6 can be combined with various embodiments of the present disclosure.

Referring to FIG. 6 , a neural network used for a data extraction model may be a neural network based on a faster R-CNN. The neural network may include a first input layer 610 , one or more first hidden layers 620 and a first output layer 630 .

For example, the server may generate a first input vector and a second input vector by performing data preprocessing on a first image of the user wearing clothes and a second image of the user's face. For example, the server may generate first input vectors and second input vectors by performing thresholding, blurring, background removal, and shape recognition on a first image in which the user wears clothes and a second image of the user's face. For example, the server may generate a first input vector and a second input vector by performing adaptive binarization on a first image of the user wearing clothes and a second image of the user's face, and detecting a plurality of regions of interest that are larger than a threshold value and larger than a specific pixel size by using a sliding window technique on the binarized images.

For example, each of the learning data consisting of a plurality of first input vectors, a plurality of second input vectors, and a plurality of correct answer user feature information is input to the first input layer 610 of the first neural network, passes through one or more first hidden layers 620 and a first output layer 630, and is output as a first output vector. The first output vector is input to a first loss function layer connected to the first output layer 630, and the first loss function layer A first loss value is output using a first loss function that compares the first output vector with a first correct answer vector for each learning data, and parameters of the first neural network are reduced in the first loss value. It can be learned in a direction. Here, characteristic information of users with a plurality of correct answers may be collected from a plurality of user terminals and pre-stored in the server.

One or more first hidden layers 620 may include a fast R-CNN and a region proposal network (RPN). RPN may include a convolution layer, a region proposal layer, a classification layer, and a regression layer. Specifically, for example, the server may train the RPN through end-to-end learning for region proposal, and use only the region proposal layer excluding the basic CNN from the learned RPN. Fast R-CNN can be trained. After that, the server fine-tunes the CNN that extracts the first feature map, fixes other weights using the trained Fast R-CNN and RPN, and fine-tunes only the layers corresponding to the RPN. Thereafter, the server may train the fast R-CNN by fixing the shared convolutional layer and fine-tuning the layer corresponding to the fast R-CNN.

For example, a first input vector and a second input vector may be input to the first input layer 610, and a feature map may be extracted through a convolutional layer. The feature map is transmitted to the RPN and the Fast R-CNN, and the RPN may search a plurality of regions through a region proposal layer based on the feature map, and pool the plurality of regions into a plurality of first regions of interest and a plurality of second regions of interest. Finally, the server may calculate values related to the plurality of first regions of interest and the plurality of second regions of interest by utilizing the feature map and the region proposal bounding boxes.

For example, Region Proposal Networks (RPNs) may receive an entire image regardless of size and may return a region proposal bounding box. Each bounding box may indicate whether an object exists or not with a score. Each sliding window can be mapped to a small-dimensional feature, and the features obtained with the sliding window can be passed to the classification layer and the regression layer. Several bounding box regions can be predicted for each center position of each sliding window. If the maximum number of predictable bounding box regions for each sliding window position is k, the regression layer may have 4k coordinate values and the classification layer may have 2k score values.

For example, when a 7x7 size feature map exists, 9 anchor boxes of different sizes may be defined for every lattice cell using a sliding window method with a kernel size of 3x3 size. RPN transforms feature maps into size (256, 7, 7) by 3x3 convolution with padding of 1, then transforms them into 2 (background/foreground) * 9 (anchor) = 18 channels by 1x1 convolution to perform classification to predict whether an object exists or not, and 4 (x,y,x,h) * 9 by 1x1 convolution to perform regression to predict the coordinates of the bounding box. (Anchor) = Can be converted to 36 channels. Through this, prediction of the classes and coordinate values of 9 anchors can be performed for each grid (7x7).

7 is an example of a preference analysis model according to an embodiment. The embodiment of FIG. 7 can be combined with various embodiments of the present disclosure.

Referring to FIG. 7 , for example, the second neural network used in the preference analysis model may be a multivariate long short term memory networks (LSTM) model. In general, a recurrent neural network (RNN) can effectively model time-series information because hidden layer values for an existing input stored therein are considered in an output for a next input value. However, since RNN is a structure that depends on past observation values, problems such as vanishing gradient or exploding gradient may occur. A model to solve this problem is LSTM, and by replacing nodes inside the LSTM with memory cells, information can be accumulated or some of past information can be deleted, and the problem of the RNN can be supplemented.

For example, the second neural network may include a second input layer 710, one or more second hidden layers 720, and a second output layer 730.

For example, the server may generate a purchase activity vector reflecting time weights and action weights through data pre-processing of the user's purchase history information and the user's SNS activity information. The time weight may be set to a larger value for an accessory product recently purchased by the user. For example, the time weight may have a value between 0 and 1.

The action weight may be set to a larger value based on the user's normalized action value as the ratio of actions related to purchase increases. For example, an action related to a purchase may include providing a rating for the purchased product, clicking on the purchased product, performing a wish function for the purchased product, staying on the purchased product page, sharing a link to the purchased product, or uploading a post about the purchased product to an SNS. Regarding the actions related to the user's purchase, the rating is expressed from 0 to 5 points, the minimum value for the number of clicks is 0, and the maximum value is different for each user, and the value range may not exist because it indicates whether the wish function for the product has been performed. Therefore, it may be necessary to normalize the behavior with the maximum and minimum values and the behavior with a boolean value to a value between 0 and 1. Here, a boolean value may refer to a value having a value of 0 or 1. In this case, the action value for which the action having the maximum value and the minimum value is normalized may be a value obtained by dividing a value obtained by subtracting the minimum value from the actual action value by a value obtained by subtracting the minimum value from the maximum value. For an action having a boolean value, a normalized action value may be determined with a probability of performing the corresponding action for a purchased product. Thereafter, an action weight for an action related to the purchase may be derived at a ratio obtained by dividing the normalized value from the sum of all normalized action values.

For example, an action weight for an action having a maximum value and a minimum value may be determined by Equation 3 below.

In Equation 3, w _a is the action weight for the action having the maximum and minimum values, x is the number of purchased products of the user, y is the number of actions related to purchase, VR _b is the actual value of the user's corresponding action for the b-th purchased product, Vmin _b is the minimum value for the user's corresponding action for the b-th purchased product, and Vmax _b is the maximum value for the user's corresponding action for the b-th purchased product, and Nb _{, c} may be a normalized action value for the user's c-th action for the b-th purchased product.

For example, an action weight for an action having a Boolean value may be determined by Equation 4 below.

In Equation 4, w _b is the action weight for the action having the boolean value, x is the number of purchased products of the user, y is the number of actions related to the purchase, n _c is the number of purchased products for which the user performed the corresponding action, and N _b,c may be a normalized action value for the c th action of the user for the b th purchased product.

The purchase activity vector may include a value related to ratings for each purchased product, a value related to the number of clicks for each purchased product, a value related to whether or not to wish for each purchased product, a value related to the number of link sharing times for each purchased product, a value related to staying time for each purchased product, and a value related to the number of SNS uploads for each purchased product.

Each learning data composed of a plurality of purchase activity vectors and preferences for each correct answer accessory product is input to the second input layer 710 of the second neural network, passes through one or more second hidden layers 720 and a second output layer 730, and is output as a second output vector. A second loss value may be output using a second loss function that compares second answer vectors, and parameters of the second neural network may be learned in a direction in which the second loss value becomes smaller.

For example, preferences for each accessory product of a plurality of users may be pre-stored in the server, and purchasing activity vectors for each user may be pre-stored in the server. In this case, the above-described time weight and action weight are applied to the purchase activity vector for each user, and the greater the sum of the respective elements of the purchase activity vector, the greater the preference of the user for the corresponding accessory product.

For example, the one or more second hidden layers 720 may include one or more LSTM blocks, and one LSTM block may include a memory cell, an input gate, an erase gate, and an output gate.

For example, the memory cell is a node that outputs a result through an activation function, and the memory cell can perform a recursive operation of using a value output from a memory cell at a previous point in time as its own input at a current point in time. For example, when the current time point is t, a value output by a memory cell at the current time point t may be influenced by values of memory cells in the past. The memory cell may output a cell state (C _t ) value and a hidden state (h _t ) value. That is, the memory cell may use the cell state value (C _t-1 ) and the hidden state value (h _t-1 ) transmitted by the memory cell at time t-1 as input values for calculating the cell state value and the hidden state value at time t.

For example, the input gate 722, the deletion gate 721, and the output gate 723 all include a sigmoid layer, and it may indicate how much information input through the sigmoid layer is transferred. For example, a sigmoid layer is a sigmoid function ( ) may be a layer whose activation function is Also, for example, the cell state 724 is controlled through the input gate 722, the deletion gate 721, and the output gate 723, and weights according to each gate and input may exist.

For example, a purchase activity vector is input to the second input layer, and the deletion gate can transfer a value between 0 and 1 as C t _-1 based on h _t-1 (hidden state at time t-1) and x _t (input value at time t) generated based on the purchase activity vector. Here, a value between 0 and 1 is the amount of information that has gone through the deletion process. The closer to 0, the more information is deleted, and the closer to 1, the more intact information can be transmitted.

For example, the input gate 722 may determine a value to be updated through a sigmoid layer, and a tanh layer may generate C _t vectors, which are new candidate values, and store them in the cell state 724 . tanh means a nonlinear activation function (hyperbolic tangent function).

For example, a new cell state C _t may be created by updating the previous cell state C _t-1 . That is, information about the cell state 724 is deleted by multiplying f _t by the cell state, and a value obtained by scaling the update value may be added.

For example, the output gate 723 may determine a portion of the cell state to be output based on the purchase activity vector in the sigmoid layer, and the output gate 723 may multiply the cell state determined in the sigmoid layer with a value output as a value between -1 and 1 through the tanh layer.

For example, the prediction model may be an LSTM model in which the deletion gate 721 and the input gate 722 are combined. In this case, since only the old information to which new information is added is deleted, the operation can be performed more quickly.

Accordingly, the server may use the parameter of the second neural network learned through the preference analysis model, and the server may determine the preference for each accessory product in consideration of the user's purchase history and SNS activity history.

8 is a flowchart illustrating a method for a server to transmit information about a recommended accessory to a user terminal using a neural network according to an embodiment. The embodiment of FIG. 8 can be combined with various embodiments of the present disclosure.

Referring to FIG. 8 , in step S801, the server may determine an accessory candidate list from accessory information pre-stored in the server based on user characteristic information and user basic information.

For example, the server may determine the user's feature information through the above-described data extraction model based on the user's appearance information including a first image of the user wearing clothes and a second image of the user's face.

In step S802, the server may transmit a collection request message to the user terminal.

For example, the server may transmit a collection request message to the user terminal based on the user's ID included in the user's basic information. In this case, the server may transmit a collection request message to the user terminal based on the determination of the accessory candidate list.

In step S803, the server may determine whether or not the collection permission message is received from the user terminal within a preset time.

For example, the preset time may be an average time for receiving a collection permission message from the user terminal after transmitting the collection request message to the user terminal.

According to an embodiment, when the collection permission message is not received from the user terminal within a preset time, the server may retransmit the collection request message.

In step S804, based on the fact that the server receives the collection permission message from the user terminal within a preset time, the server may determine whether the capacity of the user's purchase history information and SNS activity information is greater than or equal to a preset reference value.

For example, the server may determine the preset reference value by Equation 1 described above.

In step S805, the server may transmit information on recommended accessories to the user terminal based on the fact that the capacity of the user's purchase history information and SNS activity information is greater than or equal to a preset reference value.

For example, in step S805 , information on recommended accessories may be information on recommended accessories determined from the accessory candidate list according to the user's preference for each accessory product and trend score for each accessory product.

Also, for example, in step S805, information on recommended accessories may further include first discount information. Here, the first discount information may include a first discount rate and a first discount period for the recommended accessory. The first discount rate for the recommended accessory may be determined differently based on the capacity of the user's purchase history information and the capacity of the user's SNS activity information. For example, the first discount rate may be determined to be larger as the capacity of the user's purchase history information and the capacity of the user's SNS activity information are larger. In this case, the first discount rate may be changed within a preset range. The first discount period for recommended accessories may be determined differently according to the user's preference for each accessory product. For example, the first discount period may be determined to be short for an accessory product having a high preference value. At this time, the first discount period may be changed within a preset range. Through this, it is possible to increase the user's desire to purchase accessory products while providing a reward for collecting the user's information.

In step S806, the server may request gait image information from the user terminal, based on the fact that the capacity of the user's purchase history information and SNS activity information is less than a preset reference value.

Here, the gait image information is a video of a user walking, and may be a video of a user walking for a preset playback time. For example, the server may analyze the gait image information to determine the user's tendency.

In step S807, the server may determine whether to receive gait image information from the user terminal within a preset time.

According to an embodiment, when gait image information is not received from the user terminal within a preset time, the server may re-request the gait image information from the user terminal.

In step S808, the server may transmit information on recommended accessories to the user terminal based on receiving gait image information from the user terminal within a preset period of time.

For example, in step S808 , information on recommended accessories may be information on recommended accessories determined from the accessory candidate list according to the user's expected preference for each accessory product and trend score for each accessory product.

For example, the server may sum up expected preference and trend scores for each accessory product included in the accessory candidate list, and may determine as many accessory products as recommended accessories by a preset number in the order of a greater sum of the expected preference and trend scores.

Additionally, for example, the server may determine the user's walking type, the user's walking speed, and the user's stride length based on the gait image information received from the user terminal.

For example, the server determines a plurality of feature points for recognizing the user's gait from gait image information. Here, the plurality of feature points may be set in the user's shoulder region, the user's right arm region, the user's left arm region, the user's right leg region, the user's left leg region, and the user's pelvis region. In this case, the positions where the plurality of feature points are set may be different for each user according to the user's arm length, the user's leg length, and the user's height. The server may obtain position changes of a plurality of feature points for the user, and determine walking speed, stride length, angle of the face relative to the shoulder, angle of the shoulder relative to the pelvis, angle of the hip joint, angle of the knee joint, and angle of the ankle joint according to the change in position of the plurality of feature points of the user.

For example, the server may determine the user's extraversion score according to the walking speed and stride length. The higher the gait speed and the longer the stride length, the higher the extraversion score of the user may be determined.

For example, the server may determine one type of a plurality of gait trends according to the angle of the head relative to the shoulder, the angle of the shoulder relative to the pelvis, the angle of the hip joint, the angle of the knee joint, and the angle of the ankle joint. For example, set ranges for each of the angle of the head relative to the shoulder, the angle of the shoulder relative to the pelvis, the angle of the hip joint, the angle of the knee joint, and the angle of the ankle joint may be different for each gait type. That is, the server calculates the angle of the head to the shoulder, the angle of the shoulder to the pelvis, the angle of the hip joint, the angle of the knee joint, and the angle of the ankle joint from the gait image information, and among the plurality of gait types, a gait type matching the angle of the head to the shoulder, the angle of the shoulder to the pelvis, the angle of the hip joint, the angle of the knee joint, and the angle of the ankle joint may be determined as the gait type for the user.

For example, the server may classify the user terminal into one of a plurality of groups according to a value indicating the user's gait type and the user's extraversion score.

For example, the server may determine n preset groups based on values representing gait types and extroversion scores for a plurality of user terminals. For example, n groups may be predetermined based on a value representing a gait type and an extraversion score through a clustering technique using a neural network. The plurality of user terminals may be user terminals for which preference for each accessory product is determined in advance. For example, the plurality of user terminals may be user terminals for which preference for each accessory product is determined through the above-described preference analysis model. For example, clustering may refer to unsupervised learning that groups data having similar properties into a certain number of clusters. In this case, for example, the server may generate vector data based on a value indicating a gait type for each of a plurality of user terminals and an extraversion score. For example, the plurality of groups may be determined as n groups through a Density-Based Spatial Clustering of Applications with Noise (DBSCAN) technique based on a plurality of vector data. For example, DBSCAN assumes that if a specific point belongs to a cluster, it must be located close to many other points in the cluster. For this calculation, radius and minimum points can be used. For example, the diameter may be a radius based on a specific data element, and may be referred to as a dense area. For example, the minimum element could indicate how many elements are needed around the core point to specify a core point. In addition, each element of the data set can be divided into a core, a border, and an outlier point.

For example, through a neural network, the server can check the size of the diameter of each element and search for how many elements there are. Thereafter, if m or more elements exist within the diameter range, the server may determine the corresponding element as a key element. And, the server may determine an element included within a range of a diameter from the core element as a boundary element. Also, the server may determine an element not included within a diameter range from the core element as an outlier element, and the outlier element may be excluded from the corresponding cluster. In addition, when the distance between core elements is smaller than the diameter, the server may classify the corresponding elements into the same cluster.

Through this, the server may classify the plurality of user terminals into a plurality of groups by considering the values indicating the gait type and the extraversion scores. Thereafter, the server may determine an average value of preferences for each accessory product for each of a plurality of groups as an expected preference for each accessory product. That is, the server may classify the user terminal into one of a plurality of groups according to the value indicating the user's gait type and the user's extraversion score, and determine the expected preference for each accessory product of the corresponding group as the expected preference for each accessory product of the user.

Also, for example, in step S808, information on recommended accessories may further include second discount information. Here, the second discount information may include a second discount rate and a second discount period for the recommended accessory. The second discount rate for recommended accessories may be a preset value. For example, the second discount rate may be the largest value in a preset range for the first discount rate. The second discount period for recommended accessories may be determined differently according to the user's expected preference for each accessory product. For example, the second discount period may be determined to be short for an accessory product having a large expected preference value. At this time, the second discount period may be changed within a preset range. Through this, it is possible to increase the user's desire to purchase accessory products while providing a reward for collecting the user's information.

In step S809, the server may transmit information on recommended accessories to the user terminal based on whether the collection permission message is not received or the gait image information is not received within a preset time.

For example, in step S809 , information on recommended accessories may be information on recommended accessories determined from the accessory candidate list according to trend scores for each accessory product.

For example, the server may determine, as recommended accessories, as many as a preset number of accessory products included in the accessory candidate list in order of increasing trend scores.

9 is a signal exchange diagram illustrating a method in which a server transmits information on recommended accessories to a plurality of user terminals using a neural network according to an embodiment. The embodiment of FIG. 9 can be combined with various embodiments of the present disclosure.

Referring to FIG. 9 , in step S901, the server may receive external appearance information and basic information of the user from each of the first user terminal, the second user terminal, and the third user terminal.

In step S902, the server may determine an accessory candidate list for each of the first user terminal, the second user terminal, and the third user terminal.

For example, the server may determine characteristic information of the user through the above-described data extraction model for each of the first user terminal, the second user terminal, and the third user terminal based on the user's appearance information, and may determine an accessory candidate list from accessory information pre-stored in the server based on the user's characteristic information and the user's basic information.

In step S903, the server may transmit a collection request message to each of the first user terminal, the second user terminal, and the third user terminal.

For example, based on the server determining the accessory candidate list for each of the first user terminal, the second user terminal, and the third user terminal, the server may transmit a collection request message to each of the first user terminal, the second user terminal, and the third user terminal.

In step S904, the server may receive collection permission messages from the first user terminal and the second user terminal.

For example, the server may receive collection permission messages from the first user terminal and the second user terminal within a preset time.

In step S905, the server may retransmit the collection request message to the third user terminal.

For example, based on not receiving the collection request message from the third user terminal within a preset time, the server may retransmit the collection permission message to the third user terminal.

In step S906, the server may transmit information on recommended accessories to the third user terminal.

For example, based on not receiving a collection permission message from the third user terminal for a preset time after retransmitting the collection request message, the server may transmit information on recommended accessories to the third user terminal. For example, the server may determine accessory products included in the accessory candidate list for the third user terminal as recommended accessories by a preset number of accessory products in the order of increasing trend scores, and transmit information on the recommended accessories to the third user terminal.

For example, the trend score for each accessory product may be determined according to the number of posts including keywords related to accessories among a plurality of posts generated on a plurality of SNSs during a preset period.

In step S907, the server may collect the user's purchase history information and the user's SNS activity information for the first user terminal and the second user terminal.

In step S908, the server may transmit information on recommended accessories to the first user terminal.

For example, the server transmits information about recommended accessories to the first user terminal based on whether the capacity of the user's purchase history information and the capacity of the user's SNS activity information for the first user terminal are greater than or equal to a preset reference value.

For example, the server may determine the user's preference for each accessory product for the first user terminal through the above-described preference analysis model, and determine a recommended accessory from the accessory candidate list according to the user's preference for each accessory product and the trend score for each accessory product. For example, the server may sum up preference and trend scores for each accessory product included in the accessory candidate list for the first user terminal, and determine as many accessory products as recommended accessories by a preset number in the order of the sum of the preference and trend scores.

In step S909, the server may request gait image information from the second user terminal.

For example, the server may request gait image information from the second user terminal based on the fact that the capacity of the user's purchase history information and the capacity of the user's SNS activity information for the second user terminal are smaller than preset reference values.

In step S910, the server may receive gait image information from the second user terminal.

In step S911, the server may transmit information on recommended accessories to the second user terminal.

For example, the server may determine a recommended accessory from the accessory candidate list according to the user's expected preference for each accessory product for the second user terminal and a trend score for each accessory product.

For example, the server sums expected preference and trend scores for each accessory product included in the accessory candidate list for the second user terminal, and determines as many accessory products as recommended accessories by a preset number in the order of the sum of the predicted preference and trend scores.

For example, the server may determine a group including the second user terminal from among a plurality of groups pre-stored in the server based on the user's gait type and the user's extraversion score obtained by analyzing the gait image information. For example, the server may determine an average of preferences for each accessory product of a group including the second user terminal as an expected preference for each accessory product for the second user terminal.

10 is a block diagram illustrating a configuration of a server according to an exemplary embodiment. One embodiment of FIG. 10 may be combined with various embodiments of the present disclosure.

As shown in FIG. 10 , the server 1000 may include a processor 1010 , a communication unit 1020 and a memory 1030 . However, not all components shown in FIG. 10 are essential components of the server 1000 . The server 1000 may be implemented with more components than those shown in FIG. 10 , or the server 1000 may be implemented with fewer components than those shown in FIG. 10 . For example, the server 1000 according to some embodiments may further include a user input interface (not shown) and an output unit (not shown) in addition to the processor 1010, the communication unit 1020, and the memory 1030.

The processor 1010 typically controls the overall operation of the server 1000. The processor 1010 may include one or more processors to control other elements included in the server 1000 . For example, the processor 1010 may generally control the communication unit 1020 and the memory 1030 by executing programs stored in the memory 1030 . In addition, the processor 1010 may perform the functions of the server 1000 described in FIGS. 3 to 9 by executing programs stored in the memory 1030 .

The communication unit 1020 may include one or more components that allow the server 1000 to communicate with other devices (not shown) and servers (not shown). The other device (not shown) may be a computing device such as the server 1000 or a sensing device, but is not limited thereto. The communication unit 1020 may receive a user input from another electronic device or data stored in an external device from an external device through a network.

For example, the communication unit 1020 may transmit/receive a message for establishing a connection with at least one device. The communication unit 1020 may transmit information generated by the processor 1010 to at least one device connected to the server. The communication unit 1020 may receive information from at least one device connected to the server. The communication unit 1020 may transmit information related to the received information in response to information received from at least one device.

The memory 1030 may store programs for processing and control of the processor 1010 . For example, the memory 1030 may store information input to a server or information received from another device through a network. Also, the memory 1030 may store data generated by the processor 1010 . The memory 1030 may store information input to or output from the server 1000 .

The memory 1030 includes a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (eg, SD or XD memory, etc.), RAM (Random Access Memory), SRAM (Static Random Access Memory), ROM (Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), PROM (Programmable Read-Only Memory), magnetic It may include at least one type of storage medium among a memory, a magnetic disk, and an optical disk.

The embodiments described above may be implemented as hardware components, software components, and/or a combination of hardware components and software components. For example, the devices, methods, and components described in the embodiments may be implemented using one or more general purpose or special purpose computers, such as, for example, a processor, controller, arithmetic logic unit (ALU), digital signal processor, microcomputer, field programmable gate array (FPGA), programmable logic unit (PLU), microprocessor, or any other device capable of executing and responding to instructions. A processing device may run an operating system (OS) and one or more software applications running on the operating system. A processing device may also access, store, manipulate, process, and generate data in response to execution of software. For convenience of understanding, there are cases in which one processing device is used, but those skilled in the art will recognize that the processing device may include a plurality of processing elements and/or multiple types of processing elements. For example, a processing device may include a plurality of processors or a processor and a controller. Other processing configurations are also possible, such as parallel processors.

Software may include a computer program, code, instructions, or a combination of one or more of these, and may configure a processing device to operate as desired, or may independently or collectively direct a processing device. Software and/or data may be permanently or temporarily embodied in any tangible machine, component, physical device, virtual equipment, computer storage medium or device, or transmitted signal wave, to be interpreted by, or to provide instructions or data to, a processing device. Software may be distributed on networked computer systems and stored or executed in a distributed manner. Software and data may be stored on one or more computer readable media.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer readable medium. The computer readable medium may include program instructions, data files, data structures, etc. alone or in combination. Program commands recorded on the medium may be specially designed and configured for the embodiment or may be known and usable to those skilled in computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and hardware devices specially configured to store and execute program instructions such as ROM, RAM, and flash memory. Examples of program instructions include high-level language codes that can be executed by a computer using an interpreter, as well as machine language codes such as those produced by a compiler. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

As described above, although the embodiments have been described with limited drawings, those skilled in the art can apply various technical modifications and variations based on the above. For example, even if the described techniques are performed in an order different from the described method, and/or components of the described system, structure, device, circuit, etc. are combined or combined in a different form from the described method, or replaced or substituted by other components or equivalents, appropriate results can be achieved.

Therefore, other implementations, other embodiments, and equivalents of the claims are within the scope of the following claims.

Claims (12)

A method in which a server transmits information about a recommended accessory to a user terminal using a neural network, the method comprising:
receiving external appearance information of a user and basic information of the user including a first image of the user wearing clothes and a second image of the user's face from the user terminal;
determining characteristic information of the user through a data extraction model using a first neural network based on the external appearance information of the user;
determining an accessory candidate list from accessory information pre-stored in the server based on the user's characteristic information and the user's basic information;
Transmitting a collection request message to the user terminal based on the determination of the accessory candidate list;
Collecting purchase history information and social network service (SNS) activity information of a user based on receiving a collection permission message from the user terminal;
determining a user's preference for each accessory product through a preference analysis model using a second neural network based on the user's purchase history information and the user's SNS activity information;
determining a recommended accessory from the accessory candidate list based on the user's preference for each accessory product and a trend score for each accessory product;
Transmitting information on the recommended accessory to the user terminal; including,
The trend score for each accessory product is determined according to the number of posts including keywords related to accessories among a plurality of posts generated on a plurality of SNSs during a preset period.
method.
According to claim 1,
The data extraction model uses a faster regions with convolutional neural network (R-CNN) model.
A first input vector composed of a plurality of first regions of interest is generated through data preprocessing of the first image;
The plurality of first regions of interest include a region of interest related to the user's hair color, a region of interest related to the user's hair style, a region of interest related to the user's body shape, a region of interest related to the user's neck, and a region of interest related to the user's fashion style;
A second input vector composed of a plurality of second regions of interest is generated through data preprocessing of the second image;
The plurality of second regions of interest include a region of interest related to the user's skin tone, a region of interest related to the user's ears, and a region of interest related to the user's face shape;
Characteristic information of the user is determined based on input of the first input vector and the second input vector to the first neural network;
The user's characteristic information includes a value related to the user's skin tone, a value related to the user's hair color, a value related to the user's hair style, a value related to the user's body type, a value related to the user's fashion style, a value related to the user's face type, a value related to the user's ear, and a value related to the user's neck.
method.
According to claim 2,
The preference analysis model uses a multivariate long short term memory networks (LSTM) model,
The user's purchase history information includes information on the user's purchased product, rating information on the user's purchased product, number of clicks on the user's purchased product, whether or not the user has wished for the purchased product, number of times the user shared links to the purchased product, and length of stay on the user's purchased product page,
The user's SNS activity information includes the number of times the user uploaded a post related to the purchased product to the SNS,
A purchase activity vector reflecting time weights and action weights is generated through data preprocessing of the user's purchase history information and the user's SNS activity information;
The time weight is set to a larger value for a product recently purchased by the user,
The action weight is set to a larger value based on the normalized action value of the user as the ratio of actions related to purchase increases,
The purchase activity vector is composed of a value related to the rating for each purchased product, a value related to the number of clicks for each purchased product, a value related to whether or not to wish for each purchased product, a value related to the number of link sharing times for each purchased product, a value related to the stay time for each purchased product, and a value related to the number of SNS uploads for each purchased product,
The user's preference for each accessory product is determined based on the purchase activity vector being input to the second neural network.
method.
According to claim 3,
The user's preference for each accessory product is determined based on the fact that the capacity of the user's purchase history information and the capacity of the user's SNS activity information are greater than or equal to a preset reference value;
The preset reference value is determined by the following equation,

In the equation above, B _th is the preset reference value, n _allow is the number of times the collection permission message is received, n _ref is the reference number set in the server, n _def is the basic number of purchases for determining the user's preference information, k is the number of accessory types, l is the number of purchase-related types of actions, and b _ij is the capacity for storing in the server an action related to the j purchase for the ith accessory type; b _ref is the basic capacity set in the server,
method.
According to claim 3,
Based on the reception of the collection permission message, discount information is added to the information on the recommended accessory;
The discount information includes a discount rate and a discount period for the recommended accessory,
The discount rate for the recommended accessory is determined differently based on the capacity of the user's purchase history information and the capacity of the user's SNS activity information,
The discount period for the recommended accessory is determined differently according to the user's preference for each accessory product.
method.
A server that transmits information on a recommended accessory to a user terminal using a neural network,
The server includes at least one processor, at least one memory and a communication unit,
The at least one processor is:
Receiving a user's appearance information and user's basic information including a first image in which the user wears clothes and a second image of the user's face from the user terminal;
Characteristic information of the user is determined through a data extraction model using a first neural network based on the external appearance information of the user;
determining an accessory candidate list from accessory information pre-stored in the server based on the user's characteristic information and the user's basic information;
Based on the determination of the accessory candidate list, transmitting a collection request message to the user terminal;
Based on receiving a collection permission message from the user terminal, collecting the user's purchase history information and the user's SNS (social network service) activity information,
Based on the user's purchase history information and the user's SNS activity information, the user's preference for each accessory product is determined through a preference analysis model using a second neural network;
Based on the user's preference for each accessory product and the trend score for each accessory product, a recommended accessory is determined from the accessory candidate list;
Transmitting information on the recommended accessory to the user terminal;
The trend score for each accessory product is determined according to the number of posts including keywords related to accessories among a plurality of posts generated on a plurality of SNSs during a preset period.
server.
According to claim 6,
The data extraction model uses a faster regions with convolutional neural network (R-CNN) model.
A first input vector composed of a plurality of first regions of interest is generated through data preprocessing of the first image;
The plurality of first regions of interest include a region of interest related to the user's hair color, a region of interest related to the user's hair style, a region of interest related to the user's body shape, a region of interest related to the user's neck, and a region of interest related to the user's fashion style;
A second input vector composed of a plurality of second regions of interest is generated through data preprocessing of the second image;
The plurality of second regions of interest include a region of interest related to the user's skin tone, a region of interest related to the user's ears, and a region of interest related to the user's face shape;
Characteristic information of the user is determined based on input of the first input vector and the second input vector to the first neural network;
The user's characteristic information includes a value related to the user's skin tone, a value related to the user's hair color, a value related to the user's hair style, a value related to the user's body type, a value related to the user's fashion style, a value related to the user's face type, a value related to the user's ear, and a value related to the user's neck.
server.
According to claim 7,
The preference analysis model uses a multivariate long short term memory networks (LSTM) model,
The user's purchase history information includes information on the user's purchased product, rating information on the user's purchased product, number of clicks on the user's purchased product, whether or not the user has wished for the purchased product, number of times the user shared links to the purchased product, and length of stay on the user's purchased product page,
The user's SNS activity information includes the number of times the user uploaded a post related to the purchased product to the SNS,
A purchase activity vector reflecting time weights and action weights is generated through data preprocessing of the user's purchase history information and the user's SNS activity information;
The time weight is set to a larger value for a product recently purchased by the user,
The action weight is set to a larger value as the frequency of performing purchase-related actions increases based on the normalized action value of the user,
The purchase activity vector is composed of a value related to the rating for each purchased product, a value related to the number of clicks for each purchased product, a value related to whether or not to wish for each purchased product, a value related to the number of link sharing times for each purchased product, a value related to the stay time for each purchased product, and a value related to the number of SNS uploads for each purchased product,
The user's preference for each accessory product is determined based on the purchase activity vector being input to the second neural network.
server.
According to claim 8,
The user's preference for each accessory product is determined based on the fact that the capacity of the user's purchase history information and the capacity of the user's SNS activity information are greater than or equal to a preset reference value;
The preset reference value is determined by the following equation,

In the equation above, B _th is the preset reference value, n _allow is the number of times the collection permission message is received, n _ref is the reference number set in the server, n _def is the basic number of purchases for determining the user's preference information, k is the number of accessory types, l is the number of purchase-related behavior types, and b _ij is the capacity for storing in the server an action related to the j purchase for the ith accessory type,
server.
According to claim 8,
Based on the reception of the collection permission message, discount information is added to the information on the recommended accessory;
The discount information includes a discount rate and a discount period for the recommended accessory,
The discount rate for the recommended accessory is determined differently based on the capacity of the user's purchase history information and the capacity of the user's SNS activity information,
The discount period for the recommended accessory is determined differently according to the user's preference for each accessory product.
server.
A computer program stored in a computer readable recording medium in order to execute the method of claim 1 in combination with hardware.
A computer readable recording medium storing a computer program combined with hardware to execute the method of claim 1.

Images