KR102185571B1 - An apparatus for identifying purchase intent, a method therefor and a computer readable recording medium on which a program for carrying out the method is recorded - Google Patents

Abstract

An apparatus for identifying purchase intention according to an embodiment of the present invention includes a camera unit that continuously photographs an image of a customer, and a motion recognition unit that detects a product gaze motion, which is an operation by which the customer stares at the product, from the image of the customer. , An emotion recognition unit that determines the customer's purchase intention for the product by analyzing the change in the intensity of the customer's emotion through the facial image, which is a face part of the customer's image, and the determination result, that the customer's purchase intention is When it is determined, it includes an information processing unit that specifies the product the customer is staring at.

Description

An apparatus for identifying purchase intent, a method therefor and a computer readable recording medium on which a program for carrying out the method is recorded}

The present invention relates to a purchase intent identification technology, and more particularly, a device for identifying purchase intent through the behavior and expression of a customer in a store, a method for the same, and a computer-readable record in which a program performing the method is recorded. It's about the medium.

In the past, it was difficult to measure conversion rates or efficiency in the marketing space, but now it is simple to measure and improve using a variety of online tools. It is natural to focus on inbound marketing online. And it is expected to be more emphasized in the future. However, it is clear that offline is still an important factor from a marketing perspective.

Korean Patent Application Publication No. 2018-0047794 published on May 10, 2018 (Name: O2O marketing platform system based on visitor behavior patterns of offline stores and its construction method)

An object of the present invention is to provide a device capable of identifying purchase intentions through behaviors and facial expressions of customers visiting a store, a method therefor, and a computer-readable recording medium in which a program for performing the method is recorded.

An apparatus for identifying purchase intention according to a preferred embodiment of the present invention for achieving the above-described object includes a camera unit continuously photographing an image of a customer, and an operation in which the customer stares at the product from the image of the customer. A motion recognition unit that detects a product gaze motion, and an emotion recognition unit that determines the customer's purchase intention for the product by analyzing the change in the intensity of the customer's emotion through a face image, which is a face part of the customer's image, and the determination As a result, when it is determined that there is a purchase intention of the customer, an information processing unit for specifying a product to be stared by the customer is included.

The emotion recognition unit selects at least one emotion having an intensity equal to or greater than a predetermined first value among a plurality of emotions of the customer, and the intensity of the selected emotion is equal to or greater than a predetermined first value for a predetermined period from the time when the product staring motion is detected. And if it changes within a range less than or equal to a predetermined second value greater than the first value, it is characterized in that it is determined that the customer has a purchase intention.

If the intensity of at least one emotion among a plurality of emotions of the customer increases by a predetermined third value or more after the time when the product gaze action is detected than before the time when the product gaze action is detected, the customer purchases It is characterized by determining that there is an intention.

The device further comprises a communication unit for communicating with an employee device, wherein the information processing unit generates promotion information for the specified product, and transmits the generated promotion information to the employee device through the communication unit. .

The device may further include a display unit for displaying a screen, wherein the information processing unit generates promotion information for the specified product, and displays the generated promotion information through the display unit.

A method for identifying purchase intent according to a preferred embodiment of the present invention for achieving the above-described object includes the steps of continuously capturing an image of a customer by a camera unit, and a motion recognition unit to select a product from the customer's image. The step of detecting a product gaze motion, which is an action to gaze at, and determining a purchase intention of the customer for the product by analyzing the change in the intensity of the customer's emotion through the facial image, which is a face part of the customer's image, by an emotion recognition unit; and And, when the information processing unit determines that the customer has a purchase intention as a result of the determination, specifying a product to be stared at by the customer.

A computer-readable recording medium on which a program for performing a method for identifying purchase intent according to the above-described preferred embodiment of the present invention is recorded is provided.

The present invention can identify a customer's purchase intention through an expression when a customer gazes at a product, and provide a customized response strategy accordingly, thereby providing an efficient marketing strategy.

1 is a block diagram illustrating a system for identifying a purchase intention according to an embodiment of the present invention.
2 is a block diagram illustrating an identification device for identifying purchase intention according to an embodiment of the present invention.
3 is a view for explaining the internal configuration of the emotion recognition unit according to an embodiment of the present invention.
4 is a flowchart illustrating a method for identifying purchase intention according to an embodiment of the present invention.
5 to 7 are graphs for explaining a method for identifying purchase intention according to an embodiment of the present invention.

Prior to the detailed description of the present invention, terms or words used in the present specification and claims described below should not be construed as being limited to their usual or dictionary meanings, and the inventors shall use their own invention in the best way. For explanation, based on the principle that it can be appropriately defined as a concept of terms, it should be interpreted as a meaning and concept consistent with the technical idea of the present invention. Therefore, the embodiments described in the present specification and the configurations shown in the drawings are only the most preferred embodiments of the present invention, and do not represent all the technical spirit of the present invention, and various equivalents that can replace them at the time of application It should be understood that there may be water and variations.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings. In this case, it should be noted that the same components in the accompanying drawings are indicated by the same reference numerals as possible. In addition, detailed descriptions of known functions and configurations that may obscure the subject matter of the present invention will be omitted. For the same reason, some components in the accompanying drawings are exaggerated, omitted, or schematically illustrated, and the size of each component does not entirely reflect the actual size.

First, a system for identifying purchase intention according to an embodiment of the present invention will be described in detail. 1 is a block diagram illustrating a system for identifying a purchase intention according to an embodiment of the present invention. Referring to FIG. 1, a system for identifying purchase intention according to an embodiment of the present invention includes an identification device 100 and an employee device 400.

The identification device 100 is a device including a camera function for photographing the interior of the store, and is installed inside the store. The employee device 400 is a device carried by an employee.

The identification device 100 photographs a customer in the store and determines whether the customer has an intention to purchase a specific product. The identification device 100 may determine the purchase intention for the corresponding product through a change in the intensity of the emotion expressed on the customer's face at the time when the customer stares at the specific product. If it is determined that the customer has an intention to purchase a specific product, the identification device 100 allows the employee to recognize the customer and the product and provides promotion information that can promote the sale of the product. It can be provided to the staff device 400.

Next, an identification device 100 for identifying purchase intention according to an embodiment of the present invention will be described in more detail. 2 is a block diagram illustrating an identification device for identifying purchase intention according to an embodiment of the present invention.

Referring to FIG. 2, the identification device 100 according to an embodiment of the present invention includes a communication unit 110, a camera unit 120, an input unit 130, a display unit 140, an audio unit 150, and a storage unit 160. ) And a control unit 200.

The communication unit 110 is, for example, a means for communicating with the employee device 400. The communication unit 110 may include a radio frequency (RF) transmitter Tx for up-converting and amplifying a frequency of a transmitted signal, and an RF receiver Rx for low-noise amplifying and down-converting a received signal. Further, the communication unit 110 may include a modem that modulates a transmitted signal and demodulates a received signal.

The camera unit 120 is for capturing an image and includes an image sensor. The image sensor receives light reflected from a subject and converts it into an electric signal, and may be implemented based on a Charged Coupled Device (CCD) or Complementary Metal-Oxide Semiconductor (CMOS). The camera unit 120 may further include an analog to digital converter, and may convert an electrical signal output from the image sensor into a digital sequence and output it to the controller 200.

In particular, the camera unit 120 includes a 3D sensor. The 3D sensor is a sensor for obtaining 3D coordinates of each pixel of an image in a non-contact method. When the camera unit 120 captures an image, the 3D sensor measures the actual distance from a predetermined reference point (eg, focus) of the camera unit 120 to the captured object, along with the captured image. The three-dimensional coordinates of the pixel are detected, and the detected three-dimensional coordinates are transmitted to the controller 200. As the 3D sensor, various types of sensors using laser, infrared, and visible light can be used. These 3D sensors are laser type 3D scanners using any one of TOP (Time of Flight), phase-shift and Online Waveform Analysis, laser type 3D scanners using optical trigonometry, and optics using white light or modulated light. Method 3D scanner, Handheld Real Time PHOTO, optical 3D scanner, optical method using Pattern Projection or Line Scanning, laser system full body scanner, photo scanner using photogrammetry, Kinect Fusion A real time scanner to be used may be exemplified.

The input unit 130 may receive a user's key manipulation for controlling the identification device 100, generate an input signal, and transmit the input signal to the control unit 200. The input unit 130 may include various types of keys for controlling the identification device 100. When the display unit 140 is formed of a touch screen, the input unit 130 may perform functions of various keys on the display unit 140, and when all functions can be performed only with the touch screen, the input unit 130 will be omitted. May be.

The display unit 140 is for screen display and may visually provide a menu of the identification device 100, input data, function setting information, and various other information to a user. In addition, the display unit 140 performs a function of outputting screens such as a boot screen, a standby screen, a menu screen, and the like of the identification device 100. The display unit 140 may be formed of a liquid crystal display (LCD), an organic light emitting diode (OLED), an active matrix organic light emitting diode (AMOLED), or the like. Meanwhile, the display unit 140 may be implemented as a touch screen. In this case, the display unit 140 includes a touch sensor. The touch sensor detects a user's touch input. The touch sensor may be composed of a touch sensing sensor such as a capacitive overlay, a pressure type, a resistive overlay, or an infrared beam, or may be composed of a pressure sensor. . In addition to the above sensors, all kinds of sensor devices capable of sensing contact or pressure of an object may be used as the touch sensor of the present invention. The touch sensor detects a user's touch input, generates a detection signal, and transmits it to the controller 200. In particular, when the display unit 140 is formed of a touch screen, some or all functions of the input unit 130 may be performed through the display unit 140.

The audio unit 150 includes a speaker (SPK) for outputting an audio signal such as voice according to an embodiment of the present invention, and a microphone (MIKE) for collecting an audio signal such as voice. That is, the audio unit 150 may output an audio signal through the speaker SPK or transmit an audio signal input through the microphone (MIKE) to the controller 200 under control of the controller 200.

The storage unit 160 serves to store programs and data necessary for the operation of the identification device 100. Each type of data stored in the storage unit 160 may be deleted, changed, or added according to a user's manipulation.

The controller 200 may control the overall operation of the identification device 100 and a signal flow between internal blocks of the identification device 100, and perform a data processing function of processing data. In addition, the control unit 200 basically serves to control various functions of the identification device 100. The control unit 200 may be a central processing unit (CPU), a digital signal processor (DSP), or the like. The control unit 200 includes a motion recognition unit 210, an emotion recognition unit 220 and an information processing unit 230. The operation of the control unit 200 including the motion recognition unit 210, the emotion recognition unit 220, and the information processing unit 230 will be described in more detail below.

Next, the emotion recognition unit 220 according to an embodiment of the present invention will be described in more detail. The emotion recognition unit 220 is preferably implemented with convolutional neural networks (CNN). However, the emotion recognition unit 220 is not limited to a convolutional neural network, and an artificial neural network (ANN) that receives a face image and learns to output a predetermined number of expressions with probability from the face image. networks), it may be applied to an embodiment of the present invention regardless of its kind. 3 is a view for explaining the internal configuration of the emotion recognition unit according to an embodiment of the present invention.

3, the emotion recognition unit 220 includes a plurality of layers. The plurality of layers configures the next layer through a plurality of operations in which the output of any one layer is weighted. Here, the weight (W) determines the strength of the inter-layer connection.

The emotion recognition unit 220 includes an input layer (INL), a convolution layer (CL), a pooling layer (PL), a fully-connected layer (FCL), and an output layer ( Output layer: ML). The input layer IL consists of a matrix of a predetermined size. Each element of the input layer (IL) matrix corresponds to each pixel of the input image. As shown in FIG. 3, two convolution layers (CL: CLF, CLS) and two pooling layers (PL: PLF, PLS) are shown to be alternately repeated, but the present invention is not limited thereto. Those of ordinary skill in the art will appreciate that the number and arrangement order of the convolutional layers (CL) and the pooling layers (PL) may vary depending on the design of the artificial neural network. Each of the convolutional layer CL and the pooling layer PL includes a plurality of feature maps indicated by solid squares, and each of these feature maps is a matrix having a predetermined size. The value of each element of the matrix constituting the feature map is calculated by applying a convolution operation or pooling or subsampling using a kernel to the previous layer. Here, the kernel is a matrix of a predetermined size indicated by a dotted rectangle, and the value of each element of the matrix constituting the kernel becomes a weight (W).

The fully connected layer (FCL) includes a plurality of nodes (or sigmoid: C1, C2, C3......Cn), and the output layer (OUL) includes a plurality of output nodes (O1, O2, O3, .. ., O6). A weight W is also applied to the calculation of the fully connected layer FL and input to a plurality of output nodes O1, O2, O3... O6 of the output layer OL. Each of the plurality of output nodes O1, O2, O3, ... O6 corresponds to a predetermined emotion. For example, these emotions include joy, sadness, fear, surprise, disgust and anger.

For example, the first output node O1 corresponds to joy among emotions derived from a face image, and a first output value, which is an output of the first output node O1, indicates a probability that the emotion derived from the face image is joy. For example, if the first output value, which is the output of the first output node O1, is 0.35, it indicates that the probability that the emotion derived from the corresponding face image is joy is 35%.

As another example, the second output node O2 corresponds to sadness among emotions derived from a face image, and a second output value, which is an output of the second output node O2, indicates a probability that the emotion derived from the face image is sadness. For example, if the second output value, which is the output of the second output node O2, is 0.10, it indicates that the probability that the emotion derived from the corresponding face image is sadness is 10%.

As another example, the sixth output node O6 corresponds to anger among emotions derived from the face image, and the sixth output value, which is an output of the sixth output node O6, represents the probability that the emotion derived from the face image is anger. . For example, if the sixth output value, which is the output of the sixth output node O6, is 0.02, it indicates that the probability that the emotion derived from the face image is anger is 2%.

Each of the plurality of layers (INL, CL, PL, FL, and OUL) includes a plurality of operations. Each of a plurality of operations of a plurality of layers (INL, CL, PL, FL, and OUL) is applied with a weight (w), and the result of the operation to which the weight (w) is applied is transferred to the next layer. That is, the operation result of the previous layer becomes the input of the next layer.

As described above, the input layer IL is a feature map that is a matrix of a predetermined size. The elements of the matrix of the input layer IL are pixel units. Each element of the matrix may be a pixel value of each pixel of the face image, and the pixel value may be input as binary data to an element of the matrix of the input layer IL.

Then, a convolution operation using each of the plurality of kernels W is performed on the input layer matrix, and the result of the operation is input to a plurality of feature maps of the first convolutional layer CLF. Here, each of the plurality of kernels W may use a matrix having a predetermined size in which an element of the matrix is a weight W. In addition, each of the plurality of feature maps of the first convolutional layer CLF is a matrix having a predetermined size.

Next, a pooling operation (subsampling) using a plurality of kernels W is performed on a plurality of feature maps of the first convolutional layer CLF. A plurality of kernels (W) is also a matrix of a predetermined size, each of which is composed of a weight (W). The result of the subsampling operation is input to a plurality of feature maps of the first pooling layer PLF. Each of the plurality of feature maps of the first pooling layer PLF is also a matrix having a predetermined size.

Subsequently, a convolution operation (convolution) is performed using a kernel (W), which is a matrix of a predetermined size in which each element of the matrix is a weight (W) with respect to the plurality of feature maps of the first pooling layer (PLF). A second convolutional layer (CLS) consisting of a map is constructed. Next, a second pooling consisting of a plurality of feature maps is performed by performing a subsampling operation using a kernel W, which is a matrix consisting of a plurality of weights w, for a plurality of feature maps of the second convolutional layer (CLS). Construct a layer (PLS). Each of the second pooling layers PLS is also a matrix having a predetermined size.

Thereafter, a convolution operation using a plurality of kernels W is performed on a plurality of feature maps of the second pooling layer PLS. A plurality of kernels (W) is also a matrix of a predetermined size whose elements are weights (w). A fully connected layer (FCL) is generated according to a result of a convolution operation using a plurality of kernels (W). In other words, a result of a convolution operation using a plurality of kernels W is input to a plurality of nodes C1 to Cn.

Each of the plurality of nodes (C1 to Cn) of the fully connected layer (FL) performs a predetermined operation using a transfer function, etc. for an input from the second pooling layer (PLS), and applies a weight (w) to the operation. Input to each node of the output layer (OUL). Accordingly, the plurality of output nodes O1 to O6 of the output layer OUL perform a predetermined operation on a value input from the fully connected layer FCL, and output an output value as a result. As described above, each of the plurality of output nodes O1 to O6 corresponds to an emotion derived from a face image, and an output value of each of the plurality of output nodes O1 to O6 is a probability value of an emotion derived from the corresponding face image. .

As described above, each of the plurality of layers of the emotion recognition unit 220 consists of a plurality of operations, and a weight w is applied to a result of any one operation of any one layer and input to a subsequent layer. Accordingly, when a face image is input, the emotion recognition unit 220 performs a plurality of operations to which a weight w is applied in each pixel unit of the face image, and outputs a result of the operation. According to the result of this operation, the output values of each of the output nodes O1 to O6 finally become a probability value corresponding to a predetermined emotion. For example, the output values of each of the output nodes O1 to O6 are probability values of joy, sadness, fear, surprise, dislike, and anger.

The emotion recognition unit 220 may be trained using a face image FIMG in which the emotion is known as learning data. When the face image FIMG, which is the learning data, is input, the emotion recognition unit 220 sets an expected value according to the known emotion of the face image FIMG. Here, the expected value may be a probability for each of a plurality of emotions expected to be detected from the corresponding face image FIMG. For example, when the facial image FIMG represents joy, the expected value may be set to 0.600, 0.001, 0.001, 0.396, 0.001, and 0.001 for each of joy, sadness, fear, surprise, disgust, and anger.

Then, the emotion recognition unit 220 finally applies to the face image through a plurality of operations of a plurality of layers (INL, CL, PL, FL, OUL) for a plurality of pixel values of the face image FIMG, which is the training data. Outputs the probability of each of a plurality of corresponding emotions. At this time, it is assumed that each of the plurality of output nodes O1 to O6 responds to joy, sadness, fear, surprise, disgust, and anger, and that the output values are derived such as 0.400, 0.100, 0.163, 0.070, 0.233, and 0.034.

In this way, before sufficient learning is performed, that is, before the learning is completed, the output value of the emotion recognition unit 220 may be different from the expected value. Accordingly, the emotion recognition unit 220 compares the output value and the expected value, and corrects the weight W through a back-propagation algorithm so that the difference between the expected value and the output value is minimized.

In this way, the emotion recognition unit 220 performs a learning procedure of modifying the weight W through a back propagation algorithm when an output value is derived by inputting training data for which an expected value is set.

The emotion recognition unit 220 repeats the above-described learning procedure by using a plurality of learning data. In particular, when any one of the learning data is input, if the difference between the expected value and the output value is within a predetermined value and there is no change in the output value, the emotion recognition unit 220 determines that the learning has been completed, and ends the learning process.

When the learning process is finished, the emotion recognition unit 220 may determine the intensity of a plurality of emotions from the corresponding face image FIMG by using the face image FIMG of which the emotion is unknown as input data. That is, the emotion recognition unit 220 finally responds to the face image through a plurality of operations of a plurality of layers (INL, CL, PL, FL, OUL) for a plurality of pixel values of the face image FIMG as input data. Outputs the probability of each of the plurality of emotions. At this time, each of the plurality of output nodes O1 to O6 responds to joy, sadness, fear, surprise, disgust, and anger, and the output values of each of the output nodes O1 to O6 indicate emotions corresponding to the face image FIMG. It means the probability to represent. The present invention uses the probability of each emotion as the intensity of emotion. For example, if the output values of each of the output nodes O1 to O6 are derived as 0.800, 0.010, 0.010, 0.010, 0.160, and 0.010, the emotion recognition unit 220 uses the output values of the face image FIMG as joy, sadness, and Recognize each emotion as the intensity of fear, surprise, aversion, and anger.

Next, a method for identifying purchase intention according to an embodiment of the present invention will be described. 4 is a flowchart illustrating a method for identifying purchase intention according to an embodiment of the present invention. 5 to 7 are graphs for explaining a method for identifying purchase intention according to an embodiment of the present invention.

Referring to FIG. 4, the control unit 200 of the identification device 100 continuously photographs customers in the store through the camera unit 120 in step S110 and stores them in the storage unit 160. During such photographing, the motion recognition unit 210 of the control unit 200 checks whether a product gaze motion, which is an operation in which the customer stares at a specific product, is detected from the image captured in step S120.

When the motion recognition unit 210 detects a product staring motion, the emotion recognition unit 220 analyzes the change in the customer’s emotion through the face of the customer’s image captured by the camera unit 120 in step S130 Determine the customer's purchase intention for

When the motion recognition unit 210 detects a product staring motion, the emotion recognition unit 220 is an image of a customer that is photographed through the camera unit 120 and stored in the storage unit 160 from before a predetermined period based on the time point. The middle face is extracted, and the change in the intensity of the customer's emotion is analyzed through the facial image, which is an image of the extracted face. The emotion recognition unit 220 analyzes a change in the intensity of the customer's emotion by extracting a face image from the customer's image at a predetermined period (eg, once a second). For example, the emotion recognition unit 220 analyzes a face image extracted at a certain point in a face image through a convolutional neural network (CNN), and the intensity of the customer's emotion is as shown in FIG. 5. As shown, the intensity of the customer's emotions, which are joy, sadness, fear, surprise, disgust, and anger analyzed through the convolutional neural network (CNN) of the emotion recognition unit 220, is 0.0264, 0.3321, 0.1223, 0.0977, 0.3852, respectively. It may be 0.0263 or 0.0264.

According to an embodiment, the emotion recognition unit 220 selects at least one emotion having an intensity greater than or equal to a predetermined first value among a plurality of emotions of the customer, and the change in the intensity of the selected emotion is from a time point when the product staring motion is detected. If it is within a range equal to or greater than a predetermined first value and equal to or less than a predetermined second value greater than the first value for a predetermined period, it is determined that the customer has a purchase intention.

For example, referring to FIGS. 5 and 6, it is assumed that, for example, a first value is 0.30 and a second value is 0.40. Then, the emotion recognition unit 220 may select an emotion having an emotion intensity of 0.30 or more. In this case, according to FIG. 5, at least one of emotions'joy' and'surprise' may be selected. At this time, it is assumed that the emotion recognition unit 220 has selected surprise. In this case, as shown in the graph of FIG. 6, when the intensity of the surprise changes within the range of 0.30 or more and 0.40 or less, which are the first values for a predetermined period from the time point Ts at which the product staring motion is detected, the emotion recognition unit 220 It is determined that there is an intention to purchase this. As shown in the graph of FIG. 6, a change in the customer's emotion within a predetermined range that is greater than or equal to the first size and less than or equal to the second size is referred to as an awakening state, and this arousal state is a state of interest in a certain object. Accordingly, the present invention determines that the customer has an intention to purchase the product in this awakening state.

On the other hand, referring to FIGS. 5 and 7, it is assumed that, for example, a first value is 0.30 and a second value is 0.40. Then, the emotion recognition unit 220 may select an emotion having an emotion intensity of 0.30 or more. In this case, according to FIG. 5, at least one of emotions'joy' and'surprise' may be selected. At this time, it is assumed that the emotion recognition unit 220 has selected surprise. In this case, as shown in the graph of FIG. 7, since the intensity of the surprise changes to less than 0.30 or 0.40 or more, which is the first value, for a predetermined period from the time point Ts at which the product staring motion is detected, the customer's purchase intention It is judged that there is no. As shown in FIG. 7, when the intensity of the emotion changes rapidly beyond a predetermined range, it is defined as an excited state, and when the intensity of the emotion is not large and is low, it is defined as a relaxed state. In this case, in the case of excitement, the state of emotion changes due to external factors other than interest in the target product, and is irrelevant to the target product. In addition, in the case of the relaxed state, it may be determined that the customer is not actually interested in shopping.

According to another embodiment, the emotion recognition unit 220 has the intensity of at least one emotion among a plurality of emotions of the customer greater than or equal to a predetermined third value after the time when the product staring motion is detected than before the time when the product staring motion is detected. If it rises, it is determined that the customer has a purchase intention. For example, referring to FIG. 6, it is assumed that the third value is 0.30. Then, as shown in the graph of FIG. 6, when the intensity of the surprise increases by 0.30 or more after the time point Ts when the product staring motion is detected than before the time point Ts when the product staring motion is detected, the emotion recognition unit 220 It can be judged as.

Referring back to FIG. 4, as described above, as a result of the determination in step S140, if it is determined that the customer has a purchase intention, the information processing unit 230 specifies the product stared at by the customer in step S150.

Then, the information processing unit 230 generates promotion information for the product in step S160, and provides the generated promotion information. Here, the promotion information includes product identification information, which is information for identifying a product that the customer has stared at, and a customer identification image, which is an image to identify the customer. In addition, the promotion information includes information on the product, discount information, and the like for promoting the sale of the product.

According to an embodiment, the information processing unit 230 may display promotion information through the display unit 140. According to another embodiment, the information processing unit 230 may transmit promotion information to the employee device 400 through the communication unit 110.

Accordingly, an employee who reads the promotion information displayed on the display unit 140 or reads the promotion information through the employee device 400 may explain information about the product or discount information to the corresponding customer. That is, the present invention can promote product sales through customer response as described above.

Meanwhile, the methods according to the embodiments of the present invention described above may be implemented in the form of programs that can be read through various computer means and recorded on a computer-readable recording medium. Here, the recording medium may include a program command, a data file, a data structure, or the like alone or in combination. The program instructions recorded on the recording medium may be specially designed and configured for the present invention, or may be known and usable to those skilled in computer software. For example, the recording medium includes magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic-optical media such as floptical disks ( magneto-optical media), and hardware devices specially configured to store and execute program instructions such as ROM, RAM, flash memory, and the like. Examples of the program instruction may include not only machine language wires such as those made by a compiler, but also high-level language wires that can be executed by a computer using an interpreter or the like. These hardware devices may be configured to operate as one or more software modules to perform the operation of the present invention, and vice versa.

The present invention has been described above using several preferred embodiments, but these embodiments are illustrative and not limiting. As such, those of ordinary skill in the art to which the present invention pertains will understand that various changes and modifications can be made according to the equivalence theory without departing from the spirit of the present invention and the scope of the rights presented in the appended claims.

100: recognition device
110: communication unit 120: camera unit
130: input unit 140: display unit
150: audio unit 160: storage unit
200: control unit 210: motion recognition unit
220: emotion recognition unit 230: information processing unit
400: staff device

Claims (7)

In the device for identifying purchase intent,
A camera unit continuously photographing a customer's image;
A motion recognition unit that detects a product gaze motion, which is a motion for the customer to stare at the product, from the image of the customer;
An emotion recognition unit for determining a customer's purchase intention for a corresponding product by analyzing a change in the intensity of a customer's emotion through a facial image, which is a face part of the customer's image; And
If the determination result, if it is determined that the purchase intention of the customer, the information processing unit for specifying the product the customer is staring; includes,
The emotion recognition unit
An input layer including a feature map, which is a matrix of a predetermined size having elements in units of pixels,
A first convolutional layer including a plurality of feature maps, each of which is a matrix of a predetermined size,
A first pooling layer including a plurality of feature maps, each of which is a matrix of a predetermined size,
A second convolutional layer including a plurality of feature maps, each of which is a matrix of a predetermined size,
A second pooling layer including a plurality of feature maps, each of which is a matrix of a predetermined size,
A fully connected layer including a plurality of nodes,
Output layer containing multiple output nodes
Including,
Each of the plurality of output nodes corresponds to an emotion derived from the face image,
The emotion recognition unit
Inputting the pixel value of the face image into the matrix of the input layer,
The result of the convolution operation performed by performing a convolution operation using a plurality of kernels on the matrix of the input layer is input into a plurality of feature maps of the first convolution layer,
A pooling operation using a plurality of kernels is performed on a plurality of feature maps of the first convolutional layer, and the result of the performed pooling operation is input to a plurality of feature maps of the first pooling layer,
A convolution operation using a plurality of kernels is performed on a plurality of feature maps of the first pooling layer, and the result of the performed convolution operation is input to a plurality of feature maps of the second convolutional layer,
A pooling operation using a plurality of kernels is performed on a plurality of feature maps of the second convolutional layer, and the result of the performed pooling operation is input to a plurality of nodes of the fully connected layer,
The plurality of nodes of the fully connected layer performs an operation using a transfer function on an input from the second pooling layer, applies a weight to the performed operation, and inputs it to a plurality of output nodes of the output layer,
The plurality of output nodes of the output layer perform a predetermined operation on the value input from the fully connected layer, and output an output value as a result of the operation,
The output value of each of the plurality of output nodes is a probability value of each of a plurality of emotions derived from the face image,
The emotion recognition unit
Recognize the output value as the strength of each of a plurality of emotions of the customer,
Selecting at least one emotion having an intensity equal to or greater than a predetermined first value from among the plurality of emotions of the customer,
The intensity of the selected emotion
For a predetermined period from the time when the product staring motion is detected
If it changes while maintaining an arousal state other than a relaxed state and an excited state within a range of more than a predetermined first value and less than a predetermined second value greater than the first value
It is determined that the customer has an intention to purchase,
The intensity of at least one of the customer’s emotions
Than before the time when the above product gaze action was detected
After the point in time when the above product gaze action is detected
If it rises above the predetermined third value,
Characterized in that it is determined that the customer has a purchase intention
A device for identifying purchase intent. delete delete The method of claim 1,
The device is
Further includes; a communication unit for communication with the employee device,
The information processing unit
Generate promotion information for the specified product,
Characterized in that transmitting the generated promotion information to the employee device through the communication unit
A device for identifying purchase intent. The method of claim 1,
The device is
A display unit for displaying a screen; further includes,
The information processing unit
Generate promotion information for the specified product,
Displaying the generated promotion information through the display unit
A device for identifying purchase intent. In a method for identifying purchase intent,
Continuously capturing an image of a customer by a camera unit;
Detecting, by a motion recognition unit, a product gaze motion, which is an action for the customer to stare at the product, from the image of the customer;
Determining, by an emotion recognition unit, a change in the intensity of a customer's emotion through a facial image, which is a face part of the customer's image, to determine the customer's purchase intention for the corresponding product; And
When the information processing unit determines that the customer's purchase intention is determined as a result of the determination, specifying a product to be stared at by the customer; and
The emotion recognition unit
An input layer including a feature map, which is a matrix of a predetermined size having elements in units of pixels,
A first convolutional layer including a plurality of feature maps, each of which is a matrix of a predetermined size,
A first pooling layer including a plurality of feature maps, each of which is a matrix of a predetermined size,
A second convolutional layer including a plurality of feature maps, each of which is a matrix of a predetermined size,
A second pooling layer including a plurality of feature maps, each of which is a matrix of a predetermined size,
A fully connected layer including a plurality of nodes,
Output layer containing multiple output nodes
Including,
Each of the plurality of output nodes corresponds to an emotion derived from the face image,
The step of determining the purchase intention
The emotion recognition unit
Inputting the pixel value of the face image into the matrix of the input layer,
The result of the convolution operation performed by performing a convolution operation using a plurality of kernels on the matrix of the input layer is input into a plurality of feature maps of the first convolution layer,
A pooling operation performed by performing a pooling operation using a plurality of kernels on a plurality of feature maps of the first convolutional layer is input into a plurality of feature maps of the first pooling layer,
The result of the convolution operation performed by performing a convolution operation using a plurality of kernels on the plurality of feature maps of the first pooling layer is input to the plurality of feature maps of the second convolutional layer,
A pooling operation performed by performing a pooling operation using a plurality of kernels on a plurality of feature maps of the second convolutional layer is input to a plurality of nodes of the fully connected layer,
The plurality of nodes of the fully connected layer performs an operation using a transfer function on the input from the second pooling layer, applies a weight to the performed operation, and inputs it to the plurality of output nodes of the output layer,
The plurality of output nodes of the output layer perform a predetermined operation on the value input from the fully connected layer, and output the output values of each of the plurality of output nodes, which are probability values of each of a plurality of emotions derived from the face image as a result of the operation. and,
The emotion recognition unit
Recognizing the output value of each of the plurality of output nodes as the strength of each of the plurality of emotions of the customer,
Selecting at least one emotion having an intensity equal to or greater than a predetermined first value from among the plurality of emotions of the customer,
The intensity of the selected emotion
For a predetermined period from the time when the product staring motion is detected
Within the range of more than the first value and less than or equal to a predetermined second value greater than the first value
If you change while maintaining a state of arousal rather than a state of relaxation and excitement,
It is determined that the customer has an intention to purchase,
The intensity of at least one of the customer’s emotions
Than before the time when the above product gaze action was detected
After the point in time when the above product gaze action is detected
If it rises above the predetermined third value,
Characterized in that it is determined that the customer has a purchase intention
A way to identify purchase intent. A computer-readable recording medium on which a program for performing the method for identifying purchase intent according to claim 6 is recorded.

Images