JP2016062564A - Information processing device, and information processing program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an information processing device for specifying a customer's action for looking at a tag of a commodity when counting the customer's action for each action.SOLUTION: An information processing device includes: determination means which determines a customer's action for looking at a tag of a commodity from an image in which the customer is captured; and counting means which counts each action determined by the determination means.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an information processing apparatus and an information processing program.

  In Patent Document 1, it is an object to accurately grasp the number of customers who are interested in a product, and is stored in a nonvolatile memory of a storage device according to a predetermined operation (pressing operation) in an operation unit. In response to a predetermined operation (pressing operation) on the operation unit, the price data stored in the nonvolatile memory of the storage device is read and a price display based on the read price data is displayed. It is disclosed that display is performed on the display unit for a certain period of time.

  In Patent Document 2, it is an object of the present invention to provide information specific to a customer related to the product to a customer who reaches for the product displayed on a shelf or a show space. Obtains user identification information for identifying the user from a user who has reached for the corresponding product without contact, and transmits the acquired user identification information together with the product identification information corresponding to the information terminal device to the server device, The server device stores attribute information related to the user in association with the user identification information of each user, and is stored in association with the user identification information every time the user identification information and the product identification information are received from each information terminal device. It is disclosed that the user attribute information is read and the user attribute information is associated with the product group identification information and processed.

JP 2009-092794 A JP 2007-114957 A

  An object of the present invention is to provide an information processing apparatus and an information processing program for determining an action for viewing a tag of a product by a customer when counting a customer's action for each action. .

The gist of the present invention for achieving the object lies in the inventions of the following items.
The invention of claim 1 comprises a determination means for determining an action for viewing a tag of a product by the customer from an image taken by the customer, and a counting means for counting each action determined by the determination means. It is an information processing apparatus characterized by doing.

  According to a second aspect of the present invention, the tag is attached to a position where it is necessary to touch the product in order to see the tag, and the determination means touches the product after touching the product. The information processing apparatus according to claim 1, wherein the information processing apparatus determines that the action is an action for viewing the tag of the product by the customer.

  According to a third aspect of the present invention, when the photographing means for photographing the image and the tag are adjacent to each other, and the determination means is located in the image when the line of sight of the customer faces the photographing means, The information processing apparatus according to claim 1, wherein the information processing apparatus determines that the action is for viewing the tag of the product by a customer.

  According to a fourth aspect of the present invention, the tag is suspended from a lower part of the product, and when the determination means detects in the image that the tag is touched, The information processing apparatus according to claim 1, wherein the information processing apparatus determines that the action is for viewing a tag.

  According to the invention of claim 5, when a tag is hung from the product, and the distance between the tag and a predetermined position is less than or less than a predetermined distance, the product by the customer An information processing apparatus comprising: determination means for determining that the action is for viewing the tag; and counting means for counting each action determined by the determination means.

  According to a sixth aspect of the present invention, the tag is provided with two or more detection units for detecting movement, and the determination means detects movement detected by one detection unit and detection by the other detection unit. The information processing apparatus according to claim 5, wherein when the movement is in a reverse direction, the information processing apparatus determines that the action is for viewing the tag of the product by the customer.

  According to the seventh aspect of the present invention, the computer counts for each action determined by the determination means, the determination means for determining the action for viewing the product tag by the customer from the image taken by the customer. An information processing program for functioning as a means.

  According to the invention of claim 8, when a tag is suspended from a product and the distance between the tag and a predetermined position is less than or less than a predetermined distance, The information processing program for functioning as a determination unit that determines that the action is for viewing the tag of the product and a counting unit that counts for each action determined by the determination unit.

  According to the information processing apparatus of the first aspect, when the customer's behavior is counted for each behavior, it is possible to determine the behavior for the customer to see the product tag.

  According to the information processing apparatus of the second aspect, when the tag is attached to a position where it is necessary to touch the product in order to see the tag, it is possible to determine an action for viewing the tag.

  According to the information processing apparatus of the third aspect, when the photographing means for photographing an image and the tag are adjacent to each other, an action for viewing the tag can be determined.

  According to the information processing apparatus of the fourth aspect, when the tag is suspended from the lower part of the product, it is possible to determine an action for viewing the tag.

  According to the information processing apparatus of the fifth aspect, when the customer's behavior is counted for each behavior, it is possible to determine the behavior of the customer to see the product tag.

  According to the information processing apparatus of the sixth aspect, it is possible to determine that it is an action for the customer to see the tag of the product by the movement detected by the detection unit attached to the tag.

  According to the information processing program of the seventh aspect, when the customer's behavior is counted for each behavior, it is possible to determine the behavior for the customer to see the product tag.

  According to the information processing program of the eighth aspect, when the customer's behavior is counted for each behavior, it is possible to determine the behavior for the customer to see the product tag.

It is a conceptual module block diagram about the structural example of 1st Embodiment. It is explanatory drawing which shows the system configuration example in the case of implement | achieving 1st Embodiment. It is a flowchart which shows the process example (1) by 1st Embodiment. It is explanatory drawing which shows the process example (1) by 1st Embodiment. It is a flowchart which shows the process example (2) by 1st Embodiment. It is explanatory drawing which shows the process example (2) by 1st Embodiment. It is a flowchart which shows the process example (3) by 1st Embodiment. It is explanatory drawing which shows the process example (3) by 1st Embodiment. It is a conceptual module block diagram about the structural example of 2nd Embodiment. It is explanatory drawing which shows the system configuration example in the case of implement | achieving 2nd Embodiment. It is a flowchart which shows the process example by 2nd Embodiment. It is explanatory drawing which shows the process example (1) by 2nd Embodiment. It is explanatory drawing which shows the example of the tag used by 2nd Embodiment. It is explanatory drawing which shows the process example (2) by 2nd Embodiment. It is explanatory drawing which shows the example of a data structure of an operation | movement count table. It is a block diagram which shows the hardware structural example of the computer which implement | achieves this Embodiment.

Hereinafter, examples of various preferred embodiments for realizing the present invention will be described with reference to the drawings.
FIG. 1 is a conceptual module configuration diagram of a configuration example according to the first embodiment.
The module generally refers to components such as software (computer program) and hardware that can be logically separated. Therefore, the module in the present embodiment indicates not only a module in a computer program but also a module in a hardware configuration. Therefore, the present embodiment is a computer program for causing these modules to function (a program for causing a computer to execute each procedure, a program for causing a computer to function as each means, and a function for each computer. This also serves as an explanation of the program and system and method for realizing the above. However, for the sake of explanation, the words “store”, “store”, and equivalents thereof are used. However, when the embodiment is a computer program, these words are stored in a storage device or stored in memory. It is the control to be stored in the device. In addition, the modules may correspond to the functions on a one-to-one basis. However, in implementation, one module may be configured by one program, or a plurality of modules may be configured by one program. A module may be composed of a plurality of programs. The plurality of modules may be executed by one computer, or one module may be executed by a plurality of computers in a distributed or parallel environment. Note that one module may include other modules. Hereinafter, “connection” is used not only for physical connection but also for logical connection (data exchange, instruction, reference relationship between data, etc.). “Predetermined” means that the process is determined before the target process, and not only before the process according to this embodiment starts but also after the process according to this embodiment starts. In addition, if it is before the target processing, it is used in accordance with the situation / state at that time or with the intention to be decided according to the situation / state up to that point. When there are a plurality of “predetermined values”, they may be different values, or two or more values (of course, including all values) may be the same. In addition, the description having the meaning of “do B when it is A” is used in the meaning of “determine whether or not it is A and do B when it is judged as A”. However, the case where it is not necessary to determine whether or not A is excluded.
In addition, the system or device includes a plurality of computers, hardware, devices, and the like connected by communication means such as a network (including one-to-one communication connection), etc., and one computer, hardware, The case where it implement | achieves by an apparatus etc. is also included. “Apparatus” and “system” are used as synonymous terms. Of course, the “system” does not include a social “mechanism” (social system) that is an artificial arrangement.
In addition, when performing a plurality of processes in each module or in each module, the target information is read from the storage device for each process, and the processing result is written to the storage device after performing the processing. is there. Therefore, description of reading from the storage device before processing and writing to the storage device after processing may be omitted. Here, the storage device may include a hard disk, a RAM (Random Access Memory), an external storage medium, a storage device via a communication line, a register in a CPU (Central Processing Unit), and the like.

The information processing apparatus 100 according to the first embodiment determines (identifies) the behavior of a customer. As illustrated in the example of FIG. 1, an image reception module 120, an image processing module 130, an information processing module 140 and an information recording module 150.
In order to take measures for sales promotion, it is necessary to determine customer behavior (actions) with respect to a product in a store or the like and to count the behavior. This embodiment is used for that purpose. In particular, an action for a tag attached to a product is determined. The tag is a tag on which the attribute of the product is described. Examples of product attributes include price, manufacturer, and raw material. Specifically, a price tag or the like is applicable.

The imaging module 110 is connected to the image reception module 120 of the information processing apparatus 100. The photographing module 110 photographs a product and the periphery of the product (also referred to as a product area). In order to purchase the product and confirm the product, the customer enters the product area. For this reason, the image captured by the imaging module 110 may include a customer along with the product.
Specifically, the shooting module 110 is a 2D camera, a 3D camera, a thermo camera, an omnidirectional camera, or the like for shooting a still image or a moving image. Further, the imaging module 110 may be fixed, or may be capable of changing the angle, position, and the like. The relationship between the product and the imaging module 110 may be one-to-one, one-to-N, N-to-one, or N-to-N. . In this embodiment, a one-to-one case will be described as an example. In the case of N to 1, as described later, it is sufficient to recognize a product and determine the customer's behavior for each product. In the case of 1 to N, one product or a product of the product from a plurality of images. What is necessary is just to determine the action with respect to a tag and to obtain a final determination result from a plurality of determination results by using majority vote or the like.

The image reception module 120 is connected to the photographing module 110 and the image processing module 130. The image receiving module 120 receives an image shot by the shooting module 110 and passes the image to the image processing module 130.
The image processing module 130 is connected to the image receiving module 120 and the information processing module 140. The image processing module 130 includes a customer feature extraction module 132 and a tag recognition module 134.
The customer feature extraction module 132 extracts customer features from the image received by the image receiving module 120. For example, the product position, customer body, customer head position, hand (including arm) position, line-of-sight direction, face position, eyes, and the like are extracted as features. For example, the product photographed by the photographing module 110 is predetermined, and features (color, shape, size, etc.) of the predetermined product are extracted in advance and stored as a dictionary. The product can be recognized from within the image. In addition, the recognition of a person, the recognition of a person's part (hand, head, etc.), the direction of the line of sight, etc. are recognized using existing techniques. In addition, as a position, the position in the coordinate system in an image may be sufficient, and a relative position with respect to goods may be sufficient.
The tag recognition module 134 recognizes a product tag from the image received by the image receiving module 120. For example, the tags photographed by the photographing module 110 are predetermined, and the characteristics (color, shape, size, etc.) of the predetermined tags are extracted in advance and stored as a dictionary. The tag can be recognized from within the image. In addition, the tag may include a mark, an information image (for example, a two-dimensional code) that is different for each product or each product type, or may be coated with a paint that reacts to a specific wavelength. In addition, a device that emits a specific wavelength (sound) may be provided. The tag recognition module 134 recognizes these. In the case of a tag that responds to a specific wavelength, the imaging module 110 may include a sensor that responds to the specific wavelength.

The information processing module 140 is connected to the image processing module 130 and the information recording module 150. The information processing module 140 includes an action specifying module 142, a line-of-sight detection module 144, and a counting module 146.
The action specifying module 142 determines an action for viewing a tag of a product by the customer from an image in which the customer is taken.
The behavior identification module 142 saw the behavior that the customer entered the product area, the behavior that touched the product, the behavior that touched the tag, the behavior that took out the tag, the behavior that returned the tag, and the product as the behavior to be judged There are actions, actions that looked at the tag, and the like.
For example, when the tag is attached to a position where it is necessary to touch the product to see the tag, the behavior identifying module 142 detects the tag in the image after touching the product. It is determined that the action is for viewing the tag of the product. In this case, the tag is in the product and is attached to a position where it is necessary to touch the product in order to view the tag. This determination process will be described later with reference to FIGS.
In addition, when the photographing module 110 and the tag are adjacent to each other, and the customer's line of sight faces the photographing module 110 in the image, the behavior specifying module 142 is an action for seeing the tag of the product by the customer. Judge that there is. The processing result of the line-of-sight detection module 144 is used for line-of-sight detection. This determination process will be described later with reference to FIGS.
In addition, the tag is suspended from the lower part of the product, and when the action identifying module 142 detects that the tag has been touched in the image, the action is determined to be an action for the customer to see the tag of the product. To do. This determination process will be described later with reference to FIGS.

The line-of-sight detection module 144 detects the customer's line of sight using the processing result of the customer feature extraction module 132. For example, a line of sight to the imaging module 110 or a line of sight to something other than the imaging module 110 is detected.
The counting module 146 performs counting for each action determined by the action specifying module 142. That is, the number of times the action is performed is measured. You may make it count for every above-mentioned action, and you may make it count the frequency | count of predetermined action.
The information recording module 150 is connected to the information processing module 140. The information recording module 150 stores the counting result obtained by the counting module 146. Further, customer behavior may be stored in association with customer characteristics.

FIG. 2 is an explanatory diagram showing an example of a system configuration when the first embodiment is realized.
The information processing apparatus 100, the imaging module 110 </ b> A, the imaging module 110 </ b> B, the imaging module 110 </ b> C, the imaging module 110 </ b> D, and the product management apparatus 210 are connected via a communication line 290. The communication line 290 may be wireless, wired, or a combination thereof, for example, an intranet as a communication infrastructure, the Internet, or the like. A plurality of photographing modules 110 may be installed for each product in one store, or may be installed in a plurality of stores.
The merchandise management apparatus 210 obtains a processing result (information stored in the information recording module 150) by the information processing apparatus 100 and presents the information and the like to the user. In addition, in cooperation with POS (Point Of Sale system), etc., the customer's behavior with respect to the purchased product should be presented so that the behavior of the customer who did not purchase but entered the product area can be compared. May be.

FIG. 3 is a flowchart illustrating a processing example (1) according to the first embodiment.
In step S302, the imaging module 110 images the product area.
In step S304, the image reception module 120 receives an image.
In step S306, the customer feature extraction module 132 extracts customer features from the image.
In step S308, the tag recognition module 134 recognizes a tag from the image.
In step S310, the behavior identifying module 142 identifies the customer's behavior.
In step S312, the action specifying module 142 determines the specified action. If the action is an action in which the customer has entered the product area, the process proceeds to step S314. If the action touches the product or the tag, the process proceeds to step S316. If the action is to take out the tag, the process proceeds to step S318. If the action has been restored, the process proceeds to step S320.
In step S314, the counting module 146 counts an action in which a customer enters the product area.
In step S316, the counting module 146 counts an action that touches a product or a tag.
In step S318, the counting module 146 counts the action of taking out the tag.
In step S320, the counting module 146 counts the action of returning the tag.
In step S322, the counting module 146 records the count result in the information recording module 150.

FIG. 4 is an explanatory diagram illustrating a processing example (1) according to the first embodiment. In particular, an example of processing in which the behavior identifying module 142 identifies customer behavior in step S310 will be described. Hereinafter, an apparel store as a store and a shirt as a product will be described as an example.
In this example, the imaging module 110 is attached to the ceiling of the store and images a product area. Therefore, the photographed image is obtained by photographing the customer's head 430, hand 440, and shirt 410 from above, as in the example of the image column in FIG.
Further, the tag 420 is attached at a position where it is necessary to touch the shirt 410 in order to see the tag 420. For example, the tag 410 may be attached to the inside of the shirt 410, or the tag 420 may be turned upside down between the buttons of the shirt 410. In the example of the image column in FIG. 4, the tag 420 is also described. However, in the case other than FIG. 4C, it is in the shirt 410 and is not photographed in the image.

In the example of FIG. 4, it is assumed that the process proceeds in the direction of the arrow on the time axis 400.
In the example of FIG. 4 (A), an example is shown in which the behavior of the customer entering the product area is detected. This is because when the customer (customer's head 430, hand 440, etc.) is not photographed and it is detected that the customer's head 430, etc. stays for a predetermined time or more, the customer enters the product area. It may be determined as an action.
In the example of FIG. 4B, the case where the tag 420 of the shirt 410 is picked up is shown. By detecting that the customer's hand 440 enters the area within the shirt 410 and stays for a predetermined time or more, it is determined that the action is “touching the product”.
In the example of FIG. 4C, a case where the tag 420 is taken out to read the tag 420 of the shirt 410 is shown. From the state of the example of FIG. 4B, by detecting that the hand 440 (or tag 420) is away from the area of the shirt 410 by a predetermined distance and stays for a predetermined time or longer, the “tag. It is determined that the action is “taken out”. Here, as the “predetermined distance”, the length of the string 425 for attaching the tag 420 to the shirt 410 may be used. The “behavior for taking out the tag” may be set as “behavior for viewing the tag”.
In the example of FIG. 4D, the case where the tag 420 of the shirt 410 has been viewed and returned is shown. By detecting that the customer's hand 440 has moved away from the shirt 410 from the state of the example of FIG. 4C, it is determined that the action is “an action that returned the tag to the original”.

FIG. 5 is a flowchart illustrating a processing example (2) according to the first embodiment.
In step S502, the imaging module 110 captures the product area.
In step S504, the image reception module 120 receives an image.
In step S506, the customer feature extraction module 132 extracts customer features from the image.
In step S508, the action specifying module 142 specifies an action.
In step S510, the action specifying module 142 determines the specified action. If the action is an action in which the customer is in the product area, the process proceeds to step S512. If the action is the action of viewing the product, the process proceeds to step S514. If the action is the action of touching the product, the process proceeds to step S516. If it is action, it will progress to step S518.
In step S512, the counting module 146 counts an action in which a customer enters the product area.
In step S514, the counting module 146 counts the behavior of seeing the product.
In step S516, the counting module 146 counts the action of touching the product.
In step S518, the counting module 146 counts actions that have seen the tag.
In step S520, the counting module 146 records the count result in the information recording module 150.

FIG. 6 is an explanatory diagram illustrating a processing example (2) according to the first embodiment. In particular, an example of processing in which the behavior identifying module 142 identifies customer behavior in step S508 will be described.
In this example, the imaging module 110 (camera 690 in the figure) is installed at a position adjacent to the tag 620. Further, in order to photograph the shirt 610, the shirt 610 may be installed at a position away from the shirt 610. Specifically, in the example of FIG. 6, a tag 620 is attached to the lower side (floor) of the product, a camera 690 is installed beside the tag 620, and an image of the upper front side is taken.

In the example of FIG. 6A, the case where the customer touches the shirt 610 with the left hand is shown.
When it is detected that the customer 630 has stayed for a predetermined time or more from a state in which the customer 630 is not photographed, it may be determined that the customer has entered the product area.
Then, when the line-of-sight detection module 144 detects that the line of sight is directed to the shirt 610 and the line of sight is longer than a predetermined time, the line-of-sight detection module 144 determines that the action is “seeing the product”. The line-of-sight detection module 144 may detect the line of sight to the shirt 610 by detecting that the position of the pupil 651 in the eyeball 650 is in the direction of the shirt 610. Further, by detecting that the position of the white eye 652 in the eyeball 650 is opposite to the shirt 610, it may be detected that the line of sight to the shirt 610 is present.
Further, by detecting that the customer's left hand 640 has entered the area in the shirt 610 and stayed for a predetermined time or more, it is determined that the action is “touching a product”.
In the example of FIG. 6B, the case where the customer is looking at the tag 620 is shown.
When the line-of-sight detection module 144 detects that it is a line of sight to the camera 690 and the line of sight is longer than or equal to a predetermined time, it is determined that the action is “seeing the tag”. Since the tag 620 and the camera 690 are adjacent to each other, the line of sight to the camera 690 may be the line of sight to the tag 620. The line-of-sight detection module 144 may detect that it is the line of sight to the camera 690 by detecting that the position of the pupil 655 in the eyeball 654 is the center in the eyeball 654. Further, when the pupil 655 in the eyeball 654 is surrounded by the white eye 656, or when the area ratio of the pupil 655 in the eyeball 654 is greater than or equal to a predetermined threshold, the line of sight to the camera 690 is detected. You may make it do. Note that the left hand 640 may be the position of the left hand 644 that is touching the shirt 610, or may be the position of the left hand 642 that is away from the shirt 610. The “behavior for seeing the tag” may be set as “behavior for seeing the tag”.

FIG. 7 is a flowchart illustrating a processing example (3) according to the first embodiment.
In step S <b> 702, the imaging module 110 images the product area.
In step S704, the image reception module 120 receives an image.
In step S706, the customer feature extraction module 132 extracts customer features from the image.
In step S708, the tag recognition module 134 recognizes a tag from the image.
In step S710, the action specifying module 142 specifies an action.
In step S712, the action specifying module 142 determines the specified action. If the action is an action in which a customer has entered the product area, the process proceeds to step S714. If the action has touched the product, the process proceeds to step S716. If the action has touched the tag, the process proceeds to step S718.
In step S714, the counting module 146 counts the behavior that the customer has entered the product area.
In step S716, the counting module 146 counts the action of touching the product.
In step S718, the counting module 146 counts actions that touch the tag.
In step S720, the counting module 146 records the count result in the information recording module 150.

FIG. 8 is an explanatory diagram illustrating a processing example (3) according to the first embodiment. In particular, an example of processing in which the behavior identifying module 142 identifies customer behavior in step S710 will be described.
In this example, the photographing module 110 is attached so as to photograph the product area from the side. That is, it is a position where the shirt 810 and the customer 830 are photographed from the side.
The tag 820 is attached in a state of being suspended from the lower part of the shirt 810. Therefore, in the image in which the customer is not photographed, the shirt 810 and the tag 820 are not in the overlapping position. That is, the shirt 810 is in the upper area, and the tag 820 is photographed in the area below it. Moreover, the tag 820 may be turned upside down in a suspended state. In this case, in order for the customer to see the contents of the tag 820, the possibility of picking up the tag 820 increases.

In the example of FIG. 8A, the case where the customer touches the shirt 810 with the hand 840 is shown.
When it is detected that the customer 830 has stayed for a predetermined time or more from a state in which the customer 830 is not photographed, it may be determined that the customer has entered the product area.
Then, by detecting that the hand 840 enters the region in the shirt 810 and stays for a predetermined time or longer, it is determined that the action is “touching a product”.
In the example of FIG. 8B, a case where the customer touches the tag 820 is shown.
By detecting that the hand 842 enters the area in the tag 820 and stays for a predetermined time or longer, it is determined that the action is “touching the tag”. Note that this “behavior touching the tag” may be “behavior for viewing the tag”.

FIG. 9 is a conceptual module configuration diagram of a configuration example according to the second embodiment.
The information processing apparatus 900 according to the second embodiment determines customer behavior. As shown in the example of FIG. 9, the information processing apparatus 900 includes a detection information receiving module 920, an information processing module 940, and an information recording module 950. Have.
In the present embodiment, a tag is suspended from the product. A device for measuring the distance is attached to the tag. For example, it may be an RFID (Radio Frequency IDentifier) that can measure the distance by the strength of radio waves. And the apparatus which performs wireless communication with the apparatus attached to the tag is installed in the vicinity of goods. For example, when the product is clothes, the device may be embedded in a hanger or the like. In this case, the distance between the tag and the hanger when the tag is touched by the customer is different from the distance between the tag and the hanger when the tag is suspended. In order to increase the certainty of different distances in the state where the tag is picked up, the tag may be suspended from the lower part of the product.

The detection module 910 is connected to the detection information reception module 920 of the information processing apparatus 900. As described above, the detection module 910 detects the distance between the device attached to the tag and a predetermined position.
The detection information receiving module 920 is connected to the detection module 910 and the information processing module 940. The detection information reception module 920 receives the distance detected by the detection module 910 and passes it to the information processing module 940.
The information processing module 940 is connected to the detection information receiving module 920 and the information recording module 950. The information processing module 940 includes an action specifying module 942 and a counting module 946.
The action specifying module 942 determines that the action is for the customer to see the tag of the product when the distance between the tag and the predetermined position is less than or less than the predetermined distance. To do. Here, as the “predetermined position”, for example, there is a position of a device (for example, the aforementioned hanger) provided with a device corresponding to the device attached to the tag. The “predetermined distance” is a distance between the tag and a predetermined position in a state where the tag is suspended (that is, a state where the customer does not touch the tag). When the customer touches the tag (when picked up), the tag and the predetermined position are close. Therefore, if the above conditions are met (if the distance between the tag and the predetermined position is less than or less than the predetermined distance), the customer's action to view the tag of the product It judges with being.
The counting module 946 has a function equivalent to that of the counting module 146.
The information recording module 950 is connected to the information processing module 940. The information recording module 950 has the same function as the information recording module 150.

FIG. 10 is an explanatory diagram illustrating an example of a system configuration when the second embodiment is realized.
The information processing apparatus 900, the detection module 910A, the detection module 910B, the detection module 910C, the detection module 910D, and the product management apparatus 210 are connected via a communication line 290, respectively. This is the same as the system configuration example shown in the example of FIG. 2, and the imaging module 110 is changed to the detection module 910 and the information processing apparatus 100 is changed to the information processing apparatus 900.

FIG. 11 is a flowchart illustrating a processing example according to the second exemplary embodiment.
In step S1102, the detection module 910 detects the state of the sensor / radio wave transmission module 1225.
In step S1104, the detection information receiving module 920 receives state information.
In step S1106, the action specifying module 942 specifies an action.
In step S1108, the action specifying module 942 determines the specified action. If the action is an action touching a product, the process proceeds to step S1110, and if the action is an action touching a tag, the process proceeds to step S1112.
In step S <b> 1110, the counting module 946 counts actions that touched the product.
In step S1112, the counting module 946 counts actions that touch the tag.
In step S1114, the counting module 946 records the count result in the information recording module 950.

FIG. 12 is an explanatory diagram illustrating a processing example (1) according to the second embodiment. In particular, a description will be given of a processing example in which the behavior identifying module 942 identifies the behavior of the customer in step S1106.
As shown in the example of FIG. 12A, the sensor / radio wave transmission module 1225 is suspended from the lower portion of the shirt 1210. A sensor / radio wave transmission module 1225 is attached to the tag 1220. A radio wave transmission / reception module 1250 is attached to the hanger 1255. The sensor / radio wave transmission module 1225 communicates with the radio wave transmission / reception module 1250, and the radio wave transmission / reception module 1250 measures the distance to the sensor / radio wave transmission module 1225. Then, the measured distance is transmitted to the detection information receiving module 920. Further, an acceleration sensor or the like is attached to the sensor / radio wave transmission module 1225, detects the movement of the tag 1220 (sensor / radio wave transmission module 1225), transmits information indicating the movement to the radio wave transmission / reception module 1250, and transmits the radio wave. The transmission / reception module 1250 transmits information indicating the movement to the detection information reception module 920. Specifically, the movement of up, down, left, and right is detected, and information on the movement is transmitted by radio waves.
When customer 1230 touches shirt 1210 with hand 1240, tag 1220 is likely to shake. Therefore, when motion is detected by an acceleration sensor or the like of the sensor / radio wave transmission module 1225 and the distance between the sensor / radio wave transmission module 1225 and the radio wave transmission / reception module 1250 is a predetermined distance (or motion When the change in the distance before and after is within a predetermined range), it is determined that the action is a touching product.
When the customer 1230 touches the tag 1220 with the hand 1242 as in the example shown in FIG. 12B, the distance between the sensor / radio wave transmission module 1225 and the radio wave transmission / reception module 1250 is the example shown in FIG. It becomes shorter than the case of. Therefore, when movement is detected by an acceleration sensor or the like of the sensor / radio wave transmission module 1225 and the distance between the sensor / radio wave transmission module 1225 and the radio wave transmission / reception module 1250 is less than or less than a predetermined distance, , It is determined that the action is “touching the tag”. Furthermore, as a condition for this determination, it may be added that the motion is detected by the acceleration sensor of the sensor / radio wave transmission module 1225 or the like. Note that this “behavior touching the tag” may be “behavior for viewing the tag”.

FIG. 13 is an explanatory diagram illustrating an example of a tag 1300 used in the second embodiment. The tag 1220 shown in the example of FIG. 12 is a general tag that suspends one of the tags 1220 with a thread, but may be the tag 1300 shown in the example of FIG.
Four threads 1390 (thread 1390A, thread 1390B, thread 1390C, thread 1390D) are attached to the tag 1300 at four corners to keep the tag 1300 parallel to the floor. If the tag 1300 is to be kept parallel to the floor, the number of the threads 1390 may not be four (for example, three, five, etc.), and other devices (jigs) may be used. You may make it hang.
The tag 1300 is provided with two sensors / radio wave transmission modules 1310 (sensor / radio wave transmission module 1310A, sensor / radio wave transmission module 1310B) for detecting movement. The sensor / radio wave transmission module 1310 is desirably provided one above and one below in a direction in which characters (including graphics) described on the tag 1300 can be read. In other words, in order to read characters or the like, it is necessary to tilt the tag 1300. At that time, one sensor / radio wave transmission module 1310 moves upward, and the other sensor / radio wave transmission module 1310 moves downward. Will do. In addition, it is desirable that characters and the like in the tag 1300 are written on the back side (side that cannot be seen in a normal state). This is because the tag 1300 must be picked up in order to read characters and the like. Three or more sensors / radio wave transmission modules 1310 may be attached to one tag 1300.

FIG. 14 is an explanatory diagram illustrating a processing example (2) using the tag 1300 according to the second embodiment.
(1) In a certain apparel store, when a customer picks up a product 1410, two sensors / radio wave transmission modules 1310A and sensors / radio wave transmission modules 1310B attached to a tag 1300 attached to the product 1410 transmit signals, The radio wave receiving module 1450 receives the signal. The radio wave reception module 1450 receives radio wave information transmitted from the sensor / radio wave transmission modules 1310A and 1310B. It is attached to each product type. Moreover, it has the product information to which the acceleration sensor to which the radio wave information is transmitted is attached.
Information on the product 1410 recorded in advance in the radio wave receiving module 1450 and operation information of the sensor / radio wave transmission module 1310A and sensor / radio wave transmission module 1310B are transmitted to the information processing apparatus 1460 together. When the operation information of the sensor / radio wave transmission module 1310A and the sensor / radio wave transmission module 1310B is “z-axis positive direction (upward)”, the information processing device 1460 having a function as the information processing device 900 (behavior specifying module) 942), it is determined that “a product has been picked up” and recorded in the information recording module 1470 having the function as the information recording module 950. As described above, when the movements detected by both the sensors / radio wave transmission modules 1310 are movements in the same direction, it is determined that “the product has been picked up by the hand”.

(2) After that, if the customer picks up the tag 1300 (or turns it over), the signals transmitted from the sensor / radio wave transmission module 1310A and the sensor / radio wave transmission module 1310B are converted into radio waves as in the case of (1). The reception module 1450 receives the information and transmits the information to the information processing apparatus 1460 together with the information about the product 1410. The operation information of the sensor / radio wave transmission module 1310A and the sensor / radio wave transmission module 1310B is “z-axis positive direction (upward)” and “z-axis negative direction (downward)”, respectively. It is determined by the module 942) as “behavior touching the tag” (or “behavior turning the tag over”) and recorded in the information recording module 1470. The “behavior touching the tag” (or “behavior turning the tag upside down”) may be set as “behavior for viewing the tag”. That is, when the movement detected by one sensor / radio wave transmission module 1310 and the movement detected by the other sensor / radio wave transmission module 1310 are in the opposite direction, it is determined that the action is for the customer to see the product tag. To do.

FIG. 15 is an explanatory diagram showing an example of the data structure of the operation / count table 1500. The operation / count table 1500 is a result of processing recorded in the information recording module 150 and the information recording module 950 (information recording module 1470) in the first embodiment and the second embodiment.
The operation / count table 1500 has an operation column 1510 and a count column 1520. The action column 1510 stores actions. The count column 1520 stores the number of times that the action has been performed.
Further, the processing result may include the date and time when the action was performed (may be year, month, day, hour, minute, second, second or less, or a combination thereof).
Using these processing results, for example, “an action in which a product is picked up” may be determined to be “interested in the product”, and “an action for viewing a tag” may be “in the product”. It may be determined that the user is interested in the price, or if an action is detected a plurality of times within a predetermined time, it may be determined that the action is by one person (same customer). You may make it notify a salesclerk of these judgment results in real time.

  A hardware configuration example of the information processing apparatus (the information processing apparatus 100 or the information processing apparatus 900) according to the present embodiment will be described with reference to FIG. The configuration illustrated in FIG. 16 is configured by, for example, a personal computer (PC) or the like, and illustrates a hardware configuration example including a data reading unit 1617 such as a scanner and a data output unit 1618 such as a printer.

  A CPU (Central Processing Unit) 1601 includes various modules described in the above-described embodiments, that is, the image reception module 120, the image processing module 130, the customer feature extraction module 132, the tag recognition module 134, the information processing module 140, an action Processing according to a computer program describing the execution sequence of each module such as the identification module 142, the line-of-sight detection module 144, the counting module 146, the detection information receiving module 920, the information processing module 940, the behavior identifying module 942, the counting module 946, and the like. A control unit to be executed.

  A ROM (Read Only Memory) 1602 stores programs, calculation parameters, and the like used by the CPU 1601. A RAM (Random Access Memory) 1603 stores programs used in the execution of the CPU 1601, parameters that change as appropriate during the execution, and the like. These are connected to each other by a host bus 1604 including a CPU bus or the like.

  The host bus 1604 is connected to an external bus 1606 such as a PCI (Peripheral Component Interconnect / Interface) bus through a bridge 1605.

  A keyboard 1608 and a pointing device 1609 such as a mouse are input devices operated by an operator. The display 1610 includes a liquid crystal display device or a CRT (Cathode Ray Tube), and displays various types of information as text or image information.

  An HDD (Hard Disk Drive) 1611 includes a hard disk, drives the hard disk, and records or reproduces a program executed by the CPU 1601 and information. The hard disk stores images, detection information, counting results, and the like. Further, various computer programs such as various other data processing programs are stored.

  The drive 1612 reads data or a program recorded on a removable recording medium 1613 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory, and the data or program is read out from the interface 1607 and the external bus 1606. , A bridge 1605, and a RAM 1603 connected via the host bus 1604. The removable recording medium 1613 can also be used as a data recording area similar to the hard disk.

  The connection port 1614 is a port for connecting an external connection device 1615 and has a connection unit such as USB, IEEE1394. The connection port 1614 is connected to the CPU 1601 and the like via the interface 1607, the external bus 1606, the bridge 1605, the host bus 1604, and the like. A communication unit 1616 is connected to a communication line and executes data communication processing with the outside. The data reading unit 1617 is a scanner, for example, and executes document reading processing. The data output unit 1618 is a printer, for example, and executes document data output processing.

  Note that the hardware configuration of the information processing apparatus shown in FIG. 16 shows one configuration example, and the present embodiment is not limited to the configuration shown in FIG. 16, and the modules described in this embodiment are executed. Any configuration is possible. For example, some modules may be configured with dedicated hardware (for example, Application Specific Integrated Circuit (ASIC), etc.), and some modules are in an external system and connected via a communication line In addition, a plurality of systems shown in FIG. 16 may be connected to each other via communication lines so as to cooperate with each other. Further, it may be incorporated in a copying machine, a fax machine, a scanner, a printer, a multifunction machine (an image processing apparatus having any two or more functions such as a scanner, a printer, a copying machine, and a fax machine).

  Further, in the description of the above-described embodiment, “more than”, “less than”, “greater than”, and “less than (less than)” in a comparison with a predetermined value contradicts the combination. As long as the above does not occur, “larger”, “smaller (less than)”, “more”, and “less” may be used.

The program described above may be provided by being stored in a recording medium, or the program may be provided by communication means. In that case, for example, the above-described program may be regarded as an invention of a “computer-readable recording medium recording the program”.
The “computer-readable recording medium on which a program is recorded” refers to a computer-readable recording medium on which a program is recorded, which is used for program installation, execution, program distribution, and the like.
The recording medium is, for example, a digital versatile disc (DVD), which is a standard established by the DVD Forum, such as “DVD-R, DVD-RW, DVD-RAM,” and DVD + RW. Standard “DVD + R, DVD + RW, etc.”, compact disc (CD), read-only memory (CD-ROM), CD recordable (CD-R), CD rewritable (CD-RW), Blu-ray disc ( Blu-ray (registered trademark) Disc), magneto-optical disk (MO), flexible disk (FD), magnetic tape, hard disk, read-only memory (ROM), electrically erasable and rewritable read-only memory (EEPROM (registered trademark)) )), Flash memory, Random access memory (RAM) SD (Secure Digital) memory card and the like.
The program or a part of the program may be recorded on the recording medium for storage or distribution. Also, by communication, for example, a local area network (LAN), a metropolitan area network (MAN), a wide area network (WAN), a wired network used for the Internet, an intranet, an extranet, or a wireless communication It may be transmitted using a transmission medium such as a network or a combination of these, or may be carried on a carrier wave.
Furthermore, the program may be a part of another program, or may be recorded on a recording medium together with a separate program. Moreover, it may be divided and recorded on a plurality of recording media. Further, it may be recorded in any manner as long as it can be restored, such as compression or encryption.

DESCRIPTION OF SYMBOLS 100 ... Information processing apparatus 110 ... Image pick-up module 120 ... Image reception module 130 ... Image processing module 132 ... Customer feature extraction module 134 ... Tag recognition module 140 ... Information processing module 142 ... Action specification module 144 ... Gaze detection module 146 ... Count module 150 ... information recording module 210 ... merchandise management device 290 ... communication line 900 ... information processing device 910 ... detection module 920 ... detection information receiving module 940 ... information processing module 942 ... action specifying module 946 ... counting module 950 ... information recording module 1450 ... radio wave Receiving module 1460 ... Information processing device 1470 ... Information recording module

Claims (8)

A determination means for determining an action for viewing a tag of a product by the customer from an image in which the customer is photographed;
An information processing apparatus comprising: a counting unit that counts for each action determined by the determination unit. The tag is attached to a position where it is necessary to touch the product in order to see the tag,
The said determination means, when the said tag is detected in the said image after touching the said product, it determines with it being the action for seeing the said tag of the said product by the said customer. The information processing apparatus described in 1. The photographing means for photographing the image and the tag are adjacent to each other,
The said determination means determines that it is the action for seeing the said tag of the said product by the customer, when the eyes | visual_axis of the said customer is facing the said imaging | photography means in the said image. The information processing apparatus described in 1. The tag is suspended at the bottom of the product,
The said determination means determines that it is the action for seeing the said tag of the said goods by the said customer, when it detects in the said image that the said tag was touched. Information processing device. A tag is hung on the product, and the customer can view the tag of the product when the distance between the tag and the predetermined position is less than or less than the predetermined distance. A determination means for determining that the action is an action;
An information processing apparatus comprising: a counting unit that counts for each action determined by the determination unit. Two or more detection parts for detecting movement are attached to the tag,
When the movement detected by one detection unit and the movement detected by the other detection unit are in the opposite direction, the determination unit determines that the action is for the customer to see the tag of the product. The information processing apparatus according to claim 5. Computer
A determination means for determining an action for viewing a tag of a product by the customer from an image in which the customer is photographed;
An information processing program for functioning as counting means for counting each action determined by the determining means. Computer
A tag is hung on the product, and the customer can view the tag of the product when the distance between the tag and the predetermined position is less than or less than the predetermined distance. A determination means for determining that the action is an action;
An information processing program for functioning as counting means for counting each action determined by the determining means.

Images