JP2016045743A - Information processing apparatus and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an information processing apparatus which records a commodity that a customer has browsed with interest, and a program.SOLUTION: An information processing apparatus includes input means, detection means, determination means, generation means, and storage control means. The input means receives an input of an image captured by an imaging section which images a predetermined area of a shop. The detection means detects a face of a person from the image received by the input means. The determination means determines a commodity browsed by the person of the face detected by the detection means. The generation means generates browsing history information including behavior history of the person of the face detected by the detection means, who has browsed the commodity. The storage control means stores the browsing history information generated by the generation means, by person of the face detected by the detection means.SELECTED DRAWING: Figure 8

Description

  Embodiments described herein relate generally to an information processing apparatus and a program.

  In a store such as a supermarket, a customer purchases a product when the product is sold when the customer knows the product, is interested, and has a desire to acquire the product. In other words, the store can expect an increase in sales by grasping the behavior of the customer and the flow line in which the customer travels around the store, and placing products that the customer is interested in on the flow line.

  In recent years, a face authentication system for authenticating a person's face included in an image captured by a camera has been proposed. The conventional face authentication system can determine that the person of the detected face image is the same person when face images having the same characteristics are included in a series of continuous imaging data. is there. Using this technology, it is possible to determine that a customer has come to the sales floor by performing face authentication on the customer who has come to the sales floor.

  However, in order to grasp the products that the customers who travel around stores and facilities are interested in, it is insufficient as information for predicting the behavior of the customers only by grasping the customer's position and staying time.

  The problem to be solved by the present invention is to provide an information processing apparatus and program capable of recording a product that a customer is interested in and browses.

  The information processing apparatus according to the embodiment includes an input unit, a detection unit, a determination unit, a generation unit, and a storage control unit. The input means receives an input of an image captured by an imaging unit that captures a predetermined area of the store. The detection means detects a human face from the image received by the input means. The determination means determines a product viewed by the person whose face is detected by the detection means. The generating unit generates browsing history information in which an action history of browsing the product by the person of the face detected by the detecting unit is recorded. The storage control unit stores the browsing history information generated by the generation unit for each person of the face detected by the detection unit.

FIG. 1 is an explanatory diagram schematically showing the configuration of the person authentication system. FIG. 2 is an explanatory diagram showing an example when the person authentication system is installed in a store. FIG. 3 is a block diagram illustrating a hardware configuration of the server apparatus. FIG. 4 is an explanatory diagram showing an example of the data configuration of the person authentication dictionary. FIG. 5 is a block diagram illustrating a hardware configuration of the camera. FIG. 6 is a block diagram showing a functional configuration of the person authentication system. FIG. 7 is an explanatory diagram illustrating an example of a frame image used for determining a product that a person is browsing. FIG. 8 is a flowchart showing the flow of the person authentication process. FIG. 9 is a flowchart showing the flow of browsing history information generation processing.

  Hereinafter, an information processing apparatus and a program according to embodiments will be described in detail with reference to the accompanying drawings. In addition, this invention is not limited by this embodiment.

  FIG. 1 is an explanatory diagram schematically showing the configuration of the person authentication system 1. As shown in FIG. 1, the person authentication system 1 includes a server device 10 and two types of cameras 20a and 20b having different uses. The camera 20a performs face authentication as to whether or not the person is registered in a person authentication dictionary D1 (see FIG. 3) described later, and captures a moving image for registration in the person authentication dictionary D1 (see FIG. 3). Used in. On the other hand, the camera 20b is used for capturing a moving image for excluding a person registered in the person authentication dictionary D1 (see FIG. 3) from the person authentication target. The person authentication system 1 includes one or more cameras 20a and 20b, respectively, and images a predetermined area. Here, the server apparatus 10 and the cameras 20 a and 20 b are connected by a LAN (Local Area Network) 30. The LAN 30 may be wired or wireless. The LAN 30 may be various public networks.

  The server device 10 is an information processing device that authenticates a person's face from moving images captured by the cameras 20a and 20b, and generates browsing history information in which the history of products viewed by the person is stored from the moving images. Then, the server device 10 stores the generated browsing history information for each person. The server device 10 may be a plurality of clustered information processing devices.

  Next, a case where the person authentication system 1 is installed in a store M such as a supermarket will be described as an example. Here, FIG. 2 is an explanatory diagram showing an example when the person authentication system 1 is installed in the store M.

  The store M shown in FIG. 2 roughly includes a backyard M1, a sales floor M2, and an entrance M3. The backyard M1 is a place where operations relating to store operations such as a product warehouse, product packaging, and product ordering are performed. The server device 10 is installed in the backyard M1.

  The sales floor M2 is a place where products are sold. In addition, the sales floor M2 includes a camera 20a that captures a person traveling around the store M, a POS (Point Of Sale) terminal 40 for registering sales of products that the person purchases, and a shelf 50 that displays the products. Provided. When a person who is a customer visits the store M, he or she visits the sales floor M2 and selects a product to be purchased. At that time, the camera 20a images a person. Then, the server device 10 authenticates the face of the person included in the moving image captured by the camera 20a, and records the identification information of the camera 20a captured for each person and the imaging time. Thereby, action can be tracked for every person. Furthermore, the server device 10 can generate browsing history information by determining a product that the person is browsing from the direction of the face of the person. In the store M shown in FIG. 2, four cameras 20a, four POS terminals 40, and four sets of shelves 50 are described. May be set.

  Further, the camera 20 a shown in FIG. 2 is arranged in a direction parallel to the shelf 50. However, the arrangement location of the camera 20a and the direction in which the camera 20a is directed are not limited. For example, the camera 20 a may be arranged on the shelf 50 and arranged in a direction perpendicular to the shelf 50.

  The entrance / exit M3 is an entrance / exit of a person entering / exiting the sales floor M2. Further, the entrance / exit M3 includes a camera 20b directed in a direction for capturing an image of the face of a person leaving the sales floor M2. When the camera 20a captures a person's face, the server apparatus 10 determines whether there is already person authentication information. In that case, the processing load of the server apparatus 10 becomes large compared with all the person authentication information that has visited the store. Therefore, by excluding persons who leave the sales floor M2, an increase in processing load can be suppressed. In addition, you may install the camera 20a toward the direction which images the face of the person who enters the sales floor M2 in the entrance / exit M3. This makes it possible to authenticate the faces of all persons who have visited the store.

  Next, each part which the person authentication system 1 of embodiment has is demonstrated.

  First, the hardware configuration of the server device 10 will be described. FIG. 3 is a block diagram illustrating a hardware configuration of the server device 10. The server device 10 is an information processing device that controls the person authentication system 1. As shown in FIG. 3, the server device 10 includes a CPU (Central Processing Unit) 11, a memory 12, a storage unit 13, a network interface 14, a display unit 15, an input unit 16, and a clock unit 17. Prepare.

  The CPU 11 controls the entire server device 10. The memory 12 temporarily stores data. The storage unit 13 is readable and writable by commands from the CPU 11 and stores programs, data, and the like. The storage unit 13 also stores a person authentication program P1, a person authentication dictionary D1, a gender face authentication dictionary D2, an age face authentication dictionary D3, and a product layout file F1.

  The person authentication program P1 is a program that authenticates a person's face from moving images captured by the cameras 20a and 20b, and stores the person authentication information of the authenticated person in the person authentication dictionary D1.

  The person authentication dictionary D1 stores person authentication information for each person authenticated from a moving image captured by the camera 20a in the past. Here, FIG. 4 is an explanatory diagram showing an example of the data configuration of the person authentication dictionary D1. The person authentication dictionary D1 has one or more person authentication information associated with the person identification information. The person identification information is identification information assigned to each person and capable of identifying the person. The person authentication information is information related to person authentication extracted from the moving image. The person authentication information may be information related to person authentication extracted from a frame image representing a moving image with the front of the person faced.

  The person authentication information includes a moving image, a feature amount, camera identification information, an imaging time, and one or more browsing history information. The moving image is a moving image obtained by authenticating the person. The moving image may be a frame image representing a moving image in which the front of a person's face is directed. The feature amount is a feature amount of the person extracted from the moving image in which the person is authenticated. Here, the feature amount of a person is a feature amount extracted from a person's face, a feature amount extracted from a person's clothing, or the like. The feature amount of clothes is, for example, the color of clothes worn by a person. The camera identification information is identification information for identifying the camera 20a that captured the moving image. The imaging time is the time when the camera 20a captures a moving image. The browsing history information is information in which an action history of a person browsing a product in a moving image is stored for each product.

  The browsing history information includes product identification information of a product browsed by a person, a browsing time, and a browsing count. The product identification information is identification information that can identify the product. The browsing time is an action history indicating the total time when a person browsed products of the product identification information. The number of times of browsing is an action history indicating the total number of times that a person has browsed products of the product identification information.

  The gender face authentication dictionary D2 is information in which facial feature points for each gender are stored. That is, the gender face authentication dictionary D2 has a feature point of a male face and a feature point of a female face. The gender face authentication dictionary D2 is used for comparing a feature amount extracted from a person's face image with a feature amount stored in the gender face authentication dictionary D2 to determine whether the person is male or female. The

  The age-specific face authentication dictionary D3 is information in which facial feature points for each age are stored. The age-specific face authentication dictionary D3 is used to determine the age of a person by comparing the feature amount extracted from the person's face image with the feature amount stored in the age-specific face authentication dictionary D3. The age-specific face authentication dictionary D3 may be for each age. Alternatively, the age-specific face authentication dictionary D3 may be a category indicating age such as childhood, boyhood, adolescence, middle age, middle age, and older age.

  The product layout file F1 is a file in which the product arrangement of the store M2 of the store M is stored. The product layout file F1 stores, for example, shelf identification information that can identify the shelf 50 and product identification information in association with each other. As a result, it can be seen which merchandise is placed on which shelf 50. Or the coordinate which shows the position where goods are arranged may be memorized in the moving picture which camera 20a picturized.

  The network interface 14 is connected to the LAN 30. The LAN 30 may be various public networks. The display unit 15 is a display that displays various types of information. The input unit 16 controls information input from an input device such as a keyboard or a mouse. Alternatively, the input unit 16 controls information input from the touch panel stacked on the display unit 15. The clock unit 17 measures time.

  Next, the cameras 20a and 20b will be described. FIG. 5 is a block diagram illustrating a hardware configuration of the cameras 20a and 20b. Although the camera 20a and the camera 20b have different uses, the functions and configurations are the same. Since the camera 20a and the camera 20b have different uses, functions and configurations may be different. Therefore, the camera 20a will be described below as an example.

  The camera 20 a is an imaging device that captures an image of a predetermined area where the camera 20 a is installed and transmits the captured moving image to the server device 10. As shown in FIG. 5, the camera 20 a includes a CPU 21, a memory 22, a storage unit 23, a network interface 24, an imaging unit 25, and a clock unit 26.

  The CPU 21 controls the entire camera 20a. The memory 22 temporarily stores data. The storage unit 23 can be read and written by commands from the CPU 21 and stores programs, data, and the like. The network interface 24 is connected to the LAN 30. The clock unit 26 measures time.

  The imaging unit 25 captures a subject such as a person or an object existing within the imaging range of the camera 20a, and collects color image information. For example, the imaging unit 25 detects light reflected from the subject surface with a light receiving element, and converts visible light into an electrical signal. The light receiving element is, for example, a complementary metal oxide semiconductor (CMOS) or a charge coupled device (CCD). And the imaging part 25 produces | generates the color image information of 1 frame corresponding to an imaging range by converting the electrical signal into digital data. The color image information for one frame includes, for example, color information in which RGB (Red Green Blue) values are associated with each pixel included in the one frame, and a luminance value for each pixel included in the one frame. And associated luminance information.

  The imaging unit 25 generates a moving image by generating color image information of a plurality of continuous frames from visible light detected one after another. Note that the color image information generated by the imaging unit 25 may be output as a color image in which the RGB values of each pixel are arranged in a bitmap.

  Next, characteristic functions of the person authentication system 1 according to the embodiment will be described. FIG. 6 is a block diagram showing a functional configuration of the person authentication system 1.

  The CPUs 21 of the cameras 20 a and 20 b function as the imaging control unit 211 and the transmission unit 212 by operating according to the control program stored in the memory 22 or the storage unit 23.

  The imaging control unit 211 controls the imaging unit 25 of the cameras 20a and 20b to capture a moving image of a predetermined area where the cameras 20a and 20b are installed. Further, the imaging control unit 211 measures the imaging time when the imaging unit 25 images.

  The transmission unit 212 sequentially transmits the moving image captured by the imaging control unit 211 and the imaging time to the server device 10. At this time, the transmission unit 212 may transmit a moving image for each frame image, or may transmit a moving image for each of a plurality of frame images. Here, the frame image indicates color image information for each frame of a moving image.

  The CPU 11 of the server device 10 operates according to the control program stored in the memory 12 or the storage unit 13, thereby receiving the reception unit 111, the face detection unit 112, the face angle determination unit 113, the position calculation unit 114, It functions as a determination unit 115, a browsing history generation unit 116, a feature amount extraction unit 117, an attribute determination unit 118, a similarity calculation unit 119, a similarity determination unit 120, a registration unit 121, and an exclusion unit 122. .

  The receiving unit 111 as input means sequentially receives the moving images transmitted from the cameras 20a and 20b and the imaging time. Thereby, the receiving unit 111 receives input of imaging data such as a moving image.

  The face detection unit 112 serving as detection means detects a human face included in the moving image from the moving image received by the receiving unit 111. Further, the face detection unit 112 assigns new person identification information for each detected person. Specifically, the face detection unit 112 obtains coordinates indicating the face area using luminance information of the frame image. Note that any method may be used as a method of obtaining the face area. As an example of the detection method, there is a method of detecting a position that gives the highest correlation value as a face region by obtaining a correlation value while moving a template prepared in advance in input image data. Alternatively, a method using a face extraction method using an eigenspace method or a subspace method may be applied. The shape of the face area to be detected may be any shape, and the shape of the face area may be varied depending on the detection result of the face orientation and size.

  By the way, in the case of moving images captured by the cameras 20a, 20b, etc., it is assumed that the detected face of the same person is continuously shown over a series of consecutive frame images. For this reason, it is necessary to perform tracking processing of a person's face so that they can be associated as the same person. As a method for realizing the tracking process, a method of estimating and associating a position where the face detected using the optical flow exists in the next frame is used. This tracking process is continued until the face area associated with the same person is not detected by the face detection unit 112 and the tracking end is determined.

  The face angle determination unit 113 determines the face angle of the person included in the moving image captured by the camera 20a, that is, the direction in which the face is facing. Any method may be used as the method of determining the face orientation angle performed by the face angle determination unit 113, and a known technique is used. As an example, a face image template with an angle label is stored in the storage unit 13 in advance. The face angle determining unit 113 compares the captured moving image with a face image template with an angle label to determine a face angle. Alternatively, a method of detecting a characteristic part of the face and determining a face angle based on the position coordinates of the characteristic part may be used.

  The position calculation unit 114 calculates a position where a person included in the moving image captured by the camera 20a is present. The camera 20a is arranged at a position for imaging a predetermined area of the sales floor M2. The face detection unit 112 detects a face image of a person from the moving image and calculates coordinates indicating the position where the face image is detected. Therefore, the position calculation unit 114 calculates the position where the person of the face image included in the moving image is present based on the imaging range of the camera 20a and the coordinates of the face image.

  The product determination unit 115 serving as a determination unit determines a product viewed by a person whose face is detected by the face detection unit 112 from a moving image captured by the camera 20a. The product determination unit 115 calculates the position of the person calculated by the position calculation unit 114, the face angle of the person determined by the face angle determination unit 113, that is, the orientation of the person, and the product of the sales floor M2 stored in the storage unit 13 in advance. Based on the layout file F1, the product being viewed by the person is determined.

  Here, FIG. 7 is an explanatory diagram illustrating an example of a frame image used for determining a product viewed by a person. The camera 20a images a predetermined area of the sales floor M2. As a result, the person included in the frame image is in the imaging area of the camera 20a. The face detection unit 112 detects a human face from the frame image, and detects coordinates in the X direction and the Y direction. Thereby, the face detection unit 112 can detect the position of the human face in the imaging region of the camera 20a. The face angle determination unit 113 determines the face angle of the person included in the frame image. Thereby, the face angle determination unit 113 can estimate the face direction of the person, that is, the line of sight of the person. And the goods determination part 115 can determine the goods which the person is browsing by comparing the place calculated from the position of the detected person and a person's eyes | visual_axis, and the product layout of the sales floor M2. .

  Returning to FIG. 6, the browsing history generation unit 116, which is a generation unit, generates browsing history information in which an action history of a person whose face detected by the face detection unit 112 has browsed a product is recorded. The browsing history generation unit 116 generates browsing history information for each product by measuring an action history such as the number of times the product has been browsed and the browsing time of the product.

  When the product determination unit 115 determines a product, the browsing history generation unit 116 determines that the current frame is different from the product determined from the previous frame image in time series and the product determined from the current frame image. Count up the number of product browsing times determined from the image. Even when the product is determined first, the browsing history generation unit 116 counts up the number of times the determined product is viewed.

  On the other hand, when the product determined from the previous frame image in time series and the product determined from the current frame image are the same, the browsing history generation unit 116 browses the product determined from the current frame image. In addition, the time taken to capture the current frame image from the previous frame image is added. And the browsing history production | generation part 116 measures the frequency | count of browsing and browsing time for every goods by determining about all the frame images of a moving image.

  Note that the browsing history generation unit 116 may measure the browsing time by thinning out the frame image from the moving image. That is, the browsing history generation unit 116 may perform measurement for each of a plurality of frame images such as every two frame images or every three frame images instead of every frame image. Thereby, a processing burden can be reduced.

  The feature quantity extraction unit 117, the similarity calculation unit 119, and the similarity determination unit 120 authenticate whether or not the person is registered in the person authentication dictionary D1.

  First, the feature quantity extraction unit 117 extracts the feature quantity of the face person detected by the face detection unit 112. The feature amount extraction unit 117 detects facial feature points such as eyes, nose, and mouth for each frame image from face regions detected over a plurality of frame images by the tracking process. Then, the feature quantity extraction unit 117 extracts a feature quantity for each feature point. The feature amount is a parameterization of appearance features such as the shape of feature points, the color of the surface, and unevenness. The feature points of the face to be detected can be arbitrarily changed by setting or the like. In addition, any method may be used as a method for detecting facial feature points in the feature quantity extraction unit 117, and a known technique is used.

  The feature amount extraction unit 117 extracts a feature amount from the person's clothing detected by the face detection unit 112. The feature amount extraction unit 117 extracts a color as a feature amount from a person's clothes or jewelry. As a result, the clothes of the person are also added to the determination of whether or not the person is registered in the person authentication dictionary D1. Therefore, authentication accuracy can be increased.

  The attribute determination unit 118 determines the attribute of a person included in the image captured by the camera 20a. The attributes include sex and age. The attribute determination unit 118 determines the gender of the person by comparing the feature amount of the feature point extracted by the feature amount extraction unit 117 with the gender face authentication dictionary D2 stored in the storage unit 13. Further, the attribute determination unit 118 determines the age of the person by comparing the feature amount of the feature point extracted by the feature amount extraction unit 117 with the age-specific face authentication dictionary D3 stored in the storage unit 13. The age may be an age or a category indicating age.

  The similarity calculation unit 119 compares the feature amount for each person registered in the person authentication dictionary D1 with the feature amount extracted by the feature amount extraction unit 117, and calculates the similarity for each person. Here, the similarity indicates how much the two feature quantities are similar by comparing the feature quantity of the person captured by the cameras 20a and 20b with the feature quantity of each person registered in the person authentication dictionary D1. Any value (similarity) may be used.

  The similarity determination unit 120 determines whether a person whose similarity is equal to or greater than a threshold is registered. The similarity determination unit 120 determines that the person authentication information of the corresponding person is registered in the person authentication dictionary D1 when the similarity is greater than or equal to a predetermined threshold among the similarities calculated by the similarity calculation unit 119. judge. On the other hand, when the similarity is less than the threshold, the similarity determination unit 120 determines that the person authentication information of the corresponding person is not registered in the person authentication dictionary D1. If there are a plurality of pieces of person authentication information equal to or greater than the threshold, the similarity determination unit 120 determines the person with the highest similarity as the person captured by the cameras 20a and 20b.

  When there are a plurality of pieces of person authentication information having a degree of similarity equal to or greater than the threshold, the similarity degree determination unit 120 determines that the determination is not possible, and from a subsequent process, a moving image having a plurality of pieces of person authentication information having a degree of similarity equal to or greater than the threshold May be excluded. When there are a plurality of pieces of person authentication information having a degree of similarity equal to or greater than the threshold, the moving images captured by the cameras 20a and 20b may be unclear. By excluding unclear moving images with low reliability, the similarity determination unit 120 can ensure reliability. Moreover, the similarity determination part 120 shall be able to set arbitrary values as a predetermined threshold value.

  The registration unit 121, which is a storage control unit, stores the person authentication information related to the moving image captured by the camera 20a by registering it in the person authentication dictionary D1 for each face person detected by the face detection unit 112. The registration unit 121 determines from the moving image whose similarity is determined, the feature amount extracted by the feature amount extraction unit 117, the identification information of the camera 20a that transmitted the moving image, the imaging time when the moving image was captured, and the moving image. The generated browsing history information is stored in the person authentication dictionary D1 as person authentication information.

  The registration performed by the registration unit 121 includes new registration and additional registration. The registration unit 121 newly registers when the similarity determination unit 120 determines that person authentication information of a person equal to or greater than the threshold is not registered in the person authentication dictionary D1. The registration unit 121 registers the attribute information and the person authentication information in the person authentication dictionary D1 using the new person identification information assigned by the face detection unit 112.

  On the other hand, the registration unit 121 performs additional registration when the similarity determination unit 120 determines that person authentication information of a person equal to or greater than the threshold is registered in the person authentication dictionary D1. The registration unit 121 registers the person authentication information related to the current imaging in association with the person authentication information of the corresponding person registered in the person authentication dictionary D1. As described above, the registration unit 121 performs additional registration to increase the person authentication information of the corresponding person identification information. Thereby, since the similarity calculation part 119 can compare with several person authentication information, it can improve the reliability of a determination result.

  The exclusion unit 122 excludes a predetermined person in the person authentication dictionary D1 from the target of the similarity calculation unit 119. Thereby, the similarity calculation unit 119 does not calculate the similarity of the excluded person. Therefore, the exclusion unit 122 can reduce the processing burden. When the person authentication information corresponding to the person included in the moving image captured by the camera 20b is registered, the excluding unit 122 excludes the person with the corresponding person authentication information from the similarity calculation target. Alternatively, when there is person authentication information for which a predetermined time has elapsed, the exclusion unit 122 excludes the person with the corresponding person authentication information from the similarity calculation target.

  First, a case where person authentication information corresponding to a person included in a moving image captured by the camera 20b is registered in the person authentication dictionary D1 will be described. The camera 20a does not capture a person who is not in the sales floor M2. Therefore, the exclusion unit 122 can reduce the processing load on the server device 10 by excluding persons who leave the sales floor M2. Here, the camera 20b is directed in a direction to image the face of a person leaving the sales floor M2. Therefore, the camera 20b can capture an image of a person leaving the sales floor M2. Then, the excluding unit 122 detects the face of the person included in the captured moving image, extracts the feature amount, calculates the similarity, and determines the retired person who has been authenticated by determining the similarity. Exclude from degree calculation target.

  Next, a case where there is person authentication information for which a predetermined time has elapsed will be described. The excluding unit 122 excludes the person whose personal authentication information has passed a predetermined time from the similarity calculation target. There is a limit to the time for a person to travel around the sales floor M2. Therefore, since it is considered that the exclusion unit 122 is not in the sales floor M2, the person of the personal authentication information after a predetermined time has been excluded from the similarity calculation target. Thereby, even when the camera 20b cannot capture the face of the person who leaves the sales floor M2, the excluding unit 122 excludes the person with the corresponding person authentication information from the similarity calculation target. be able to. Alternatively, even if the image can be captured but cannot be determined to be a person included in the person authentication dictionary D1, the excluding unit 122 excludes the person with the corresponding person authentication information from the similarity calculation target. Can do. The starting point for the predetermined time is not limited. An example of the starting point of the predetermined time is when the person authentication information is first registered in the person authentication dictionary D1, or when the person authentication information is registered last in the person authentication dictionary D1.

  Next, a person authentication process executed by the CPU 11 of the server device 10 according to the above-described embodiment according to the control program will be described. The server apparatus 10 according to the embodiment executes person authentication processing using a moving image captured by the camera 20a.

  FIG. 8 is a flowchart showing the flow of person authentication processing executed by the CPU 11 of the server device 10 according to the control program.

  First, the CPU 11 (reception unit 111) of the server device 10 receives a moving image captured by the camera 20a (step S11).

  Next, the CPU 11 (face detection unit 112) of the server device 10 determines whether or not a face can be detected from the received moving image (step S12). When a face cannot be detected (step S12; No), the CPU 11 (reception unit 111) of the server device 10 returns to step S11 and receives a moving image.

  On the other hand, when the face can be detected (step S12; Yes), the CPU 11 (face detection unit 112) of the server device 10 assigns new person identification information to the detected face (step S13).

  Next, the CPU 11 (reception unit 111) of the server device 10 stores the received moving image (step S14). Next, the CPU 11 (face detection unit 112) of the server device 10 determines whether or not the face detected from the received moving image is out of frame (step S15).

  When the detected face is not out of frame (step S15; No), the CPU 11 of the server device 10 returns to step S14 and stores the received moving image.

  On the other hand, when the detected face is out of frame (step S15; Yes), the CPU 11 of the server device 10 executes processing for generating browsing history information described later (step S16).

  Next, the CPU 11 (feature amount extraction unit 117, similarity calculation unit 119, similarity determination unit 120) of the server device 10 determines whether or not the detected face person is the same person registered in the person authentication dictionary D1. The authentication process is executed (step S17). The CPU 11 of the server device 10 extracts the detected face feature amount. The CPU 11 of the server device 10 compares the extracted feature quantity with the feature quantity of the person authentication information registered in the person authentication dictionary D1, and calculates the similarity. Then, the CPU 11 of the server device 10 authenticates whether or not the person is registered in the person authentication dictionary D1 based on the calculated similarity.

  Next, the CPU 11 of the server device 10 determines whether or not the same person as the detected face person has been registered in the person authentication dictionary D1 by person authentication (step S18). When it is authenticated that the same person as the detected face person is registered in the person authentication dictionary D1 (step S18; Yes), the CPU 11 (registration unit 121) of the server device 10 corresponds to the person authentication dictionary D1. The person authentication information is additionally registered in the person identification information to be executed (step S19).

  On the other hand, when it is not authenticated that the same person as the detected face person is registered in the person authentication dictionary D1 (step S18; No), the CPU 11 (attribute determination unit 118) of the server device 10 receives the camera 20a. The attribute of the person included in the moving image captured by is determined and attribute information is generated (step S20). The CPU 11 (attribute determination unit 118) of the server device 10 determines the gender and age of the person.

  Next, the CPU 11 (registration unit 121) of the server device 10 adds the person identification information and newly registers the person authentication information in the person authentication dictionary D1 (step S21).

  As described above, the CPU 11 of the server device 10 ends the person authentication process.

  Next, browsing history information generation processing executed by the CPU 11 of the server device 10 according to the above-described embodiment according to the control program will be described.

  FIG. 9 is a flowchart showing a flow of browsing history information generation processing executed by the CPU 11 of the server device 10 according to the control program.

  First, the CPU 11 (face angle determination unit 113) of the server device 10 determines the detected face angle, that is, the face orientation (step S31). Next, the CPU 11 (position calculation unit 114) of the server device 10 calculates the position where the person included in the moving image captured by the camera 20a is present (step S32). Next, the CPU 11 (product determination unit 115) of the server device 10 determines a product viewed by a person included in the moving image captured by the camera 20a (step S33).

  Next, the CPU 11 (product determination unit 115) of the server device 10 determines whether or not a person is browsing the same product in the previous frame image and the current frame image (step S34).

  When a person is browsing the same product (step S34; Yes), the CPU 11 (browsing history generation unit 116) of the server device 10 determines the time taken to capture the current frame image from the previous frame image. Then, it is added to the browsing time of the product being browsed (step S35).

  On the other hand, when the person is not browsing the same product (step S34; No), the CPU 11 (browsing history generation unit 116) of the server device 10 counts up the number of browsing of the product being browsed (step S36). .

  Next, the CPU 11 of the server device 10 determines whether or not processing has been completed for all frame images of the moving image until the person is out of frame (step S37). When the processing has not been completed for all the frame images of the moving image (step S37; No), the CPU 11 of the server device 10 returns to step S31.

  On the other hand, when the processing is completed for all the frame images of the moving image (step S37; Yes), the CPU 11 of the server device 10 ends the series of browsing history information generation processing.

  As described above, according to the server device 10 of the present embodiment, a moving image captured by the camera 20a that captures a predetermined area of a store is received. The server device 10 detects a person's face from the received moving image and determines the direction in which the person's face is facing. In addition, the server device 10 calculates coordinates indicating a position where the face is present in the moving image, and calculates a position where the person is present. Then, the server device 10 determines the product that the person is browsing from the direction in which the determined person's face is facing, the position where the person is, and the product layout of the store, and generates browsing history information Remember me. Therefore, it is possible to record a product that the customer is interested in and browses.

  Although several embodiments of the present invention have been described, these embodiments are presented by way of example and are not intended to limit the scope of the invention. These novel embodiments can be implemented in various other forms, and various omissions, replacements, and changes can be made without departing from the scope of the invention. These embodiments and modifications thereof are included in the scope and gist of the invention, and are included in the invention described in the claims and the equivalents thereof.

  In the above-described embodiment, the server device 10 accepts the input of a moving image when the receiving unit 111 receives the moving images sequentially captured by the cameras 20a and 20b. However, the moving image input method is not limited to this. The moving image may be a moving image file captured by the cameras 20a and 20b in the past.

  In the above-described embodiment, the product determination unit 115 of the server device 10 determines the person from the position of the person calculated by the position calculation unit 114, the orientation of the person determined by the face angle determination unit 113, and the product layout file F1. Is determining the item being viewed. However, the determination of the product that the person is browsing is not limited to this. For example, a product (product name) may be recognized from a frame image of a moving image captured by the camera 20a using a known object recognition technique.

  The program executed by each device of the above-described embodiment or modification is provided by being incorporated in advance in a storage medium (ROM or storage unit) included in each device, but is not limited thereto. For example, an installable or executable file is recorded on a computer-readable recording medium such as a CD-ROM, a flexible disk (FD), a CD-R, or a DVD (Digital Versatile Disk). You may comprise. Furthermore, the storage medium is not limited to a medium independent of a computer or an embedded system, but also includes a storage medium in which a program transmitted via a LAN, the Internet, or the like is downloaded and stored or temporarily stored.

  Further, the program executed by each device of the above-described embodiment or modification may be stored on a computer connected to a network such as the Internet and provided by being downloaded via the network, or the Internet It may be configured to be provided or distributed via a network.

DESCRIPTION OF SYMBOLS 1 Person authentication system 10 Server apparatus 11 CPU
111 reception unit 112 face detection unit 113 face angle determination unit 114 position calculation unit 115 product determination unit 116 browsing history generation unit 117 feature amount extraction unit 118 attribute determination unit 119 similarity calculation unit 120 similarity determination unit 121 registration unit 122 exclusion unit 13 storage unit D1 person authentication dictionary D2 gender face authentication dictionary D3 age face authentication dictionary F1 product layout file 20a, 20b camera 21 CPU
211 Imaging control unit 212 Transmission unit

JP2013-65119A

Claims (6)

An input unit that receives an input of an image captured by an imaging unit that captures a predetermined area of the store;
Detecting means for detecting a human face from the image received by the input means;
Determination means for determining a product viewed by the person of the face detected by the detection means;
Generating means for generating browsing history information in which an action history in which the person of the face detected by the detecting means has browsed a product is recorded;
Storage control means for storing the browsing history information generated by the generating means for each person of the face detected by the detecting means;
An information processing apparatus comprising: When the product determined by the determination unit from the previous image in time series and the product determined by the determination unit are different, the generation unit increases the number of times the product is viewed.
The information processing apparatus according to claim 1. When the product determined by the determination unit from the previous image in time series is the same as the product determined by the determination unit, the generation unit adds the browsing time of the product.
The information processing apparatus according to claim 1 or 2. The storage control means further stores the gender of the person determined by comparing the feature amount of the person extracted from the image with the feature amount of each gender stored in advance.
The information processing apparatus according to any one of claims 1 to 3. The storage control means further stores the age of the person determined by comparing the feature amount of the person extracted from the image with a feature amount for each age stored in advance.
The information processing apparatus according to any one of claims 1 to 4. Computer
An input unit that receives an input of an image captured by an imaging unit that captures a predetermined area of the store;
Detecting means for detecting a human face from the image received by the input means;
Determination means for determining a product viewed by the person of the face detected by the detection means;
Generating means for generating browsing history information in which an action history in which the person of the face detected by the detecting means has browsed a product is recorded;
Storage control means for storing the browsing history information generated by the generating means for each person of the face detected by the detecting means;
Program to function as.

Images