JP2022075904A - Object detection method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an appropriate service in accordance with a user.

SOLUTION: An object detection method for acquiring image data from a system including an imaging apparatus and an information processing apparatus comprises the steps of: detecting at least one of belongings, clothes, accessories, and a vehicle related to a person from the acquired image data; recognizing, by the system, a shape or a color of the object on the basis of characteristic information of the detected object; recognizing, by the system, the person from the image data; associating, by the system, the recognized object with the recognized person; and identifying, by the system, the person associated with the recognized object.

SELECTED DRAWING: Figure 1

COPYRIGHT: (C)2022,JPO&INPIT

Description

Embodiments of the present invention relate to image processing techniques.

People (users or users) who visit commercial facilities, public facilities (government offices), or facilities such as hospitals are intended to receive services. However, since the content of the requested service is different, it was necessary for the individual to voluntarily carry out the confirmation work for receiving the service and the operation for obtaining the information. For this reason, there are inconveniences such as missing opportunities for users to obtain useful information and loss of sales opportunities for service providers.

To deal with such circumstances, some solutions applying image processing technology have been proposed. For example, there is a technique for grouping a plurality of people included in an image and estimating group attributes. However, the estimated attributes cannot be used to provide services. Also, even if the image contains objects such as cars and possessions, the relationship between the person and them is not considered.

In addition, a method of providing a selection menu according to the composition of a customer group by determining attributes based on gender and height has also been proposed. However, even with this technique, the information used to select menus is still limited, and it is not possible to provide services from a comprehensive perspective.

Japanese Patent No. 4198951 Japanese Unexamined Patent Publication No. 2008-262435 Japanese Unexamined Patent Publication No. 2011-141610

As mentioned above, there is probably room for improvement in providing appropriate services for each user. There is a demand for a technique that enables more appropriate services to be provided in consideration of not only people but also accompanying objects.
The purpose is to provide a target detection method capable of providing an appropriate service according to a user.

According to the embodiment, the target detection method is a target detection method that acquires image data from a system including an image pickup device and an information processing device. The target detection method is based on the process of detecting at least one of the possessions, clothes, wears and vehicles related to the person from the acquired image data, and the system based on the characteristic information of the detected target. The process of recognizing the shape or color of an object, the process of the system recognizing a person from image data, the process of associating the recognized object with the recognized person, and the process of the system recognizing the object. It includes a process of personally identifying a person associated with.

FIG. 1 is a diagram showing an example of a service providing system according to the first embodiment. FIG. 2 is a diagram showing an example of an object included in the image data. FIG. 3 is a diagram showing an example of the structure of information managed by the feature information management unit 500. FIG. 4 is a diagram showing an example of the information flow of the service in the above configuration. FIG. 5 is a diagram showing an example of a service providing system according to the second embodiment. FIG. 6 is a diagram showing an example of the service providing system according to the third embodiment. FIG. 7 is a diagram showing an example of a table for changing the service provision menu according to the determination result.

[First Embodiment]
FIG. 1 is a diagram showing an example of a service providing system according to the first embodiment. The service providing system shown in FIG. 1 includes a computer 100 and a face-to-face device 200 connected to the computer 100 via a communication line 800 such as a LAN (Local Area Network).
The face-to-face device 200 is a device that faces a user, such as an automated teller machine, a card payment terminal, a hospital accounting terminal, or a terminal for procedures at a government office. In this embodiment, consider a device installed in a gas station or a charging station. Other facilities and stores are often formed around this type of equipment. The computer 100 can communicate with the peripheral facilities / stores 700 via the communication line 800.

The face-to-face device 200 includes a camera 300 and a display 400. The camera 300 is installed so as to capture, for example, an object facing the display 400 in the field of view, and outputs image data obtained by photographing the field of view. The cameras 300 are installed at at least one or a plurality of points around the facing device 200. The image data includes the person 900 standing in front of the face-to-face device 200, another person with the person 900, the property owned by the person 900, the vehicle, the person in the driver's seat of the vehicle, and the passenger in the passenger seat. Can include people who are.

The camera 300 is, for example, an ITV camera or an IP camera, and obtains image data by analog / digital conversion of optical information obtained through a lens. The camera 300 can also capture images that are continuous in time. FIG. 2 is a diagram showing an example of an object included in the image data.

The display 400 is a display medium to which digital signage technology is applied, and displays operation information, advertisement information, and the like.
The computer 100 includes an input unit 1, a detection unit 2, a recognition unit 3, an operation unit 4, a control unit 5, and an output unit 6 as processing functions according to the embodiment.
The input unit 1 acquires image data output from the camera 300. The detection unit 2 performs image processing on the acquired image data, individually detects and identifies an object in the field of view of the camera 300. For example, a person 900, a vehicle, or the property of the person 900 is detected. In addition, at least one of the detected person 900's age, gender, height, race, wear, line of sight, face orientation, vehicle license plate, vehicle type, and vehicle color combination is targeted. Detected.

The recognition unit 3 individually recognizes the detected target by comparing it with the feature information stored in the feature information management unit 500. As a result of the recognition, characteristic information such as the target attribute, type, and individual identification information (ID) can be obtained. The feature information management unit 500 is a database that stores feature information related to an object that can be photographed.

The operation unit 4 is a user interface such as a touch panel or a numeric keypad, which is arranged within the reach of a user, for example, of the face-to-face device 200. The user can input operation information into the system via the operation unit 4.

The service information management unit 600 is a database that stores service information to be provided to the user, information related to the target identified by the recognition unit 3, and the like. The control unit 5 reads out the service information according to the recognition result obtained by the recognition unit 3 from the plurality of service information stored in the service information management unit 600, and passes it to the output unit 6. It is also possible to select service information based on the operation information input from the operation unit 4.

The output unit 6 displays the read service information on the display 400 or outputs it as voice information from a speaker (not shown). The output unit 6 also has a function of notifying the peripheral facilities / stores 700 of the service information and outputting a message signal for requesting the provision of the service. Further, the output unit 6 also has a function of notifying the mobile terminal of a person registered in advance of the service information. Next, the above configuration will be described in more detail.

The input unit 1 acquires image data of a person, an object, a vehicle, or the like who is about to use the face-to-face device 200 or has approached the vicinity. In addition to acquiring image data directly from the camera 300, it is also possible to capture an image file or a moving image file taken by a digital still camera via a medium.

This type of data can be obtained from the camera 300 as well as from a mobile device owned by a person. For example, the input unit 1 can acquire image data by a communication means such as a wireless LAN provided in a mobile phone, infrared communication, Bluetooth (registered trademark), RFID (wireless ID), and the like. Of course, it is also possible to send and receive image data by inserting and removing a recording medium such as a contact-type information exchange means and a flash memory card.

Furthermore, image data can be exchanged by exchanging information with other possessions and equipment mounted on the vehicle. For example, information for vehicle identification can be obtained from a vehicle by using the information on the ETC card, and information on the remaining capacity of fuel and battery, travel history information managed by a navigation system, etc. can be transmitted by wireless communication. Is also feasible. Similarly, these can be realized by wireless information means, dedicated media, contact type information exchange means, and the like. In short, the input unit 1 only needs to be able to acquire the digital image data to be processed by some method.

The detection unit 2 detects a person, an object, a vehicle, or the like included in the image data. For example, in order to detect a person, it is preferable to detect the face area of the person. In order to detect the face region, the detection unit 2 uses the luminance information on the image to obtain the coordinates indicating the face region. For example, Reference 1 (Yushi Mita et al .: "Co-occurrence-based Joint Haar-like Features Suitable for Face Detection" Journal of the Institute of Electronics, Information and Communication Engineers (D), vol.J89-D, 8, pp1791-1801 (2006)) It is possible to use the method described in.

The information indicating the detected result may have any shape depending on the orientation and size of the face. In this embodiment, for the sake of simplicity, the face area is indicated by rectangular information, and the coordinates of the corners of the rectangle are used as the detection result. In addition, a method in which the position giving the highest correlation value is set as the face area by obtaining the correlation value while moving the template prepared in advance in the image, a face extraction method using the eigenspace method or the subspace method, etc. Is also available.

Further, a technique for detecting not only the face but the entire human area may be used. Reference 2 (Watanabe et al., "Co-occurrence Histograms of Oriented Gradients for Pedestrian Detection, In Proceedings of the 3rd Pacific-Rim Symposium on Image and Video Technology" (PSIVT2009), pp. 37-47. , Hereinafter referred to as "Co-HOG features").

If the area of the face or the person is detected, it is possible to detect the type of clothes and the possessions worn by the person based on the relative positional relationship from the head or the face. Using the above-mentioned Joint Haar-like feature and Co-HOG feature, it is possible to detect these items by collecting a large number of images of the detection target in advance and performing statistical learning. Similarly, it is possible to detect all or part of the vehicle in the image.

It is assumed that the same object is continuously reflected over a plurality of frames in time. Therefore, it is desirable to perform tracking processing in order to manage the same type and number of target objects that exist after a predetermined time has passed. This can be realized by, for example, a method of associating the position of the face detected by using optical flow in the next frame, or a known technique (for example, described in Japanese Patent Application Laid-Open No. 2011-170711).

Further, when information other than the image data is input to the input unit 1, it is possible to acquire the existence of the communication target by the protocol formulated in advance. This makes it possible to detect the target in the same way as using image data.

The feature information management unit 500 is a database for managing the information of a person, an object, and a vehicle obtained from the input unit 1 in association with each other. That is, the feature information management unit 500 is a database that manages feature information required when the recognition unit 3 identifies an individual person of a target object and determines the type of vehicle.

FIG. 3 is a diagram showing an example of the structure of information managed by the feature information management unit 500. As shown in FIG. 3, the feature information management unit 500 manages the facial feature information output from the detection unit 2 in association with each individual. Information about the person concerned such as face image, name, gender, age, and height, and related information such as the person's property, clothes to wear, and information on the vehicle to be boarded are managed in association with each face feature information. It is also possible to do.

The content actually managed as the face feature information may be the data itself output by the face feature extraction means, or may be an m × n feature vector or a correlation matrix immediately before performing subspace or KL expansion.
In addition to the above-mentioned feature information, statistical information such as attribute information such as gender and age can also be managed by the feature information management unit 500. For gender discrimination, characteristic information for men and characteristic information for women are stored and used for identification in the attribute discrimination process. For other attributes, manage the feature information used to classify the categories.

In order to newly register the face and related information in the feature information management unit 500, the system administrator may register after obtaining the consent of the user. Alternatively, as a result of recognition by the recognition unit 3, if the similarity of the characteristic information of any person does not exceed a predetermined threshold value, that is, if the corresponding person does not exist, the person is additionally registered as a new person. May be good.

The recognition unit 3 generates information including detailed types, individual identifications, and states from information such as people, objects, and vehicles detected by the detection unit 2.
As a method of recognizing a person, a method of recognizing a person using a face will be described below. As a method of extracting facial feature information indicating an individual's feature based on the detected facial region, for example, as shown in (Japanese Patent Laid-Open No. 2007-4767), a model is used for one person image information. It is possible to acquire face feature information by creating an image in which the orientation and state of the face are intentionally changed by using. It is also possible to perform more accurate recognition processing by using a plurality of consecutive images for the same person and performing a calculation using a moving image.

For example, it is disclosed in Reference 3 (Kazuhiro Fukui, Osamu Yamaguchi, Kenichi Maeda: "Face Recognition System Using Moving Images", IEICE Technical Report PRMU, vol97, No.113, pp17-24 (1997)). , A method using the mutual subspace method is known. In this method, an image of m × n pixels is cut out from continuously obtained image data. Then, the correlation matrix of the feature vector is obtained from these data, and the orthonormal vector obtained by KL expansion is obtained to calculate the subspace showing the facial features obtained from the continuous images. In addition, Reference 4 (Kazuhiro Fukui, Osamu Yamaguchi: "Facial feature point extraction by combining shape extraction and pattern collation" Journal of the Institute of Electronics, Information and Communication Engineers (D), vol.J80-D-II, No.8, pp2170 -The technique described in 2177 (1997)) is also known.

The subspace can be calculated by finding the correlation matrix (or covariance matrix) of the feature vector and finding the orthonormal vector (eigenvector) by its KL expansion. In the subspace, k eigenvectors corresponding to the eigenvalues are selected in descending order of the eigenvalues and expressed using the eigenvector set.

In this embodiment, the correlation matrix Cd is obtained from the feature vector and diagonalized to the correlation matrix Cd = ΦdΛdΦdT to obtain the eigenvector matrix Φ. This information becomes a subspace that shows the facial features of the person to be recognized. The feature information such as the subspace output by the above method is used as the personal facial feature information for the face detected from the input image data.

A person can be identified by comparing the face feature information calculated in this way with the face feature information managed by the feature information management unit 500. As a result of this search process, the person ID managed to identify the individual by the feature information management unit 500 and the index showing the similarity as the calculation result are returned in order from the one with the highest similarity. Further, the information managed by the feature information management unit 500 may be returned together. In addition, since it is basically possible to associate by the identification ID, it is not always necessary to send and receive attached information in the search process itself.

The similarity index is the degree of similarity between subspaces managed as facial feature information. To obtain the similarity, a calculation method such as the subspace method or the compound similarity method can be used. In these methods, both the recognition data in the registered information stored in advance and the input data are expressed as a subspace calculated from a plurality of images. Then, the "angle" formed by the two subspaces is defined as the degree of similarity.

Similarly, the correlation matrix Cin is obtained for the input data string, diagonalized to Cin = ΦinΛinΦinT, and the eigenvector Φin is obtained. The degree of similarity between subspaces (0.0 to 1.0) of the subspace represented by two Φin and Φd is obtained, and the degree of similarity is used for recognizing this. Again, it is possible to use the method disclosed in Reference 3.

It is also possible to improve the accuracy by projecting a plurality of person images that can be identified as the same person in advance into a subspace and identifying whether or not the person is the person. It is clear that distances on the feature space and Mahalanobis distances can also be used as similarity indicators other than the degree of similarity. When the distance is used, the larger the numerical value is, the lower the similarity is. Therefore, the smaller the value is, the higher the similarity with the registered person is. That is, the recognition unit 3 identifies an individual of the person detected by the detection unit 2, and refers to the feature information stored in the feature information management unit 500 based on the identified individual, thereby improving the recognition accuracy. Let me.

In order to identify an individual person, it is necessary to register the face information of the specific target in advance. However, if only attribute information such as age and gender is determined, it is possible to obtain information on a person without prior registration. For example, by using the technique disclosed in (Japanese Unexamined Patent Publication No. 2012-3623), it is possible to estimate the age and gender of a person. By utilizing this technique and preparing images according to the category to be discriminated in advance, it is possible to discriminate between race and facial wear such as hats, glasses and masks.

It is also possible to obtain the height from the relative ratio to the entire image by using the above-mentioned person detection, face detection result, and angle of view information of the camera 300. In addition, by using the technique described in (Japanese Patent Laid-Open No. 2003-141551), it is possible to obtain facial movement information such as the line of sight and the direction of the face.

In addition, from the relative position from the top position of the person detection coordinates and the relative position from the face detection position, wears such as clothes and hats, possessions in hand, and possessions such as bags and watches are previously checked. It is also possible to identify it by distinguishing it from the dictionary learned in.

Further, the recognition unit 3 determines at least one of the license plate, the vehicle type, and the color combination of the vehicle on which the person is riding. It is also possible to identify a manufacturer or brand by collecting images for each manufacturer or brand and creating a dictionary for identification of the property or vehicle type.

In addition, for vehicles, it is possible to discriminate the vehicle type by a method using CoHOG features by collecting and learning the images of the vehicle type to be detected corresponding to the variation of the expected orientation. It is possible to discriminate the color information of the vehicle by using the color information, and it is also possible to identify the individual vehicle by using the technique described in Japanese Patent No. 4901676.

Further, it is possible to improve the discrimination accuracy by combining a plurality of the discriminated information. For example, when a vehicle is detected, the vehicle can be individually identified by the license plate information of the vehicle, and the individual person who is on board can be identified only to the person associated with the vehicle. If the combination is not used, it is necessary to identify the candidate by matching with all the faces of the person registered in the feature information management unit 500, but the feature information management unit 500 associates the features with the facial features of the individual. By associating and managing the vehicle type and license plate of the car owned by the company, the person is erroneously identified because the person is collated by focusing on the person associated with the detected license plate car. It is possible to prevent it from happening.

Even if you do not know the license plate of the car, you can manage the information of the color and model of the car, or the type of clothes and belongings in association with the person who visited, and manage the combination statistically. Is preferable. In this way, even if multiple candidates show similar indicators based on facial features alone, the accuracy of identifying a person can be improved by adding information from possessions other than the face, clothing, and vehicles. Is possible. In this case, it can be realized by a method of adding to the similarity by multiplying the similarity by a predetermined coefficient when other possessions or vehicle information exists in addition to the similarity due to the facial features of the person.

In addition, since a person wears and possesses the property, it is easy to associate it with the person. On the other hand, in order to determine that a person and a person are a group or to associate an owner with a vehicle, for example, there is a method of associating those detected simultaneously on the same screen for a predetermined time or longer. Alternatively, it is determined whether or not the number of simultaneous detections in the past history is a predetermined number or more, or whether the time during which the distance between a person and a person or a person and a vehicle is within a predetermined time is a predetermined time or more. You may try to discriminate. In addition, as disclosed in Japanese Patent Application Laid-Open No. 2007-249953, a method of grouping using the positional relationship and time passed at a plurality of points can also be realized.

Similarly, while the user is away from the face-to-face device 200, it detects that the user's property, another person from the user has approached the vehicle, or that the property or vehicle has been taken away. It is possible. For example, when a person is detected while an object or a vehicle is present on the screen, it can be determined by face matching whether or not the person is the owner of the target object. This process may be performed when the control unit 5 determines that it is necessary to monitor the target object.

Whether or not the detected person is the owner of the vehicle may be determined by using the information managed by the feature information management unit 500, or is present together immediately before the control unit 5 starts monitoring. The person may be registered as the owner. If it is determined that the owner is not the owner, it can be determined that a suspicious person may be approaching. Further, when it is detected that the target object has moved out of the screen, it can be determined that the target object has been taken away.

By the above processing, from the image data obtained from the input unit 1, it becomes possible to obtain a method for determining a combination of the user of the face-to-face device 200, an accessory, a nearby object, or a vehicle, and detailed information about each. ..

The operation unit 4 is a means for inputting the result of the user's operation (operation) into the system. For example, by attaching a touch panel sensor to the display 600, it is possible to input operation information according to the screen. In addition, a keyboard or mouse may be installed, or a dedicated button may be installed near the display 600. The operation unit 4 can be used for selecting a menu, inputting characters for information retrieval, a pointer operation for indicating a location, and the like.

A mobile terminal such as a mobile phone or a smartphone owned by the user may be wirelessly connected to the operation unit 4 so that information can be input using the mobile terminal. Input may be performed using non-contact communication such as Bluetooth (registered trademark) or wireless LAN, or operation information may be transmitted using a cable connection. As described above, it is possible to input information indicating the user's intention from the operation unit 4.

The service information management unit 600 is a database that stores information to be provided to the user, information related to the target identified by the recognition unit 3, and the like. That is, the service information management unit 600 is a database for collecting and providing service information for providing to users. Further, it is also possible to record and manage the information obtained while the user is using the service as history information.

For example, the service information management unit 600 stores purchase history information, usage date / time, usage location, usage time, and the like related to the service. These pieces of information can also be displayed as reference information in the output unit 6. These can be realized by recording the information displayed to the user from the display 400 and the information input from the operation unit 4 in combination.

Further, by confirming which terminal of the terminals installed in each place is used in addition to the face-to-face device 200, the living area of the user can be recognized. Therefore, when the service cannot be received at a nearby facility (out of stock, etc.), the information of another store can be provided with priority within the range of the user's living area.

In addition, if the information of the mobile phone that is the property is input from the operation unit 4 at the time of use, the profile such as the user's e-mail address, name or date of birth can be acquired and recorded, or obtained from the vehicle. It is also possible to record daily driving history as information and use it for providing service information.

In addition, you can connect to the Internet to share information with remote terminals, information on sales information, inventory information, preferential information services, etc. provided by facilities in the vicinity that have been affiliated in advance, congestion of parking lots and roads, and accidents. Information such as information, traffic closure information, and other traffic regulation information may be managed by the service information management unit 600. The managed information is used as a source of information provided by the control unit 5. The information stored in the service information management unit 600 can be automatically updated by installing a sensor or the like, or may be input and updated by the manager on the facility side.

As a result, it is possible to record the information provided to the user and the information related to the target identified by the recognition unit 3, and manage the service information provided to the user so that it can be collected and provided. It becomes.

The control unit 5 provides a service to the user based on the information of the person who is using the face-to-face device 200, the information of the person's property, the vehicle information, and the information input by the operation unit 4 means. Determine the method of provision and the content of information provision. The information of the service information management unit 600 is read out based on the determined information and provided to the user.

The control unit 5 can refer to, for example, the following information in order to determine the service information to be provided.

-The recognition result obtained by the recognition unit 3 (personal attribute, individual identification, companion, possession, wear, vehicle information, etc.).
-Information (purchase history, etc.) determined by the recognition unit 3 that is associated with an individual and managed by the feature information management unit 500.
・ Service usage details and usage time.
-The content of the information output by the output unit 6.
-Information operated by the operation unit 4.
The service information management unit 600 can also store content information provided regardless of the user. This kind of information includes preferential information, preferential information on surrounding facilities, product sales information (time sales, inventory, price) of peripheral facilities, and congestion status of surrounding roads. In addition, when the user is specified, the past purchase information of the user himself / herself, the history of frequently used services, and the like are also stored in the service information management unit 600.

The output unit 6 is a user interface for transmitting information to the user, and displays an image on a display device (display 400) such as a liquid crystal screen, or transmits information to a person 900 by audio output from a speaker. Further, necessary equipment may be incorporated according to the content of the service provided by the face-to-face device 200. For example, in the case of a device that refuels or charges a vehicle, by incorporating the refueling device or charging device and the display 400 and the speaker into the same housing, the user confirms the information provision content while using the service, and also. It is also possible to execute an operation by the operation unit 4.

The display 400 has a screen that displays the sales status of surrounding commercial facilities, a menu screen that allows procedures such as government offices, and a screen that can be used for hospital examinations and medical examinations, depending on the installation location of the face-to-face device 200. It may be displayed. Further, the face-to-face device 200 may be equipped with at least one device capable of depositing and withdrawing cash, and a plurality of combinations. By incorporating a plurality of devices corresponding to the services into the face-to-face device 200, the user can use the plurality of services at the same time.

Further, the output unit 6 may be provided with a function for connecting to the Internet in order to cooperate with an external terminal or an external facility. For example, by using a communication means such as a mobile phone line or a wireless LAN, it is possible to provide the same service even when the terminal is composed of a movable terminal. Further, as a means for contacting the user when he / she leaves the face-to-face device 200, a communication program for executing e-mail communication or socket communication may be built in. As a result, it is possible to secure a means of contacting a user who is away from the face-to-face device 200.

FIG. 4 is a diagram showing an example of the information flow of the service in the above configuration. For the person A, person B, and car C extracted from the image data, information such as age, gender, combination of people A and B, vehicle type information of car C, and color of car C can be obtained. This information is processed statistically and the statistical information is calculated. Then, with respect to the information related to these objects, the information judged to be statistically optimal is preferentially output as recommended information.

For example, if the vehicle type of the vehicle recognized by the recognition unit 3 is known, the estimated value of the vehicle price can be known, so that information on whether or not the person is a person who makes a high-priced purchase can be known. In addition, it will be possible to provide recommended products according to the vehicle type and preferential information on those products.

In addition, if the number of passengers is one, it can be judged that there is a high possibility that the appearance will occur only by using the target service such as refueling or charging. If the number of passengers is two or more, other passenger members may go to the surrounding facilities while using the service such as charging. Therefore, it is meaningful for the output unit 6 to notify the preferential treatment information, inventory status, event information, etc. at the surrounding facilities. In addition, if the travel history information can be obtained from the vehicle C, it is possible to provide information such as congestion on the surrounding roads to be traveled after that, and information on stores around the travel route.

In addition to services such as refueling and charging of vehicles, cash automatic teller machines, card payment terminals, procedures at government offices, etc. can be used for similar judgments based on information about users, companions, and personal belongings. It is possible to get. For example, if an adult and a child who have luxury goods are detected, we will introduce information on toys for children as it is highly likely to shop, and provide it in consideration of the combination of people and belongings. Information may be selectable.

For example, as shown in FIG. 7, by preparing a table associating services to be provided according to the combination of the detected person's age and gender, personal belongings, car information, and the like. The service provision menu can be changed according to the determination result of the combination. In addition, if the fields marked with ○ × in the table in Fig. 7 can be further subdivided by the number of people, it is possible to change the menu depending on whether the same "adult woman" is one or more. can. It is clear that this combination may be any combination as long as the information can be obtained by the processing already described. Further, in addition to the method using a table, any method may be used as long as a method capable of uniquely managing the output for the combination of detected targets is used.

In addition, if the individual identification of a person or the individual identification by the license plate information of the vehicle is successful, it becomes possible to confirm the past usage status of the service. By doing so, it is not necessary to search for frequently used menus from general menus when users try to use the service, and it is possible to preferentially display frequently used menus in the past history. it is obvious. As described above, according to the embodiment, it is possible to provide an appropriate service according to the combination of the user's person, property, and vehicle.

In addition, if an anniversary such as a birthday is registered in advance, a congratulatory message will be sent, purchase information for products that are likely to be purchased on the anniversary will be provided, and discount information and special treatment information will be displayed. May be good.

In addition, there is a method of changing the displayed contents depending on the time during which the service is being used.

For example, at the start of using a service, preferential information regarding the use of the service is displayed, frequently used service menus are displayed preferentially based on past purchase history information, and news about belongings and applicable vehicles. Etc. should be displayed. This makes it possible to provide useful information to a person who is about to use the service. It is desirable to output the above-mentioned birthday message etc. at the start of using the service.

If the service is in use, the content that the user can do will change depending on the remaining time of the service, so the content will be switched and displayed. For example, if the remaining time is long, the information about the service that can be used away from the place is provided, and if the remaining time of the service is short, the information about the payment information is displayed. In addition, when the user is away from the place, it is possible to notify that the service will be terminated soon by using e-mail or the like when the remaining time is running low.

As for the display after the service ends, information for contacting the parcel mailing service may be provided according to the size of the car and the purchase information. In addition, it is possible to issue special treatment information and coupons that can be used from the next time onward at this timing. As described above, it is possible to change the content of the provided service according to the remaining time of the user's service usage time.

It is also possible for the user to visit the surrounding facilities and receive another service while using the service. Further, it is possible to request a service agency from the menu displayed on the display 400 of the face-to-face device 200.

For example, the inventory status and sales information (price information) of stores around the gas station can be displayed on the display 400, and the user can be presented with a menu to select a purchase list. At that time, the recognized user profile can be reflected in the menu. The user confirms the shopping by paying with cash or a card by the payment function provided in the face-to-face device 200. The purchased goods may be delivered from the store side to the location of the face-to-face device 200, or may be sent to the user's home by a delivery service.

In particular, according to the embodiment, it is possible to estimate the living area for each user from the travel history information of the vehicle and the history of the place where the face-to-face device 200 is used. It is also possible to provide shopping information from information on facilities other than neighboring facilities based on the estimated living area. For example, if the desired service cannot be received at a nearby facility such as when the product is out of stock, the user may be informed that the service can be received at another facility within the user's range of activity. As described above, according to the embodiment, it is possible for the user to request the peripheral facilities to provide the service. In addition, the face-to-face device 200 may be able to request a proxy for procedures to be carried out at a public institution or a government office.

In addition, if the user so desires, it may be possible to leave the property or vehicle while using the service and leave the place until the end of the service. In that case, you can manage your belongings by putting them in a safe, but you can manage your vehicles in a mobile parking lot or monitor suspicious persons and their removal using image data from the input unit 1. You may try to do it. As already described, it is possible to recognize that the removal has occurred by the processing of the recognition unit 3. Based on the result of recognition, leaving the place that a person who is not a pre-registered person has approached the deposited property or vehicle, or that the deposited property or vehicle has been taken away. It is possible to notify existing users. As described above, according to the embodiment, it is possible to realize a monitoring / storage service for the user's property or vehicle.

As described above, according to the embodiment, it becomes possible to recognize at least one object of the person, the property of the person, or the vehicle information on which the person is riding in the image taken by the camera 300. .. Therefore, it is possible to provide appropriate information according to the combination of targets for which the service is used.

That is, from the image data acquired by the camera 300, at least one of the target person attribute included in the image data, the accompanying person and person combination, the person and possession, and the person and boarding vehicle combination is selected by the computer 100. recognize. In addition, the recognition accuracy is improved by identifying an individual from the recognized information and combining it with surrounding information. In addition, it has a function to obtain information by communication means, and obtains the running history of a mobile phone or a vehicle owned by a person.

The face-to-face device 200 includes a display 400 that outputs video, a speaker that outputs audio, and an operation unit 4. The operation unit 4 can perform information communication with a terminal device owned by the user, and can display and input on the user's terminal device (mobile phone or the like). Further, the face-to-face device 200 may be further provided with at least one of a refueling function, a charging function, an automatic cash payment, a payment function such as a credit card, and the like.

Further, the computer 100 displays information common to the same person on the display 400 based on the attribute information for each specified person. The display 400 can switch between a recommendation screen for displaying recommended information and a screen for selecting and displaying an arbitrary menu on the user side.

The computer 100 provides recommendation information (special treatment information (coupon ticket, etc.)) according to the result of authenticating the target. If the user is a repeater, the past history will be referred to. Further, the computer 100 estimates income information from the information of the possession or the car to be used, and changes the recommended product. Further, the computer 100 acquires the traveling history of the vehicle and provides information such as congestion information of surrounding roads and profitable information on the way back based on the contents.

Further, the computer 100 switches the content output from the output unit 6 according to the service usage time of the operation terminal 200. That is, the computer 100 provides necessary information around the user while using the service (charging / refueling), or notifies the peripheral facility / store 700 of the message information in order to act for the use of another service. This enables the user to select a desired product or service from the surrounding facilities during the waiting time. In other words, the user can have the system use the service on behalf of the user and receive another service in the meantime. Furthermore, when the service is used, the user can entrust the property and the vehicle to the system and manage them, monitor the suspicious person, detect the suspicious person, and receive the service to call when the service end time is specified.

With existing technology, users have to wait for a service to complete while they are using it. On the other hand, according to the embodiment, it is possible to integrate information such as a person who uses a predetermined service, its property, a companion, and a vehicle to be used, and provide appropriate information and services. Further, according to the embodiment, the user receives necessary information from the surroundings while using the service, causes the system to use another service on behalf of the system, or causes the user to manage the service usage time on behalf of the user. Therefore, the user himself / herself can receive another service.

From these things, it becomes possible to switch the information output form, the information provision method, and the service provision form according to the user of the face-to-face device 200, the approaching person, the object and the surrounding situation, and therefore, It becomes possible to provide a target detection method that can provide an appropriate service according to the user.

[Second Embodiment]
FIG. 5 is a diagram showing an example of a service providing system according to the second embodiment. In FIG. 5, the same parts as those in FIG. 1 are designated by the same reference numerals, and only different parts will be described here. In the second embodiment, it is assumed that a plurality of face-to-face devices 200 and computers 100 are installed over a wide area. The computer 100 is connected by a wide area communication network (Internet, IP-VPN, etc.), and can exchange and use managed information with each other.

In FIG. 5, the feature information management unit 500 and the service information management unit 600 are centrally managed in the data center 1000. A plurality of computers 100 connected to the camera 300 and the face-to-face device 200, respectively, are connected to the data center 1000 via a communication line 1100 laid over a wide area, and perform various data with the feature information management unit 500 and the service information management unit 600. Can be exchanged.

According to the above configuration, information for detecting and recognizing a person, his / her companion, an accessory, and a vehicle is centrally managed in the data center 1000. As a result, in addition to reducing the amount of data to be managed by the computer 100 on the terminal side, it becomes possible to centrally manage information at the level of, for example, a city, prefecture, or country, and it is possible to improve the uniformity of services. You can get the benefits.

It is also possible to understand the data center 1000 as a so-called cloud computing system. In this form, the feature information management unit 500 and the service information management unit 600 are formed in a distributed database on a wide area network.

[Third Embodiment]
FIG. 6 is a diagram showing an example of the service providing system according to the third embodiment. In FIG. 6, the parts common to FIGS. 1 and 5 are designated by the same reference numerals, and only different parts will be described here.

The system shown in FIG. 6 corresponds to a further integrated configuration of FIG. That is, each of the plurality of computers (101, 102, 103) has an input unit 1, an operation unit 4, and an output unit 6 for acquiring image data from the camera 300, and the data center 1000 has a feature information management unit 500. In addition to the service information management unit 600, a detection unit 2, a recognition unit 3, and a control unit 5 are provided. The computers 101, 102, and 103 are provided with a communication unit 7, and the data center 1000 is provided with a communication unit 70 so that they can communicate with each other via a communication line 1100 such as TCP / IP.

With the above configuration, the computers 101, 102, and 103 can be provided with the minimum functions necessary for the site where the service is provided, and the processing related to the detection and recognition of the target can be outsourced to the data center 1000. As a result, the computers 101, 102, 103 and the face-to-face device 200 can be reduced in size, and merits such as cost saving can be obtained.

Further, the detection unit 2 may be provided in the computers 101, 102, 103, and the processing after the detection of the target may be outsourced to the data center 1000. In this way, the extent to which the processing is outsourced to the data center 1000 can be determined according to the system request, the amount of data to be communicated, or the hardware specifications of the device.

Furthermore, it is of course possible to understand the data center 1000 as a cloud computing system. In this case, the detection unit 2, the recognition unit 3, and the control unit 5 can be understood as functional objects distributed and arranged on a plurality of computers.

The present invention is not limited to the above embodiment. For example, in the above embodiment, the application to a gas station has been described, but the present invention can also be applied to parking lots, commercial facilities, public facilities (government offices), hospitals, beauty salons, and the like.

Although some embodiments of the present invention have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These novel embodiments can be implemented in various other embodiments, and various omissions, replacements, and changes can be made without departing from the gist of the invention. These embodiments and modifications thereof are included in the scope and gist of the invention, and are also included in the scope of the invention described in the claims and the equivalent scope thereof.

100, 101, 102 ... Computer, 200 ... Face-to-face device, 300 ... Camera, 400 ... Display, 500 ... Feature information management department, 600 ... Service information management department, 700 ... Peripheral facilities / stores, 800 ... Communication line, 900 ... Person, 1 ... Input unit, 2 ... Detection unit, 3 ... Recognition unit, 4 ... Operation unit, 5 ... Control unit, 6 ... Output unit, 1000 ... Data center, Communication line 1100, 7 ... Communication unit, 70 ... Communication unit ..

According to the embodiment, the target detection method is a target detection method that acquires image data from a system including an image pickup device and an information processing device. The target detection method is based on the process of detecting at least one of the possessions, clothes, wears, and vehicles related to a person from the acquired image data, and the system based on the characteristic information of the detected target. The process of recognizing the shape or color of an object, the process of the system recognizing a person from image data, the process of associating the recognized object with the recognized person, and the process of the system recognizing the object. The system includes a process of personally identifying the person associated with the personal identification person and a process of confirming the past service usage status of the personally identified person .

Claims (5)

It is a target detection method that acquires image data from a system consisting of an image pickup device and an information processing device.
The process of detecting at least one of a person's property, clothing, wear, and vehicle from the acquired image data, and
The process by which the system recognizes the shape or color of the object based on the detected feature information of the object.
The process by which the system recognizes the person from the image data,
A target detection method comprising a process in which the system associates the recognized object with the recognized person, and a process in which the system personally identifies a person associated with the recognized object. The target detection method according to claim 1, wherein the process of the system recognizing the person from the image data includes the system determining the gender of the person. The target detection method according to claim 1, wherein the property of the person is a cane. The target detection method according to claim 1, wherein the property of the person is a bag. The target detection method according to claim 1, wherein an ETC card is mounted on the vehicle, and a correlation between the person and the history of the ETC card is recorded.

Images