JP7237871B2 - Target detection method - Google Patents

Description

Embodiments of the present invention relate to image processing techniques.

People (users or users) who visit facilities such as commercial facilities, public facilities (government offices), and hospitals aim to receive services. However, since the content of the service requested differs from person to person, it has been necessary for the individual to voluntarily carry out the confirmation work for receiving the service and the operation for obtaining the information. As a result, the user misses an opportunity to obtain useful information, and the service provider loses a sales opportunity.

In response to such circumstances, several solutions applying image processing technology have been proposed. For example, there is a technique of grouping a plurality of persons included in an image and estimating group attributes. However, the estimated attributes cannot be used to provide services. Also, even if the image contains objects such as cars and possessions, the relationship between people and them is not considered.

Also proposed is a method of providing a selection menu according to the composition of a customer group by discriminating attributes based on gender and height. However, even with this technology, the information used to select a menu is still limited, and services from a comprehensive point of view cannot be provided.

Japanese Patent No. 4198951 JP 2008-262435 A JP 2011-141610 A

As described above, there is much room for improvement in providing appropriate services for each user. There is a demand for a technology capable of providing more appropriate services by considering not only people but also accompanying objects.
An object of the present invention is to provide an object detection method capable of providing appropriate services according to users.

According to the embodiment, the object detection method is an object detection method that acquires image data from a system that includes an imaging device and an information processing device. The object detection method includes a process of detecting at least one of personal belongings, clothes, clothing, and a vehicle related to a person from acquired image data; a process of recognizing the shape or color of an object; a process of recognizing a person from image data; a process of recognizing a correspondence between the recognized object and the person; a process of providing information for emphasizing and displaying the person as reference information; and a process of emphasizing the provided reference information and providing it to the user of the system.

FIG. 1 is a diagram showing an example of a service providing system according to the first embodiment. FIG. 2 is a diagram showing an example of an object included in image data. FIG. 3 is a diagram showing an example of the structure of information managed by the feature information management unit 500. As shown in FIG. FIG. 4 is a diagram showing an example of the flow of service information in the above configuration. FIG. 5 is a diagram showing an example of a service providing system according to the second embodiment. FIG. 6 is a diagram showing an example of a service providing system according to the third embodiment. FIG. 7 is a diagram showing an example of a table for changing service provision menus according to determination results.

[First Embodiment]
FIG. 1 is a diagram showing an example of a service providing system according to the first embodiment. The service providing system shown in FIG. 1 includes a computer 100 and a facing device 200 connected to the computer 100 via a communication line 800 such as a LAN (Local Area Network).
The face-to-face device 200 is a face-to-face device, such as an automatic teller machine, a card settlement terminal, a hospital accounting terminal, or a government office procedure terminal. This embodiment considers a device installed at a gas station or charging station. Other facilities and stores are often formed around this type of facility. Computer 100 can communicate with peripheral facility/store 700 via communication line 800 .

Face-to-face device 200 includes camera 300 and display 400 . The camera 300 is installed, for example, so as to capture an object facing the display 400 in its field of view, and outputs image data obtained by photographing the field of view. The cameras 300 are installed at at least one or a plurality of points around the facing device 200 . The image data includes a person 900 standing in front of the facing apparatus 200, another person with the person 900, property owned by the person 900, a vehicle, a person sitting in the driver's seat of the vehicle, and a person sitting in the passenger's seat. can include people who are

The camera 300 is, for example, an ITV camera or an IP camera, and obtains image data by analog/digital conversion of optical information obtained through a lens. The camera 300 can also capture images that are continuous in time. FIG. 2 is a diagram showing an example of an object included in image data.

A display 400 is a display medium to which digital signage technology is applied, and displays operation information, advertisement information, and the like.
The computer 100 includes an input unit 1, a detection unit 2, a recognition unit 3, an operation unit 4, a control unit 5, and an output unit 6 as processing functions according to the embodiment.
The input unit 1 acquires image data output from the camera 300 . The detection unit 2 performs image processing on the acquired image data, and individually detects and identifies objects within the field of view of the camera 300 . For example, a person 900, a vehicle, or a property owned by the person 900 is detected. In addition, at least one of the detected person 900's age, sex, height, race, clothing, line of sight, direction of face, license plate of the vehicle in which he/she rides, vehicle type, and color combination of the vehicle is targeted. detected.

The recognition unit 3 individually recognizes the detected target by comparing it with the feature information stored in the feature information management unit 500 . As a result of recognition, feature information such as target attributes, species, and individual identification information (ID) is obtained. The feature information management unit 500 is a database that stores feature information related to subjects that can be photographed.

The operation unit 4 is a user interface such as a touch panel and a numeric keypad, which is arranged within reach of the user of the facing device 200, for example. A user can input operation information to the system via the operation unit 4 .

The service information management unit 600 is a database that stores service information to be provided to users, information related to objects identified by the recognition unit 3, and the like. The control unit 5 reads service information corresponding to the recognition result obtained by the recognition unit 3 from a plurality of pieces of service information stored in the service information management unit 600 and transfers the service information to the output unit 6 . It is also possible to select service information based on operation information input from the operation unit 4 .

The output unit 6 displays the read service information on the display 400 or outputs it as audio information from a speaker (not shown). The output unit 6 also has a function of notifying the peripheral facilities/stores 700 of service information and outputting a message signal for requesting the provision of services. Furthermore, the output unit 6 also has a function of notifying the mobile terminal of a pre-registered person of the service information. Next, the above configuration will be described in more detail.

The input unit 1 acquires image data of a person, an object, a vehicle, or the like, which is about to use the facing device 200 or approaches nearby. Image data can be acquired directly from the camera 300, or image files shot with a digital still camera or moving image files can be acquired via a medium.

This type of data can be obtained from the camera 300 as well as from a portable device owned by the person. For example, the input unit 1 can acquire image data by communication means such as wireless LAN, infrared communication, Bluetooth (registered trademark), and RFID (wireless ID) provided in the mobile phone. Further, it is of course possible to send and receive image data by inserting and removing recording media such as contact information exchange means and flash memory cards.

Furthermore, image data can be sent and received by exchanging information with other belongings and equipment mounted on the vehicle. For example, information for vehicle identification can be obtained from the vehicle by using the information on the ETC card, and information such as fuel and battery remaining capacity information, driving history information managed by the navigation system, etc. can be transmitted by wireless communication. is also feasible. These can also be realized by wireless information means, dedicated media, contact information exchange means, or the like. In short, the input unit 1 only needs to acquire digital image data to be processed by some method.

The detection unit 2 detects persons, objects, vehicles, and the like included in image data. For example, to detect a person, it is preferable to detect the person's face area. In order to detect the face area, the detector 2 obtains the coordinates indicating the face area using luminance information on the image. For example, reference 1 (Yuji Mita et al.: "Joint Haar-like features based on co-occurrence suitable for face detection", IEICE Transactions on Information and Systems (D), vol.J89-D, 8, pp1791-1801 (2006)) It is possible to use the method described in

The information indicating the detection result may be of any shape depending on the orientation and size of the face. In this embodiment, for the sake of simplicity, the face area is represented by rectangular information, and the coordinates of the corners of the rectangle are used as the detection result. In addition, there are methods such as moving a prepared template in the image to determine the correlation value, and determining the position that gives the highest correlation value as the face region, face extraction methods using the eigenspace method and the subspace method, etc. is also available.

Furthermore, a technique for detecting not only the face but also the entire person area may be used. Reference 2 (Watanabe et al., ``Co-occurrence Histograms of Oriented Gradients for Pedestrian Detection, In Proceedings of the 3rd Pacific-Rim Symposium on Image and Video Technology'' (PSIVT2009), pp. 37-47. , hereinafter referred to as “Co-HOG features”).

Once the area of the face or person is detected, it is possible to detect the type of clothing worn by the person and the belongings of the person based on the relative positional relationship from the head and face. These items can be detected by using the aforementioned Joint Haar-like features and Co-HOG features, collecting a large number of images of detection targets in advance and performing statistical learning. It is also possible to detect all or part of the vehicle in the image in the same manner.

In addition, it is assumed that the same object is displayed continuously over a plurality of frames temporally. Therefore, it is desirable to perform tracking processing in order to associate and manage the same type and number of target objects that have been present after a predetermined time has passed. This can be realized, for example, by a method of associating the position of a face detected using optical flow in the next frame, or by a known technique (eg, disclosed in Japanese Patent Application Laid-Open No. 2011-170711).

Also, when information other than image data is input to the input unit 1, it is possible to acquire the existence of the communication target by a protocol established in advance. This makes it possible to detect the object in the same way as using image data.

The feature information management unit 500 is a database for managing information on persons, objects, and vehicles obtained from the input unit 1 in association with each other. In other words, the feature information management unit 500 is a database that manages feature information required when the recognition unit 3 identifies the person of the target object or discriminates the type of vehicle.

FIG. 3 is a diagram showing an example of the structure of information managed by the feature information management unit 500. As shown in FIG. As shown in FIG. 3, the feature information management unit 500 manages the facial feature information output from the detection unit 2 in association with each individual. Information related to the person, such as face image, name, gender, age, and height, as well as related information such as the person's belongings, clothes worn, and vehicle information are associated with each facial feature information and managed. It is also possible to

The content actually managed as facial feature information may be the data itself output by the facial feature extraction means, or may be an m×n feature vector, a partial space, or a correlation matrix immediately before performing KL expansion.
In addition to the characteristic information described above, statistical information such as attribute information such as gender and age can also be managed by the characteristic information management unit 500 . For gender discrimination, feature information for men and feature information for women are stored and used for identification in attribute discrimination processing. It also manages feature information used to classify categories for other attributes.

In order to newly register a face and related information in the feature information management unit 500, a method may be adopted in which the system administrator obtains the consent of the user before registering. Alternatively, as a result of recognition by the recognition unit 3, when the degree of similarity of the characteristic information of any person does not exceed a predetermined threshold, that is, when the corresponding person does not exist, the person is additionally registered as a new person. good too.

The recognizing unit 3 generates information including detailed types, individual identifications, and states from the information on people, objects, vehicles, etc. detected by the detecting unit 2 .
As a method of recognizing a person, a method of recognizing a person using a face will be described below. As a method for extracting facial feature information indicating individual features based on the detected face area, for example, as shown in Japanese Patent Application Laid-Open No. 2007-4767, a model is obtained for one piece of person image information. It is possible to obtain facial feature information by creating an image in which the orientation and state of the face are intentionally changed using the . By using a plurality of continuous images for the same person and performing calculation using moving images, it is possible to perform recognition processing with higher accuracy.

For example, it is disclosed in Reference 3 (Kazuhiro Fukui, Osamu Yamaguchi, Kenichi Maeda: "Face Recognition System Using Moving Images" Institute of Electronics, Information and Communication Engineers Research Report PRMU, vol97, No.113, pp17-24 (1997)). , the mutual subspace method is known. In this method, an image of m×n pixels is cut out from continuously obtained image data. Then, a correlation matrix of feature vectors is obtained from these data, and orthonormal vectors are obtained by KL expansion, thereby calculating a partial space representing facial features obtained from consecutive images. In addition, reference 4 (Kazuhiro Fukui, Osamu Yamaguchi: ``Extraction of facial feature points by combination of shape extraction and pattern matching'', Transactions of the Institute of Electronics, Information and Communication Engineers (D), vol.J80-D-II, No.8, pp2170 -2177 (1997)) is also known.

A subspace can be calculated by obtaining a correlation matrix (or covariance matrix) of feature vectors and obtaining orthonormal vectors (eigenvectors) by KL expansion thereof. A subspace is represented by selecting k eigenvectors corresponding to eigenvalues in descending order of eigenvalues and using the set of eigenvectors.

In this embodiment, the correlation matrix Cd is obtained from the feature vector, and the correlation matrix Cd=ΦdΛdΦdT is diagonalized to obtain the eigenvector matrix Φ. This information becomes a subspace indicating the features of the face of the person to be recognized. The feature information such as the partial space output by the above method is used as personal face feature information for the face detected from the input image data.

By comparing the facial feature information calculated in this manner with the facial feature information managed by the feature information management unit 500, a person can be specified. As a result of this search processing, the person ID managed by the feature information management unit 500 for identifying an individual and the index indicating the similarity which is the calculation result are returned in descending order of similarity. In addition, information managed for each individual by the characteristic information management unit 500 may be returned together. It should be noted that it is not always necessary to send and receive attached information in the search process itself, since correspondence can basically be made by means of the identification ID.

The similarity index is the degree of similarity between subspaces managed as facial feature information. Calculation methods such as the subspace method and the composite similarity method can be used to obtain the degree of similarity. In these methods, both recognition data in prestored registration information and input data are represented as subspaces calculated from a plurality of images. The "angle" formed by the two subspaces is defined as the degree of similarity.

Correlation matrix Cin is similarly obtained for the input data string, diagonalized as Cin=ΦinΛinΦinT, and eigenvector Φin is determined. The inter-subspace similarity (0.0 to 1.0) of the two subspaces represented by Φin and Φd is obtained and used as the similarity for recognition. Again, the technique disclosed in reference 3 can be used.

It is also possible to improve the accuracy by collectively projecting a plurality of person images that are known to be the same person in advance into a partial space, and identifying whether the person is the person or not. As a similarity index other than the degree of similarity, it is clear that a distance on the feature space and the Mahalanobis distance can also be used. When the distance is used, the larger the numerical value, the lower the similarity. Therefore, the smaller the value than the predetermined threshold value, the higher the similarity with the registered person. That is, the recognition unit 3 identifies the individual of the person detected by the detection unit 2, and refers to the characteristic information stored in the characteristic information management unit 500 based on the identified individual, thereby improving recognition accuracy. Let

In order to identify an individual person, it is necessary to register the face information of the specific target in advance. However, if only attribute information such as age and sex is determined, it is possible to obtain information on a person without prior registration. For example, it is possible to estimate the age and gender of a person by using the technology disclosed in (Japanese Patent Application Laid-Open No. 2012-3623). By using this technology and preparing images according to categories to be identified in advance, it is possible to identify races, and items worn on the face such as hats, glasses, and masks.

By using the above-described person detection, face detection results, and angle-of-view information of the camera 300, it is possible to obtain the height from the relative ratio to the entire image. In addition, by using the technique described in (Japanese Patent Application Laid-Open No. 2003-141551), it is also possible to obtain facial movement information such as line of sight and face direction.

In addition, from the relative position from the top of the head position of the person detection coordinate and the relative position from the face detection position, the wearable items such as clothes and hats, the possessions held in the hand, and the belongings worn such as bags and wristwatches are calculated in advance. It is also possible to specify by discriminating from the previously learned dictionary.

Furthermore, the recognition unit 3 discriminates at least one combination of the number plate of the vehicle in which the person was riding, the type of vehicle, and the color of the vehicle. In addition, it is also possible to discriminate the manufacturer and the brand by collecting images for each manufacturer and brand and creating a dictionary for discrimination with respect to belongings and vehicle models.

For vehicles, it is also possible to discriminate vehicle types using a method using CoHOG features by collecting and learning images of vehicle types to be detected corresponding to variations in the assumed orientation. It is also possible to determine the color information of the vehicle by using the color information, and it is also possible to identify the individual vehicle by using the technology described in Japanese Patent No. 4901676.

Furthermore, it is possible to improve the discrimination accuracy by combining a plurality of pieces of discriminated information. For example, when a vehicle is detected, it is possible to individually identify the vehicle based on the license plate information of the vehicle, and to identify the person riding in the vehicle by limiting the person associated with the vehicle. If the combination is not used, it is necessary to match all the faces of persons registered in the feature information management unit 500 to identify candidates. By correlating and managing the model and license plate of the car owned by the system, the matching process of the person is narrowed down to the person associated with the car with the detected license plate. You can prevent it from happening.

Even if the license plate of the car is not known, information on the color and model of the car, or the type of clothes and personal belongings of the car are associated with the visitor and managed as history information, and the combinations of these are managed statistically. is preferred. In this way, even when a plurality of candidates show similar indices based only on facial features, it is possible to increase the accuracy of identifying a person by taking into account information other than the face, such as belongings, clothes, and vehicles. is possible. In this case, in addition to the degree of similarity based on the facial features of the person, it can be realized by a method of adding a predetermined coefficient to the degree of similarity when there is other property or vehicle information.

Moreover, since a person wears and possesses a property, it is easy to associate the property with the person. On the other hand, in order to determine whether a person is a group or to associate an owner with a vehicle, for example, there is a method of associating objects detected simultaneously within the same screen for a predetermined time or longer. Alternatively, it is determined whether or not the number of simultaneous detections in the past history is a predetermined number or more, or whether or not the distance between a person and a person or between a person and a vehicle is within a predetermined time period or longer. You may make it discriminate|determine. In addition, as disclosed in Japanese Patent Application Laid-Open No. 2007-249953, it is also possible to implement a grouping method using the positional relationships and times passed at a plurality of points.

Similarly, while the user is away from the facing device 200, the user's belongings, the approach of a person other than the user to the vehicle, and the removal of the belongings or the vehicle are detected. Is possible. For example, when a person is detected while a property or a vehicle exists in the screen, it can be determined by face matching whether or not the person is the owner of the target object. This process may be performed when the control unit 5 determines that the target object needs to be monitored.

Whether or not the detected person is the owner of the vehicle may be determined using information managed by the characteristic information management unit 500, or the information exists together with the control unit 5 immediately before the start of monitoring. A person may be registered as an owner. If it is determined that the person is not the owner, it can be determined that there is a possibility that a suspicious person is approaching. Also, when it is detected that the target object has moved out of the screen, it is possible to determine that the target object has been taken away.

Through the above processing, it is possible to obtain, from the image data obtained from the input unit 1, a method for determining a combination of the user of the facing device 200, an accessory, a nearby object, or a vehicle, and detailed information about each. .

The operation unit 4 is means for inputting the results of user's operations to the system. For example, by attaching a touch panel sensor to the display 600, it is possible to input operation information in accordance with the screen. In addition, a keyboard and mouse may be installed, or dedicated buttons may be installed near the display 600 . The operation unit 4 can be used for menu selection, character input for information retrieval, pointer operation for indicating a location, and the like.

A mobile terminal such as a mobile phone or a smart phone owned by the user may be wirelessly connected to the operation unit 4 so that information can be input using the mobile terminal. The input may be made using non-contact communication such as Bluetooth (registered trademark) or wireless LAN, or the operation information may be transmitted using a cable connection. As described above, it is possible to input information indicating the user's intention from the operation unit 4 .

The service information management unit 600 is a database that stores information to be provided to users, information related to objects identified by the recognition unit 3, and the like. That is, the service information management unit 600 is a database for collecting and providing service information to be provided to users. Furthermore, it is also possible to record and manage information obtained while the user is using the service as history information.

For example, the service information management unit 600 stores purchase history information, date and time of use, place of use, time of use, and the like regarding services. These pieces of information can also be displayed as reference information on the output unit 6 . These can be realized by combining and recording information displayed to the user from the display 400 and information input from the operation unit 4 .

In addition, by confirming which terminal other than the facing device 200 is using among terminals installed in various places, the living area of the user can be recognized. Therefore, when the service cannot be received at a nearby facility (out of stock, etc.), it is possible to provide information on another store preferentially within the range of the user's living area.

Also, when using the mobile phone, which is owned by the user, by inputting information from the operation unit 4, profiles such as the user's e-mail address, name, date of birth, etc. can be obtained and recorded, or obtained from the vehicle. It is also possible to record daily driving history etc. as information and use it for providing service information.

In addition, information sharing with remote terminals by connecting to the Internet, information such as sales information, inventory information, special treatment information service information provided by pre-affiliated facilities, traffic conditions of parking lots and roads, and information on accidents The service information management unit 600 may manage information such as information, road closure information, and other traffic control information. The managed information is used as a source of information provided by the control unit 5 . The information stored in the service information management unit 600 can be automatically updated by installing a sensor or the like, or can be updated by being input by the manager of the facility.

As described above, information to be provided to the user and information related to the object identified by the recognition unit 3 are recorded, and service information to be provided to the user can be collected and managed so as to be provided. becomes.

The control unit 5 provides services to the user based on the information of the person who is using the facing device 200, the information of the person's belongings, the information of the vehicle, etc., and the information input by means of the operation unit 4. Decide how to provide information and what information to provide. Information in the service information management unit 600 is read based on the determined information and provided to the user.

The control unit 5 can refer to, for example, the following information in order to determine service information to be provided.

- Recognition results obtained by the recognition unit 3 (character attributes, individual identification, companions, belongings, worn items, vehicle information, etc.).
Information (purchase history, etc.) managed by the characteristic information management unit 500 in association with the individual determined by the recognition unit 3;
・Contents and hours of use of services.
• Contents of information output by the output unit 6 .
- Information operated by the operation unit 4 .
The service information management unit 600 can also store content information provided independently of the user. This type of information includes preferential treatment information, preferential treatment information of surrounding facilities, merchandise sales information of surrounding facilities (time sales, inventory, price), congestion status of surrounding roads, and the like. In addition, when the user is specified, the service information management unit 600 also stores the user's past purchase information and the history of frequently used services.

The output unit 6 is a user interface for transmitting information to the user, and displays an image on a display device (display 400) such as a liquid crystal screen or transmits information to the person 900 by outputting sound from a speaker. Moreover, necessary devices may be incorporated according to the content of the service provided by the facing device 200 . For example, in the case of a device for refueling or charging a vehicle, by incorporating a refueling device, a charging device, a display 400, and a speaker into the same housing, the user can check the content of information provided while using the service, and also It is also possible to execute operations by the operation unit 4 .

On the display 400, depending on the installation location of the facing apparatus 200, a screen displaying the sales status of commercial facilities in the surrounding area, a menu screen enabling procedures such as government offices, and a screen usable for examination and examination procedures at a hospital are displayed. may be displayed. Further, at least one or a plurality of devices capable of depositing and withdrawing cash may be incorporated in the facing device 200 in combination. By incorporating a plurality of devices corresponding to services into the facing device 200, the user can use a plurality of services at the same time.

Furthermore, the output unit 6 may be provided with a function for connecting to the Internet for cooperation with external terminals and external facilities. For example, if a communication means such as a mobile phone line or a wireless LAN is used, it is possible to provide the same service even if the mobile terminals are used. Furthermore, as means for contacting when the user leaves the facing apparatus 200, a communication program for executing e-mail communication or socket communication may be incorporated. This makes it possible to secure means of communication with the user who is away from the facing device 200 .

FIG. 4 is a diagram showing an example of the flow of service information in the above configuration. For person A, person B, and car C extracted from the image data, information such as age, sex, combination of persons A and B, vehicle type information of car C, and color of car C is obtained. These pieces of information are statistically processed to calculate statistical information. Information that is statistically determined to be optimal for the information on these objects is preferentially output as recommended information.

For example, if the model of the vehicle recognized by the recognizing unit 3 is known, the estimated value of the vehicle price can be known, so information as to whether or not the person makes expensive purchases can be known. In addition, it becomes possible to provide recommended products suitable for the vehicle type and information on preferential treatment of the products.

Also, if the number of passengers is one, it can be determined that there is a high possibility that the vehicle appeared just by using the intended service, such as refueling or charging. If the number of passengers is two or more, there is a possibility that other boarding members may go to nearby facilities while using services such as charging. Therefore, it is meaningful to use the output unit 6 to notify the preferential treatment information, inventory status, event information, and the like in the surrounding facilities. In addition, if the travel history information can be obtained from the vehicle C, it is also possible to provide congestion information on surrounding roads to be traveled later, information on shops located around the travel route, and the like.

In addition to services such as refueling and charging services for vehicles, automatic teller machines, card payment terminals, government office procedures, etc., similar judgment materials are used based on information on users, companions, and belongings. It is possible to obtain For example, if an adult and a child carrying luxury goods are detected, information on toys for children will be introduced based on the high probability that they will be shopping. Information may be selectable.

For example, as shown in FIG. 7, by preparing a table in which the service to be provided is associated with the combination of detected person's age, gender, belongings, vehicle information, etc., The service provision menu can be changed according to the determination result of the combination. In addition, if the columns filled with ◯ and × in the table of FIG. 7 can be further subdivided by the number of people, it is possible to change the menu depending on whether the same "adult female" is one person or multiple people. can. It is clear that this combination may be any combination of information that can be obtained by the processing already described. In addition to the method using a table, any method may be used as long as the method is capable of uniquely managing outputs for combinations of detected targets.

In addition, if individual identification of a person or vehicle license plate information is successful, it is possible to check the past service usage status. As a result, when the user tries to use the service, it is not necessary to search for the frequently used menu from the general menu, and it is possible to preferentially display the frequently used menu in the past history. it is obvious. As described above, according to the embodiment, it is possible to provide an appropriate service according to the combination of the user's person, property, and vehicle.

In addition, when an anniversary such as a birthday is registered in advance, a congratulatory message is sent, along with providing purchase information on products likely to be purchased on the anniversary, and displaying discount information and preferential treatment information. good too.

In addition, there is a method of changing the display contents depending on the time during service use.

For example, when you start using the service, you can display preferential information related to the use of the service, preferentially display frequently used service menus based on past purchase history information, and receive news about your belongings and the vehicle in question. etc. should be displayed. This makes it possible to provide useful information to a person who is about to use the service. It is desirable to output the above-mentioned birthday message and the like at the start of service use.

If the service is being used, the content that the user can do will change according to the remaining time of the service, so the content is switched and displayed. For example, if the remaining time is long, the user leaves and provides information about available services, and when the remaining time of the service becomes short, information about payment information is displayed. In addition, when the user is away from the site, it is possible to send an e-mail or the like to inform the user that the service will be terminated soon when the remaining time becomes short.

As for the display after the end of the service, information for contacting the parcel mailing service may be provided according to the size of the car and purchase information. Also, at this timing, it is possible to issue preferential treatment information and coupons that can be used from the next time onwards. As described above, it is possible to change the content of the service to be provided according to the remaining time of the user's use of the service.

In addition, the user can visit the surrounding facilities by himself/herself and receive another service while using the service. Furthermore, it is possible to request a proxy service from the menu displayed on the display 400 of the facing device 200 .

For example, the inventory status and sales information (price information) of stores around the gas station can be displayed on the display 400, and a menu can be presented to the user to select a purchase list. At that time, the profile of the recognized user can be reflected in the menu. The user settles the purchase by paying with cash or a card using the payment function provided in the facing device 200.例文帳に追加The purchased item may be delivered from the store to the location of the facing apparatus 200, or sent to the user's home by a delivery service.

In particular, according to the embodiment, it is possible to estimate the living area of each user from the driving history information of the car and the history of the place of use of the facing device 200 . It is also possible to provide shopping information from information on facilities other than nearby facilities based on the estimated living area. For example, if the desired service cannot be received at a nearby facility due to lack of inventory, the user should be notified that the service can be received at another facility within the user's range of activity. As described above, according to the embodiment, it is possible for the user to request the peripheral facilities to provide services. In addition, it is also possible to make a request from the facing device 200 to act as a proxy for procedures to be carried out at a public institution or a government office.

Also, if the user desires, it may be possible to leave the property or vehicle while using the service and leave the place until the end of the service. In that case, it is possible to manage the belongings entrusted to the custody by placing them in a safe, etc., but the vehicle may be managed in a mobile parking lot, or the image data from the input unit 1 may be used to monitor suspicious persons or takeaways. You may make it As already described, it is possible to recognize the fact that the object has been taken away by the processing of the recognition unit 3 . Based on the recognition results, it can be determined that a person other than a pre-registered person has approached the deposited property or vehicle, or that the deposited property or vehicle has been taken away. It is possible to notify existing users. As described above, according to the embodiment, it is possible to realize a monitoring/storage service for user's belongings and vehicles.

As described above, according to the embodiment, it is possible to recognize at least one object of the information of the person appearing in the image captured by the camera 300, the property of the person, or the vehicle in which the person is riding. . Therefore, it is possible to provide appropriate information according to the combination of targets using the service.

That is, from the image data acquired by the camera 300, the computer 100 extracts at least one of the attributes of the target person included in the image data, the combination of the accompanying person and the person, the person and the property, and the combination of the person and the boarding vehicle. recognize. In addition, the recognition accuracy is improved by identifying the individual from the recognized information and combining it with the surrounding information. It also has a function of obtaining information by means of communication, and obtains the mobile phone owned by the person, the travel history of the vehicle, and the like.

The facing device 200 includes a display 400 that outputs video, a speaker that outputs audio, and an operation unit 4 . The operation unit 4 is capable of information communication with a terminal device owned by the user, and is capable of display and input on the user's terminal device (mobile phone, etc.). Further, the face-to-face device 200 may further include at least one of a refueling function, a charging function, an automatic cash payment function, and a credit card settlement function.

Further, the computer 100 displays common information for the same person on the display 400 based on the identified attribute information for each person. The display 400 can switch between a recommendation screen displaying recommended information and a screen displaying an arbitrary menu selected by the user.

The computer 100 provides recommendation information (preferential treatment information (coupon ticket, etc.)) according to the result of authenticating the object. Also, if the user is a repeater, the past history is referred to. Also, the computer 100 estimates income information from the information of the owned property or the car to be used, and changes the recommended products. Furthermore, the computer 100 acquires the travel history of the vehicle, and provides information such as congestion information on surrounding roads and profit information on the return trip based on the content of the travel history.

Further, the computer 100 switches the content output from the output unit 6 according to the service usage time of the operation terminal 200 . That is, the computer 100 provides necessary information about the surroundings while using the service (charging/refueling), and notifies the peripheral facility/store 700 of message information so as to use another service on behalf of the user. This allows the user to select desired products and services from surrounding facilities while waiting. In other words, the user can use the service on behalf of the system and receive another service during that time. In addition, when using the service, users can leave their belongings and cars in the system for management, can monitor suspicious people, and can receive services to detect suspicious people and call when the service ends at a specified time.

With the existing technology, the user had to wait until the service was completed while using the service. On the other hand, according to the embodiment, it is possible to provide appropriate information and services by integrating information on a person who uses a predetermined service, his/her belongings, companions, vehicles used, and the like. Further, according to the embodiment, the user receives necessary information about the surroundings while using the service, uses another service on behalf of the system, or manages service usage time on behalf of the user. It becomes possible for the user himself/herself to receive another service.

From these, it becomes possible to switch the information output form, the information providing method, and the service providing form according to the user of the facing apparatus 200, the person who approaches, the object, and the surrounding situation. It is possible to provide an object detection method capable of providing an appropriate service according to the user.

[Second embodiment]
FIG. 5 is a diagram showing an example of a service providing system according to the second embodiment. In FIG. 5, parts common to those in FIG. 1 are denoted by the same reference numerals, and only different parts will be described here. In the second embodiment, a configuration is assumed in which a plurality of facing devices 200 and computers 100 are installed over a wide area. Computers 100 are connected by a wide area communication network (the Internet, IP-VPN, etc.), and can mutually exchange and use information to be managed.

In FIG. 5, the feature information management unit 500 and the service information management unit 600 are collectively managed in the data center 1000 . A plurality of computers 100 connected to cameras 300 and facing apparatuses 200 are connected to a data center 1000 via a communication line 1100 laid over a wide area, and exchange various data with a feature information management unit 500 and a service information management unit 600. can give and receive.

According to the above configuration, the data center 1000 centrally manages information for detecting and recognizing a person, his/her accompanying person, accessory, and vehicle. As a result, in addition to reducing the amount of data to be managed by the computer 100 on the terminal side, information can be centrally managed at the level of city, prefecture, or country, for example, and service uniformity can be improved. you can get the benefits.

It is also possible to understand the data center 1000 as a so-called cloud computing system. In this form, the feature information management unit 500 and the service information management unit 600 are formed in a distributed database on a wide area network.

[Third embodiment]
FIG. 6 is a diagram showing an example of a service providing system according to the third embodiment. In FIG. 6, parts common to those in FIGS. 1 and 5 are denoted by the same reference numerals, and only different parts will be described here.

The system shown in FIG. 6 corresponds to a more integrated configuration of FIG. That is, each of the plurality of computers (101, 102, 103) is provided with an input unit 1, an operation unit 4, and an output unit 6 for acquiring image data from the camera 300, and the data center 1000 has a feature information management unit 500. , a detection unit 2, a recognition unit 3 and a control unit 5 in addition to the service information management unit 600. FIG. Computers 101, 102, and 103 are provided with communication units 7, and data center 1000 is provided with communication units 70, which can communicate with each other via communication line 1100 such as TCP/IP.

With the above configuration, the computers 101 , 102 , 103 are provided with the minimum functions necessary for service provision sites, and the data center 1000 can be entrusted with processing related to target detection and recognition. As a result, the size of the computers 101, 102, 103 and the facing apparatus 200 can be reduced, and merits such as cost reduction can be obtained.

Alternatively, the computers 101, 102, and 103 may have the detection unit 2, and the data center 1000 may be entrusted with the processing after detection of the target. In this way, the extent of processing to be entrusted to the data center 1000 can be determined according to the system request, the amount of data to be communicated, or the hardware specifications of the device.

Furthermore, it is of course possible to understand the data center 1000 as a cloud computing system. In this case, the detection unit 2, the recognition unit 3 and the control unit 5 can be understood as functional objects distributed over a plurality of computers.

The present invention is not limited to the above embodiments. For example, in the above embodiments, application to a gas station has been described, but the present invention can also be applied to parking lots, commercial facilities, public facilities (government offices), hospitals, beauty salons, and the like.

While several embodiments of the invention have been described, these embodiments are provided by way of example and are not intended to limit the scope of the invention. These novel embodiments can be implemented in various other forms, and various omissions, replacements, and modifications can be made without departing from the scope of the invention. These embodiments and modifications thereof are included in the scope and gist of the invention, and are included in the scope of the invention described in the claims and equivalents thereof.

DESCRIPTION OF SYMBOLS 100, 101, 102... Computer, 200... Meeting apparatus, 300... Camera, 400... Display, 500... Characteristic information management part, 600... Service information management part, 700... Peripheral facility/peripheral store, 800... Communication line, 900... DESCRIPTION OF SYMBOLS 1... Input part 2... Detection part 3... Recognition part 4... Operation part 5... Control part 6... Output part 1000... Data center Communication line 1100 7... Communication part 70... Communication part .

Claims (4)

An object detection method for acquiring image data from a system composed of an imaging device and an information processing device,
a process of detecting belongings, clothing, and clothing pertaining to a person from the acquired image data;
a process in which the system recognizes the shape or color of the object based on the detected characteristic information of the object;
a process in which the system recognizes the person from image data;
A method of object detection comprising: the system associating the recognized object with the recognized person; and the system identifying the person associated with the recognized object. 2. The object detection method of claim 1, wherein the process of recognizing the person from image data by the system includes determining the gender of the person by the system. 2. The object detection method according to claim 1, wherein said person's property is a cane. 2. The object detection method according to claim 1, wherein the person's property is a bag.

Images