JP7375079B2 - Target detection method - Google Patents

Description

Embodiments of the present invention relate to image processing technology.

People (users) who visit commercial facilities, public facilities (government offices), hospitals, and other facilities aim to receive services. However, since the content of the desired services differs from person to person, individuals have had to voluntarily carry out confirmation work and operations to obtain information in order to receive the service. This has been inconvenient for users, such as missing out on opportunities to obtain useful information, and for service providers, resulting in lost sales opportunities.

In response to this situation, several solutions applying image processing technology have been proposed. For example, there is a technique for grouping multiple people included in an image and estimating group attributes. However, the estimated attributes cannot be used to provide services. Furthermore, even if objects such as cars or property are included in the image, the relationship between the person and these objects is not taken into account.

Furthermore, a method has been proposed in which a selection menu is provided according to the composition of a customer group by determining attributes based on gender and height. However, even with this technology, the information used to select a menu is still limited, and it is not possible to provide services from a comprehensive perspective.

Patent No. 4198951 Japanese Patent Application Publication No. 2008-262435 Japanese Patent Application Publication No. 2011-141610

As mentioned above, there is much room for improvement in providing appropriate services to each user. There is a need for technology that can provide more appropriate services by taking into account not only the person but also accompanying objects.
The purpose is to provide an object detection method that can provide appropriate services depending on the user.

According to the embodiment, the object detection method is an object detection method that acquires image data from a system including an imaging device and an information processing device. The object detection method consists of a process of detecting at least one of a person's possessions, clothes, worn items, and a vehicle from the acquired image data, and a process in which the system detects the object based on characteristic information of the detected object. A process in which the system recognizes the shape or color of an object, a process in which the system recognizes a person from image data, a process in which the system associates the recognized object with the recognized person, and a process in which the system recognizes the recognized object. The system includes a step of personally identifying a person associated with the person, and a step of the system confirming past service usage status of the identified person .

FIG. 1 is a diagram showing an example of a service providing system according to the first embodiment. FIG. 2 is a diagram showing an example of objects included in image data. FIG. 3 is a diagram showing an example of the structure of information managed by the feature information management section 500. FIG. 4 is a diagram showing an example of the flow of service information in the above configuration. FIG. 5 is a diagram illustrating an example of a service providing system according to the second embodiment. FIG. 6 is a diagram illustrating an example of a service providing system according to the third embodiment. FIG. 7 is a diagram showing an example of a table for changing the service provision menu according to the determination result.

[First embodiment]
FIG. 1 is a diagram showing an example of a service providing system according to the first embodiment. The service providing system shown in FIG. 1 includes a computer 100 and a face-to-face device 200 connected to the computer 100 via a communication line 800 such as a LAN (Local Area Network).
The face-to-face device 200 is a device that faces the user, such as an automatic teller machine, a card payment terminal, a hospital accounting terminal, or a procedure terminal in a government office. In this embodiment, a device installed at a gas station or a charging station will be considered. Other facilities and stores are often formed around this type of facility. The computer 100 can communicate with surrounding facilities/stores 700 via a communication line 800.

The facing device 200 includes a camera 300 and a display 400. The camera 300 is installed so as to capture, for example, an object directly facing the display 400 in its field of view, and outputs image data obtained by photographing the field of view. The camera 300 is installed at at least one or multiple locations around the facing device 200. The image data includes a person 900 standing in front of the facing device 200, other people with the person 900, property owned by the person 900, a vehicle, a person in the driver's seat of the vehicle, and a person in the passenger seat. It can include people who are present.

The camera 300 is, for example, an ITV camera or an IP camera, and obtains image data by converting optical information obtained through a lens from analog to digital. The camera 300 is also capable of capturing temporally continuous images. FIG. 2 is a diagram showing an example of objects included in image data.

The display 400 is a display medium using digital signage technology, and displays operation information, advertising information, and the like.
The computer 100 includes an input section 1, a detection section 2, a recognition section 3, an operation section 4, a control section 5, and an output section 6 as processing functions according to the embodiment.
The input unit 1 acquires image data output from the camera 300. The detection unit 2 performs image processing on the acquired image data, and individually detects and specifies objects within the field of view of the camera 300. For example, the person 900, a vehicle, or property of the person 900 is detected. In addition, at least one of the following combinations of the detected person 900's age, gender, height, race, clothing, line of sight, direction of face, license plate of the vehicle in which the person is riding, vehicle type, and vehicle color may be considered. Detected.

The recognition unit 3 individually recognizes the detected object by comparing it with the feature information stored in the feature information management unit 500. As a result of recognition, characteristic information such as object attributes, types, and individual identification information (ID) is obtained. The feature information management unit 500 is a database that stores feature information related to objects that can be photographed.

The operation unit 4 is a user interface such as a touch panel or a numeric keypad, which is arranged within the user's reach of the face-to-face device 200, for example. A user can input operation information into the system via the operation unit 4.

The service information management unit 600 is a database that stores service information to be provided to users, information related to the object identified by the recognition unit 3, and the like. The control unit 5 reads service information corresponding to the recognition result obtained by the recognition unit 3 from a plurality of pieces of service information stored in the service information management unit 600 and passes it to the output unit 6. It is also possible to select service information based on operation information input from the operation unit 4.

The output unit 6 displays the read service information on the display 400 or outputs it as audio information from a speaker (not shown). The output unit 6 also has a function of outputting a message signal for notifying the surrounding facilities/stores 700 of service information and requesting the provision of services. Furthermore, the output unit 6 also has a function of notifying service information to the mobile terminal of a person registered in advance. Next, the above configuration will be explained in more detail.

The input unit 1 acquires image data of a person, object, vehicle, or the like who is about to use the facing device 200 or has approached the vicinity. In addition to acquiring image data directly from the camera 300, it is also possible to import an image file shot with a digital still camera or a video file via a medium.

This type of data can be obtained not only from the camera 300 but also from a mobile device owned by the person. For example, the input unit 1 can acquire image data using communication means such as wireless LAN, infrared communication, Bluetooth (registered trademark), and RFID (radio ID) provided in the mobile phone. It is also of course possible to exchange image data by inserting and removing a contact type information exchange means or a recording medium such as a flash memory card.

Furthermore, image data can also be sent and received by exchanging information with other belongings or equipment mounted on the vehicle. For example, information for vehicle identification can be obtained from a vehicle using ETC card information, and information such as fuel and battery remaining capacity information, driving history information managed by a navigation system, etc. can be transmitted via wireless communication. is also possible. These can also be realized using wireless information means, dedicated media, or contact information exchange means. In short, the input unit 1 only needs to be able to acquire digital image data to be processed by some method.

The detection unit 2 detects people, objects, vehicles, etc. included in the image data. For example, in order to detect a person, it is preferable to detect the face area of the person. In order to detect the face area, the detection unit 2 uses brightness information on the image to find coordinates indicating the face area. For example, Reference 1 (Yushi Mita et al.: "Joint Haar-like features based on co-occurrence suitable for face detection" IEICE Transactions (D), vol. J89-D, 8, pp1791-1801 (2006)) It is possible to use the method described in .

The information indicating the detected result may have any shape depending on the direction and size of the face. In this embodiment, for simplicity, the face area is represented by rectangular information, and the coordinates of the corners of the rectangle are used as the detection results. Other methods include a method of determining the correlation value while moving a pre-prepared template within the image, and determining the position that gives the highest correlation value as the face region, and a face extraction method that uses the eigenspace method or subspace method. is also available.

Furthermore, a technique for detecting not only the face but also the entire human region may be used. This type of technology is described in Reference 2 (Watanabe et al., “Co-occurrence Histograms of Oriented Gradients for Pedestrian Detection, In Proceedings of the 3rd Pacific-Rim Symposium on Image and Video Technology” (PSIVT2009), pp. 37-47. , hereinafter referred to as "Co-HOG features").

Once a face or a person's area is detected, it is possible to detect the type of clothing worn by the person or the belongings they own based on the relative positional relationship from the head or face. These items can be detected by collecting many images of objects to be detected in advance and performing statistical learning using the aforementioned Joint Haar-like features and Co-HOG features. It is also possible to detect all or part of a vehicle within an image in a similar manner.

Note that it is assumed that the same object is temporally shown continuously over multiple frames. Therefore, it is desirable to perform a tracking process in order to manage the existence of the same type and number of target objects after a predetermined period of time has elapsed. This can be realized by, for example, a method of associating the position of a detected face in the next frame using optical flow, or a known technique (for example, described in Japanese Patent Laid-Open No. 2011-170711).

Furthermore, when information other than image data is input to the input unit 1, it is possible to acquire the existence of a communication target using a protocol established in advance. This makes it possible to detect objects in the same manner as using image data.

The feature information management unit 500 is a database for managing information on people, objects, and vehicles obtained from the input unit 1 in a manner that associates them with each other. In other words, the feature information management section 500 is a database that manages feature information necessary for the recognition section 3 to identify the person of the target object or to determine the type of vehicle.

FIG. 3 is a diagram showing an example of the structure of information managed by the feature information management section 500. As shown in FIG. 3, the feature information management section 500 manages the facial feature information output from the detection section 2 in association with each individual. In association with each piece of facial feature information, information about the relevant person such as facial image, name, gender, age, and height, as well as related information such as the person's belongings, clothing worn, vehicle information, etc., are managed in association with each other. It is also possible to do so.

The content actually managed as facial feature information may be the data itself output by the facial feature extraction means, or may be an m×n feature vector, a subspace, or a correlation matrix immediately before KL expansion.
In addition to the above characteristic information, statistical information such as attribute information such as gender and age can also be managed by the characteristic information management unit 500. Regarding gender discrimination, characteristic information for men and characteristic information for women are stored and used for identification in attribute discrimination processing. For other attributes as well, feature information used to classify categories is managed.

In order to newly register a face and related information in the feature information management unit 500, a system administrator may register the face and related information after obtaining the consent of the user. Alternatively, as a result of the recognition by the recognition unit 3, if the similarity of the feature information of any person does not exceed a predetermined threshold, that is, if the corresponding person does not exist, the person is additionally registered as a new person. Good too.

The recognition unit 3 generates information including detailed type, individual identification, and state from information on the person, object, vehicle, etc. detected by the detection unit 2 .
As a method for recognizing a person, a method for recognizing a person using a face will be described below. As a method for extracting facial feature information indicating individual characteristics based on the detected facial area, for example, as shown in (Japanese Patent Laid-Open No. 2007-4767), a model is extracted for one person image information. It is possible to obtain facial feature information by creating an image in which the orientation and state of the face are intentionally changed using the . For the same person, more accurate recognition processing can be performed by using multiple consecutive images and performing calculations using moving images.

For example, it is disclosed in Reference 3 (Kazuhiro Fukui, Osamu Yamaguchi, Kenichi Maeda: "Face recognition system using moving images" Institute of Electronics, Information and Communication Engineers Research Report PRMU, vol97, No.113, pp17-24 (1997)) , a method using a mutual subspace method is known. In this method, an m×n pixel image is cut out from continuously obtained image data. Then, a correlation matrix of feature vectors is determined from these data, and an orthonormal vector is determined by KL expansion, thereby calculating a subspace representing facial features obtained from consecutive images. In addition, reference 4 (Kazuhiro Fukui, Osamu Yamaguchi: "Facial feature point extraction by combination of shape extraction and pattern matching" IEICE Transactions (D), vol. J80-D-II, No. 8, pp2170) -2177 (1997)) is also known.

The subspace can be calculated by finding the correlation matrix (or covariance matrix) of the feature vectors and finding orthonormal vectors (eigenvectors) by KL expansion thereof. The subspace is expressed by selecting k eigenvectors corresponding to the eigenvalues in descending order of the eigenvalues and using the set of eigenvectors.

In this embodiment, the correlation matrix Cd is determined from the feature vectors, and the correlation matrix Cd=ΦdΛdΦdT is diagonalized to determine the eigenvector matrix Φ. This information becomes a subspace indicating the facial features of the person targeted for recognition. The feature information such as the partial space output by the above method is used as the individual facial feature information for the face detected from the input image data.

By comparing the facial feature information calculated in this manner with the facial feature information managed by the feature information management section 500, a person can be identified. As a result of this search process, the person ID managed by the feature information management unit 500 to identify individuals and indices indicating similarity, which are calculation results, are returned in order of similarity. Further, information managed for each individual by the characteristic information management unit 500 may also be returned. Note that since correspondence can basically be established using identification IDs, it is not necessarily necessary to exchange attached information in the search process itself.

The similarity index is the degree of similarity between subspaces managed as facial feature information. Calculation methods such as a subspace method and a composite similarity method can be used to determine the degree of similarity. In these methods, both recognition data in registered information stored in advance and input data are expressed as a subspace calculated from a plurality of images. Then, the "angle" formed by the two subspaces is defined as the degree of similarity.

Correlation matrix Cin is similarly obtained for the input data string, diagonalized as Cin=ΦinΛinΦinT, and eigenvector Φin is determined. The inter-subspace similarity (0.0 to 1.0) of the two subspaces represented by Φin and Φd is determined, and this is used as the similarity for recognition. Here, too, it is possible to use the technique disclosed in reference document 3.

It is also possible to improve accuracy by projecting a plurality of human images that are known to be the same person in advance into a partial space and identifying whether or not the person is the same person. It is clear that distance on a feature space or Mahalanobis distance can also be used as a similarity index other than similarity. When distance is used, the larger the numerical value, the lower the similarity; therefore, the smaller the value is than the predetermined threshold value, the higher the similarity with the registered person. In other words, the recognition unit 3 identifies the individual of the person detected by the detection unit 2, and improves recognition accuracy by referring to the characteristic information stored in the characteristic information management unit 500 based on the identified individual. let

In order to identify a person, it is necessary to register the face information of the target in advance. However, if only attribute information such as age and gender is determined, it is possible to obtain information about a person without prior registration. For example, by using the technology disclosed in (Japanese Unexamined Patent Publication No. 2012-3623), it is possible to estimate a person's age and gender. By using this technology and preparing images according to the categories to be determined in advance, it becomes possible to determine race and what people are wearing on their faces, such as hats, glasses, and masks.

By using the above-described person detection, face detection results, and view angle information of the camera 300, it is also possible to determine the height from the relative proportion to the entire image. In addition, by using the technology described in (Japanese Patent Application Laid-Open No. 2003-141551), it is also possible to obtain facial movement information such as line of sight and face orientation.

In addition, based on the relative position of the person detection coordinates from the top of the head position and the relative position from the face detection position, we can determine in advance what people are wearing such as clothes and hats, what they are holding in their hands, and what they are wearing such as bags and watches. It is also possible to identify the dictionary by comparing it with the dictionary learned in the previous study.

Further, the recognition unit 3 determines at least one of the combination of the license plate, the type of vehicle, and the color of the vehicle in which the person was riding. Furthermore, it is also possible to identify the manufacturer and brand of owned items and car models by collecting images for each manufacturer and brand and creating a dictionary for identification.

In addition, it is also possible to identify vehicle types using a method using CoHOG features by collecting and learning images of vehicle types to be detected based on variations in the expected orientation. In addition to being able to determine the color information of a car by using color information, it is also possible to individually identify a vehicle by using the technology described in Japanese Patent No. 4901676.

Furthermore, it is also possible to improve the discrimination accuracy by combining a plurality of pieces of discriminated information. For example, when a vehicle is detected, it is possible to identify the vehicle individually based on the license plate information of the vehicle, and to identify the person riding in the vehicle, limited to the person associated with the vehicle. If a combination is not used, it would be necessary to identify a candidate by matching all the faces of people registered in the feature information management unit 500. By associating and managing the car model and license plate of the car owned by the user, the matching process will focus on the person associated with the car with the detected license plate, which will prevent erroneous identification of individuals. You can prevent it from happening.

Even if you do not know the license plate of a car, you can manage historical information by associating the color and model of the car, or the type of clothing and belongings with the person who visited the event, and statistically manage these combinations. It is preferable. In this way, even if multiple candidates show similar indicators based on facial features alone, the accuracy of identifying a person can be increased by taking into account information from possessions, clothing, and vehicles other than the face. is possible. In this case, in addition to the similarity based on the facial features of the person, if other property or vehicle information is present, this can be achieved by multiplying the similarity by a predetermined coefficient and adding the similarity.

Furthermore, since the property is worn or owned by the person, it is easy to associate it with the person. On the other hand, in order to determine whether a person is a group or to associate an owner with a vehicle, there is a method of associating objects detected simultaneously on the same screen for a predetermined period of time or longer, for example. Alternatively, it is possible to determine whether the number of simultaneous detections in the past history is greater than or equal to a predetermined number of times, or to determine whether the distance between a person and a person or a person and a vehicle is within a predetermined distance for a predetermined time or more. It may be determined. In addition, it is also possible to use a method of grouping using the positional relationship and time of passing at multiple points, as disclosed in Japanese Patent Application Laid-open No. 2007-249953.

Similarly, while the user is away from the facing device 200, it is detected that the user's belongings, a person different from the user approaches the vehicle, or that the user's belongings or the vehicle are taken away. Is possible. For example, if a person is detected while property or a vehicle is present on the screen, it is possible to determine whether the person is the owner of the object by face matching. This process may be performed when the control unit 5 determines that monitoring of the target object is necessary.

Whether or not the detected person is the owner of the vehicle may be determined using information managed by the characteristic information management section 500, or information that is present immediately before the control section 5 starts monitoring. A person may be registered as the owner. If it is determined that the person is not the owner, it can be determined that a suspicious person may be approaching. Furthermore, if it is detected that the target object has moved out of the screen, it can be determined that the target object has been taken away.

Through the above processing, it becomes possible to obtain a method for determining a combination regarding the user of the facing device 200, an incidental object, a nearby object, or a vehicle, and detailed information regarding each, from the image data obtained from the input unit 1. .

The operation unit 4 is a means for inputting the results of user operations into the system. For example, by attaching a touch panel sensor to the display 600, it is possible to input operation information that matches the screen. In addition, a keyboard and a mouse may be installed, or a dedicated button may be installed near the display 600. The operation unit 4 can be used to select a menu, input characters for information retrieval, operate a pointer to indicate a location, and the like.

Note that the operation unit 4 may be wirelessly connected to a mobile terminal such as a mobile phone or a smartphone owned by the user, so that information can be input using the mobile terminal. The input information may be input using non-contact communication such as Bluetooth (registered trademark) or wireless LAN, or the operation information may be transmitted using a cable connection. In the manner described above, it is possible to input information indicating the user's intention from the operation unit 4.

The service information management unit 600 is a database that stores information to be provided to users, information related to objects identified by the recognition unit 3, and the like. In other words, the service information management unit 600 is a database that collects and makes available service information to be provided to users. Furthermore, it is also possible to record and manage information obtained while the user is using the service as historical information.

For example, the service information management unit 600 stores purchase history information, date and time of use, place of use, time of use, etc. regarding the service. This information can also be displayed on the output unit 6 as reference information. These can be realized by combining and recording the information displayed to the user from the display 400 and the information input from the operation unit 4.

In addition, by checking which of the terminals installed in various places the user is using in addition to the face-to-face device 200, the user's living area can be recognized. Therefore, if the service is not available at a nearby facility (for example, out of stock), information on another store within the user's living area can be given priority.

In addition, when using the mobile phone, if information from the mobile phone that is owned is entered from the operation unit 4, the user's profile such as email address, name or date of birth can be acquired and recorded, and information obtained from the vehicle can be obtained and recorded. It is also possible to record daily driving history and the like as information and use it to provide service information.

In addition, by connecting to the Internet, information can be shared with remote terminals, information such as sales information, inventory information, preferential information services, etc. provided by surrounding facilities with which we have partnered in advance, information on parking lot and road congestion conditions and accidents. The service information management unit 600 may manage information such as information, road closure information, and other traffic regulation information. The managed information is used as a source of information provided by the control unit 5. The information stored in the service information management section 600 may be automatically updated by installing a sensor or the like, or may be updated by input by a facility manager.

As described above, it is possible to record information to be provided to users and information related to objects identified by the recognition unit 3, and to collect and manage service information to be provided to users. becomes.

The control unit 5 provides services to the user based on information such as information on the person using the face-to-face device 200, information on the person's property or vehicle, and information input through the operation unit 4. Decide on the provision method and content of information provision. Information from the service information management unit 600 is read out based on the determined information and provided to the user.

The control unit 5 can refer to the following information, for example, in order to determine the service information to be provided.

- Recognition results obtained by the recognition unit 3 (person attributes, individual identification, companions, possessions, things worn, vehicle information, etc.).
- Information (purchase history, etc.) that is determined by the recognition unit 3 and is associated with the individual and managed by the characteristic information management unit 500.
・Service usage details and usage time.
- Contents of information output by the output unit 6.
- Information operated on the operation unit 4.
Note that the service information management unit 600 can also store content information provided independently of the user. This type of information includes preferential treatment information, preferential treatment information of surrounding facilities, product sales information (time sales, inventory, prices) of surrounding facilities, congestion status of surrounding roads, and the like. In addition, when a user is identified, the user's past purchase information and history of frequently used services are also stored in the service information management unit 600.

The output unit 6 is a user interface for transmitting information to the user, and displays an image on a display device (display 400) such as a liquid crystal screen, and transmits information to the person 900 by outputting audio from a speaker. Further, necessary equipment may be incorporated depending on the content of the service provided by the face-to-face device 200. For example, if it is a device that refuels or charges a vehicle, by incorporating the refueling device or charging device, the display 400, and the speaker into the same housing, the user can check the information provided while using the service, and also It is also possible to execute operations using the operation unit 4.

Depending on the installation location of the face-to-face device 200, the display 400 includes a screen that displays the sales status of nearby commercial facilities, a menu screen that allows procedures at government offices, etc., and a screen that can be used for procedures for hospital tests and consultations. It may be displayed. Further, the face-to-face device 200 may include a combination of at least one or a plurality of devices capable of depositing and withdrawing cash. By incorporating a plurality of devices corresponding to services into the face-to-face device 200, a user can use a plurality of services at the same time.

Furthermore, the output unit 6 may be provided with a function for connecting to the Internet for cooperation with external terminals and external facilities. For example, if a communication means such as a mobile phone line or a wireless LAN is used, it is possible to provide similar services even when the terminal is composed of a movable terminal. Furthermore, as a means for contacting the user when he or she leaves the face-to-face device 200, a communication program that executes e-mail communication or socket communication may be built-in. This makes it possible to secure a means of contacting a user who is located away from the facing device 200.

FIG. 4 is a diagram showing an example of the flow of service information in the above configuration. Information such as age, gender, combination of persons A and B, vehicle type information of car C, and color of car C can be obtained for person A, person B, and car C extracted from the image data. This information is statistically processed to calculate statistical information. Then, among the information regarding these objects, information that is statistically determined to be optimal is preferentially output as recommended information.

For example, if the model of the vehicle recognized by the recognition unit 3 is known, the estimated price of the vehicle can be known, and therefore information on whether the person is a person who makes expensive purchases can be known. In addition, it will be possible to provide recommended products according to the car model and information on preferential treatment of those products.

Furthermore, if there is only one person on board, it can be determined that there is a high possibility that the vehicle appeared simply by using the desired service, such as refueling or charging. If the number of passengers is two or more, there is a possibility that other members of the vehicle may go to nearby facilities while charging or using the service. Therefore, it is meaningful to notify the user of preferential treatment information, inventory status, event information, etc. at surrounding facilities through the output unit 6. In addition, if driving history information can be obtained from the vehicle C, it is also possible to provide information such as congestion on surrounding roads on which the vehicle will travel later, and information on stores located around the driving route.

In addition to vehicle refueling and charging services, cash machines, card payment terminals, government office procedures, etc., similar judgment materials are used based on information about the user, companions, and belongings. It is possible to obtain. For example, if an adult and a child are found to be carrying luxury items, information on toys for children will be provided based on the combination of people and their belongings, as they are likely to shop there. Information may be selectable.

For example, as shown in Figure 7, by preparing a table that associates the services to be provided according to the combination of the detected person's age, gender, belongings, car information, etc. The service provision menu can be changed according to the combination determination result. Additionally, by making it possible to further subdivide the ○/x fields in the table in Figure 7 by number of people, it is also possible to change the menu depending on whether there is one "adult woman" or multiple people. can. It is clear that this combination may be any combination of information that can be obtained through the processing described above. In addition to the method using a table, any method may be used as long as it is possible to uniquely manage outputs for combinations of detected objects.

Furthermore, if the individual identification of a person or vehicle based on license plate information is successful, it will be possible to check the usage status of past services. As a result, when users try to use the service, they do not have to search for frequently used menus from the general menu, and it is possible to preferentially display frequently used menus based on past history. it is obvious. As described above, according to the embodiment, it is possible to provide appropriate services depending on the combination of the user's person, property, and vehicle.

In addition, if an anniversary such as a birthday is registered in advance, it will send a congratulatory message, provide purchase information for products likely to be purchased on the anniversary, and display information on discounts and preferential treatment. Good too.

Another method is to change the displayed content depending on the time you are using the service.

For example, at the time of starting to use a service, preferential treatment information related to the use of the service may be displayed, frequently used service menus may be preferentially displayed based on past purchase history information, and news regarding personal belongings or the applicable vehicle may be displayed. It would be better to display something like this. This makes it possible to provide useful information to people who are about to use the service. It is desirable to output the aforementioned birthday message and the like when starting to use the service.

If the user is using the service, the content that the user can do changes depending on the remaining time of the service, so the content is switched and displayed. For example, if there is a long time remaining, the user can leave the location and provide information about available services, and if the time remaining for the service is getting short, information about payment information may be displayed. Furthermore, if the user is away from the location, when the remaining time is running low, it is possible to send an e-mail or the like to inform the user that the service will soon end.

Further, regarding the display after the service ends, information for contacting the package mailing service may be provided in accordance with the size of the car and purchase information. You can also use this timing to issue preferential treatment information and coupons that can be used from next time onwards. As described above, it is possible to change the content of the provided service according to the remaining time of the user's service usage time.

Furthermore, while using the service, the user can also visit nearby facilities and receive other services. Furthermore, it is possible to request a service on behalf of the user from the menu displayed on the display 400 of the face-to-face device 200.

For example, the inventory status and sales information (price information) of stores around the gas station can be displayed on the display 400, and a menu can be presented to the user to select a purchasing list. At that time, the recognized user's profile can also be reflected in the menu. The user finalizes the purchase by paying with cash or card using the payment function provided in the face-to-face device 200. The purchased item may be delivered from the store to the location of the facing device 200, or may be sent to the user's home by a delivery service.

In particular, according to the embodiment, the living area of each user can be estimated from the driving history information of the car and the history of the usage location of the facing device 200. It is also possible to provide shopping information from information on facilities other than nearby facilities based on the estimated living area. For example, if the desired service is not available at a nearby facility due to lack of stock, the user may be informed that the service is available at another facility within the user's range of activity. As described above, according to the embodiment, the user can request surrounding facilities to provide services. In addition, requests may be made from the face-to-face device 200 to perform procedures on behalf of public institutions or government offices.

Furthermore, if the user desires, the user may be allowed to leave his/her belongings or vehicle while using the service and leave the location until the end of the service. In that case, you can manage the deposited property by putting it in a safe, etc., but you can manage the vehicle in a mobile parking lot, or use the image data from input unit 1 to monitor for suspicious persons or people taking things away. You may also do so. As already mentioned, it is possible to recognize that the item has been taken away by the processing of the recognition unit 3. Based on the recognition results, it is possible to detect when a person who is not registered in advance approaches the property or vehicle that has been entrusted to the vehicle, or that the property or vehicle that has been entrusted to the vehicle is being taken away. It is possible to notify the current users. As described above, according to the embodiment, it is possible to realize a monitoring and storage service for user's belongings and vehicles.

As described above, according to the embodiment, it becomes possible to recognize at least one object of the person, the property of the person, or the vehicle information on which the person is riding that is shown in the image taken by the camera 300. . Therefore, it is possible to provide appropriate information depending on the combination of objects using the service.

That is, from the image data acquired by the camera 300, the computer 100 determines at least one of the attributes of the target person included in the image data, the combination of the accompanying person and the person, the person and property, and the combination of the person and the vehicle they are boarding. recognize. It also identifies individuals from the recognized information and increases recognition accuracy by combining it with surrounding information. It also has a function to obtain information through communication means, and obtains the driving history of a person's mobile phone or vehicle.

The face-to-face device 200 includes a display 400 that outputs video, a speaker that outputs audio, and an operation unit 4. The operation unit 4 can communicate information with a terminal device owned by the user, and can perform display and input on the user's terminal device (such as a mobile phone). Further, the meeting device 200 may further include at least one of a refueling function, a charging function, an automatic cash payment function, a credit card payment function, and the like.

Furthermore, the computer 100 displays information common to the same person on the display 400 based on the attribute information for each identified person. The display 400 can be switched between a recommendation screen that displays recommended information and a screen that allows the user to select and display an arbitrary menu.

The computer 100 provides recommendation information (preferential information (coupons, etc.)) according to the result of authenticating the object. Also, if the user is a repeat user, past history is used as a reference. Furthermore, the computer 100 estimates income information from information about possessions or cars used, and changes the recommended products. Further, the computer 100 obtains the driving history of the car, and based on the information, provides information such as congestion information on surrounding roads and advantageous information on the way home.

Further, the computer 100 switches the content output from the output unit 6 according to the service usage time of the operation terminal 200. That is, the computer 100 provides necessary information about the surroundings while using a service (charging/refueling), or notifies surrounding facilities/stores 700 of message information in order to use another service on your behalf. This allows users to select desired products and services from nearby facilities during their waiting time. In other words, the user can have the system use the service on his/her behalf and receive other services during that time. Furthermore, when using the service, users can entrust their belongings and cars to the system and have them managed, monitor for suspicious persons, and receive services that will be called when a suspicious person is detected or the service ends at a specified time.

With existing technology, users have to wait until the service is completed while using the service. On the other hand, according to the embodiment, it becomes possible to provide appropriate information and services by integrating information on the person using a predetermined service, his/her belongings, companions, vehicle used, etc. Further, according to the embodiment, while using the service, the user can receive necessary information from the surroundings, have the system use another service on his or her behalf, or have the system manage the service usage time on his or her behalf. Users themselves can receive other services.

For these reasons, it is now possible to switch the output form of information, the method of providing information, and the form of service provision according to the user of the face-to-face device 200, the person approaching, the object, and the surrounding situation. It becomes possible to provide an object detection method that can provide appropriate services depending on the user.

[Second embodiment]
FIG. 5 is a diagram illustrating an example of a service providing system according to the second embodiment. In FIG. 5, parts common to those in FIG. 1 are denoted by the same reference numerals, and only the different parts will be explained here. In the second embodiment, it is assumed that a plurality of face-to-face devices 200 and computers 100 are installed over a wide area. The computers 100 are connected via a wide area communication network (such as the Internet or IP-VPN), and can mutually exchange and use managed information.

In FIG. 5, a feature information management unit 500 and a service information management unit 600 are collectively managed in a data center 1000. A plurality of computers 100 each connected to a camera 300 and a face-to-face device 200 are connected to a data center 1000 via a communication line 1100 installed over a wide area, and exchange various data with a characteristic information management section 500 and a service information management section 600. Can be given and received.

According to the above configuration, information for detecting and recognizing people, their companions, accompanying objects, and vehicles is centrally managed in the data center 1000. This not only reduces the amount of data that must be managed on the computer 100 on the terminal side, but also makes it possible to centrally manage information at the city, prefecture, or country level, improving the uniformity of services, etc. You can get benefits.

It is also possible to understand data center 1000 as a so-called cloud computing system. In this form, the feature information management section 500 and the service information management section 600 are formed in a distributed database on a wide area network.

[Third embodiment]
FIG. 6 is a diagram illustrating an example of a service providing system according to the third embodiment. In FIG. 6, parts common to those in FIGS. 1 and 5 are denoted by the same reference numerals, and only the different parts will be described here.

The system shown in FIG. 6 corresponds to a further integrated version of the configuration shown in FIG. That is, each of the plurality of computers (101, 102, 103) is provided with an input section 1, an operation section 4, and an output section 6 for acquiring image data from the camera 300, and the data center 1000 is provided with a feature information management section 500. , a detection section 2, a recognition section 3, and a control section 5 are provided in addition to the service information management section 600. The computers 101, 102, and 103 are equipped with a communication section 7, and the data center 1000 is equipped with a communication section 70, so that they can communicate with each other via a communication line 1100 such as TCP/IP.

With the above configuration, the computers 101, 102, and 103 can be provided with the minimum functions required at the site where services are provided, and processing related to object detection and recognition can be entrusted to the data center 1000. This makes it possible to reduce the size of the computers 101, 102, 103 and the face-to-face device 200, and to obtain benefits such as cost savings.

Alternatively, the detection unit 2 may be provided in the computers 101, 102, and 103, and processing after detection of the target may be entrusted to the data center 1000. In this way, the extent to which processing is entrusted to the data center 1000 can be determined depending on the system requirements, the amount of data to be communicated, or the hardware specs of the equipment.

Furthermore, it is of course possible to understand data center 1000 as a cloud computing system. In this case, the detection section 2, the recognition section 3, and the control section 5 can be understood as functional objects distributed and arranged in a plurality of computers.

Note that the present invention is not limited to the above embodiments. For example, in the above embodiment, the application to a gas station has been described, but the present invention can also be applied to parking lots, commercial facilities, public facilities (government offices), hospitals, beauty salons, and the like.

Although several embodiments of the invention have been described, these embodiments are presented by way of example and are not intended to limit the scope of the invention. These novel embodiments can be implemented in various other forms, and various omissions, substitutions, and changes can be made without departing from the gist of the invention. These embodiments and their modifications are included within the scope and gist of the invention, as well as within the scope of the invention described in the claims and its equivalents.

100, 101, 102... Computer, 200... Face-to-face device, 300... Camera, 400... Display, 500... Feature information management department, 600... Service information management department, 700... Surrounding facilities/surrounding stores, 800... Communication line, 900... Person, 1...Input unit, 2...Detection unit, 3...Recognition unit, 4...Operation unit, 5...Control unit, 6...Output unit, 1000...Data center, communication line 1100, 7...Communication unit, 70...Communication unit .

Claims (5)

An object detection method for acquiring image data from a system comprising an imaging device and an information processing device, the method comprising:
A step of detecting at least one of the person's possessions, clothes, and items to be worn, and the person's vehicle from the acquired image data;
a step in which the system recognizes the shape or color of the detected object based on characteristic information of the detected object;
a step in which the system recognizes the person from image data;
a step in which the system associates the recognized object with the recognized person; a step in which the system individually identifies the person associated with the recognized object;
a process in which the system verifies the past usage status of information services of the identified person ;
the system notifying the identified person of the approach of an unregistered person to the deposited property or the vehicle or the removal of the deposited property or the vehicle; An object detection method comprising: 2. The object detection method according to claim 1, wherein the step of the system recognizing the person from image data includes the system determining the gender of the person. The object detection method according to claim 1, wherein the property of the person is a cane. The object detection method according to claim 1, wherein the property of the person is a bag. 2. The object detection method according to claim 1, wherein the vehicle is equipped with an ETC card, and a correlation between the history of the person and the ETC card is recorded.