JP2008257381A - Information analyzing system, information analyzing device, information analyzing method, information analyzing program, and recording medium - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To acquire information about a user from a sensor device, and to analyze the acquired information, and to derive a specific conclusion, and to provide it to a user. <P>SOLUTION: Image data acquired by a sensor device, for example, a camera device are transmitted together with a user ID to a server, and primary analysis processing is applied to the transmitted image data at a server device. In the primary analysis processing, predetermined processing is applied to the image data, and the image data are compared with reference information to be a reference. Threshold decision is made to the result of the primary analysis processing, and when it is decided that the result of the primary analytic processing exceeds the threshold, secondary analysis processing is applied to the image data or the primary analysis result. The secondary analysis processing is performed for analyzing the factor of the fact that the result of the primary analysis processing exceeds the threshold. A user is notified of the result of the secondary analysis processing together with the corresponding image data, for example. <P>COPYRIGHT: (C)2009,JPO&INPIT

Description

  The present invention relates to an information analysis system, an information analysis apparatus, an information analysis method, an information analysis program, and a recording medium that distribute information to a user based on a result of analyzing information acquired about the user.

There is a system in which a user carries a user medium in which his / her position information and time information are stored, and installs a plurality of imaging devices connected to the network in a predetermined area or facility so as to always shoot. It is proposed by the applicant of the present invention and described in Patent Document 1. In the system according to Patent Document 1, when a user wants to receive a photograph or video that he / she has taken, the system reads information stored in user media. On the system side, based on the position and time information read from the user media, the photograph or video taken by the user is extracted and recorded on a portable recording medium such as a disk or memory card, or communicated via a network. Use to provide to users.
JP 2004-336089 A

  In the system described in Patent Document 1, there is a demand for a system that can analyze a photograph or video taken by a photographing apparatus and provide an analysis result to a user. In particular, there is a demand for a system that can derive a specific conclusion based on the analysis result and provide the conclusion to the user.

  Therefore, an object of the present invention is to provide an information analysis system, an information analysis apparatus, and an information analysis method that acquire information related to a user by a sensor device, analyze the acquired information, derive a specific conclusion, and provide the user with the information. And providing an information analysis program and a recording medium.

  In order to solve the above-described problems, a first invention is an information analysis apparatus that outputs information according to a result of analyzing image data. An image detection unit that detects an image related to a user's body from image data; A first analysis unit that performs a first predetermined analysis process based on an image related to the user's body detected by the image detection unit, and a comparison unit that compares an analysis result of the first analysis unit with a threshold value A second analysis unit that performs a second predetermined analysis process based on an image related to the user's body detected by the image detection unit, and based on a comparison result by the comparison unit, It is an information analysis apparatus characterized by determining whether or not to perform a second predetermined analysis process.

  According to a second aspect of the present invention, there is provided an information analysis method for outputting information according to a result of analyzing image data, the image detection step for detecting an image related to the user's body from the image data, and the image detection step. A first analysis step for performing a first predetermined analysis process based on the detected image relating to the user's body, a comparison step for comparing an analysis result of the first analysis step with a threshold value, and an image And a second analysis step for performing a second predetermined analysis process based on the image related to the user's body detected in the detection step, and based on the comparison result in the comparison step, It is an information analysis method characterized by determining whether or not to perform a second predetermined analysis process in steps.

  According to a third aspect of the present invention, there is provided an information analysis program for causing a computer to execute an information analysis method for outputting information corresponding to a result of analyzing image data. The information analysis method includes: An image detection step to detect, a first analysis step for performing a first predetermined analysis process based on an image related to the user's body detected by the image detection step, and an analysis by the first analysis step A comparison step for comparing the result with a threshold value, and a second analysis step for performing a second predetermined analysis process based on an image related to the user's body detected by the image detection step. And determining whether or not to perform the second predetermined analysis process in the second analysis step based on the comparison result in the step of It is a lamb.

  According to a fourth aspect of the present invention, there is provided a computer-readable recording medium on which an information analysis program for causing a computer to execute an information analysis method for outputting information corresponding to a result of analyzing image data is recorded. A difference between an image detection step for detecting an image related to the body, an image related to the user's body detected by the image detection step, and a value based on a reference image corresponding to the image related to the user's body; A first analysis step for performing a first predetermined analysis process, a comparison step for comparing the analysis result of the first analysis step with a threshold value, and a user's body detected by the image detection step. A second analysis step for performing a second predetermined analysis process based on the related image, and based on the comparison result of the comparison step An information analysis program for causing a computer to execute an information analysis method for determining whether or not to perform the second predetermined analysis process in the second analysis step, and a reference image and / or a value based on the reference image A computer-readable recording medium characterized by being recorded.

  According to a fifth aspect of the present invention, there is provided an information analysis system that outputs information according to a result of analyzing image data, an imaging device that images a subject and outputs image data, and an image related to a user's body from the image data. An image detection unit that detects the image, a first analysis unit that performs a first predetermined analysis process based on an image related to the user's body detected by the image detection unit, an analysis result and a threshold value by the first analysis unit And a second analysis unit that performs a second predetermined analysis process based on an image related to the user's body detected by the image detection unit, and based on a comparison result by the comparison unit And an information analysis device configured to determine whether or not to perform the second predetermined analysis processing by the second analysis unit, and image data output from the imaging device is transmitted to the information analysis device via the network Sent to Image detection unit is an information analysis system being characterized in that to detect the images associated from the image data transmitted via the network to the user's body.

  As described above, the first, second, and third inventions detect an image related to the user's body from the image data, and the first predetermined analysis process based on the detected image related to the user's body. And determining whether or not to perform the second predetermined analysis process based on the image related to the user's body based on the result of comparing the first analysis process result and the threshold value. Specific conclusions about the body can be drawn from the data to the user.

  The recording medium according to the fourth aspect of the invention detects an image related to the user's body from the image data, and detects an image related to the detected user's body and a reference image corresponding to the image related to the user's body. A first predetermined analysis process is performed based on the difference from the value based on the result, and a second predetermined analysis based on an image related to the user's body is performed based on a result of comparing the analysis result of the first analysis process with a threshold value. Since an information analysis program for determining whether to perform processing and a reference image and / or a value based on the reference image are recorded, the information analysis program recorded on the recording medium, the reference image and / or the reference image are recorded. By reading the value based on the computer and executing the program, the user can draw a specific conclusion when comparing the reference image with the image data related to the body. It can be.

  According to a fifth aspect of the present invention, there is provided an imaging device that captures an image of a subject and outputs image data, an image related to the user's body is detected from the image data, and the first is based on the detected image related to the user's body. To determine whether or not to perform the second predetermined analysis processing based on the image related to the user's body based on the result of comparing the first analysis processing result and the threshold value. The information analysis device is configured to detect an image related to the user's body based on the image data output from the imaging device and transmitted via the network. Any location where an imaging device connected to the analysis device via a network is installed, the information analysis device is instructed by the user from the image data captured by the imaging device to the user. It is possible to derive a conclusion.

  As described above, the first, second, and third inventions detect an image related to the user's body from the image data, and the first predetermined analysis process based on the detected image related to the user's body And determining whether or not to perform the second predetermined analysis process based on the image related to the user's body based on the result of comparing the first analysis process result and the threshold value. This has the effect of allowing the user to draw specific conclusions about the body from the data.

  In addition, as described above, the fourth invention detects an image related to the user's body from the image data, and a reference corresponding to the detected image related to the user's body and the image related to the user's body. A first predetermined analysis process is performed based on a difference from a value based on the image, and a second predetermined process based on an image related to the user's body is performed based on a result of comparing the analysis result of the first analysis process with a threshold value. An information analysis program for determining whether or not to perform the analysis process and a reference image and / or a value based on the reference image are recorded, so that the information analysis program recorded on the recording medium, the reference image and / or the reference By reading the values based on the image on the computer and executing the program, the user can draw a specific conclusion when comparing the reference image with the image data related to the body. There is an effect that can.

  In addition, as described above, the fifth aspect of the invention relates to an imaging device that images a subject and outputs image data, and detects an image related to the user's body from the image data, and relates to the detected user's body. Whether to perform the first predetermined analysis process based on the image, and to perform the second predetermined analysis process based on the image related to the user's body based on the result of comparing the first analysis process result and the threshold value The information analysis device is configured to detect an image related to the user's body based on image data output from the imaging device and transmitted via the network. The user can obtain information from the image data captured by the imaging device from any location where the imaging device connected to the information analysis device via the network is installed. There is an effect that can be derived the specific conclusions about the body.

Hereinafter, embodiments of the present invention will be described in the following order.
1. Invention concept 2. First Embodiment of the Invention 2-1-1. About the system 2-1-2. Schematic processing flow according to first embodiment 2-1-3. Supplement 2-2. First application example of first embodiment 2-2-1. Overview of system 2-2-2. System applicable to the first application example 2-2-3. Example of analysis processing according to first application example 2-2-4. Supplement 2-3. Second application example of first embodiment 2-3-1. Overview of system 2-3-2. System applicable to the second application example 2-3-3. Example of analysis processing according to second application example 2-3-4. Supplement 2-4. Third application example of first embodiment 2-4-1. Overview of system 2-4-2. System applicable to the third application example 2-4-3. 2. An example of analysis processing according to the third application example First Modification of First Embodiment of the Invention 3-1-1. About system 3-1-2. Example of user identification method 3-1-3. Schematic processing flow 3-2. Application example to first modification of first embodiment 3-3-1. System applicable to application examples 3-3-2. Schematic processing flow 3-3-3. 3. Application of the first embodiment to the second and third application examples Regarding the second modification of the first embodiment 4-1-1. About the system 4-1-2. Specification of data based on history information 4-1-3. Outline of processing flow 4-2. Application example to second modification of first embodiment 4-2-1. System applicable to application examples 4-2-2. Schematic processing flow 4-2-3. 4. Application of the first embodiment to the second and third application examples Second embodiment of the invention 5-1. Overview of system 5-2. Schematic processing flow 5-3. Application example of second embodiment 5-3-1. First application example (example applied to makeup support system)
5-3-2. Second application example (example applied to a sports learning system)
5-3-3. Third application example (example applied to a learning system for foreign language conversation)
6). Third Embodiment of the Invention 6-1. Outline 6-2. Applicable system 6-3. Schematic processing flow

  It should be noted that the following description is disclosed to the extent that the present invention can be implemented, and the present invention is not limited to the following embodiments and application examples, as long as they do not depart from the gist of the present invention. Various modifications and applications are possible.

1. Concept of the Invention First, the concept of the present invention will be described. In this invention, the information of the target user is acquired by a sensor device arranged in a predetermined manner, and the acquired information is analyzed (hereinafter referred to as primary analysis). A threshold is determined based on the analysis result, and it is determined whether or not to further perform secondary analysis (hereinafter referred to as secondary analysis) on the information according to the determination result. When it is determined to perform the secondary analysis, a notification based on the secondary analysis result is transmitted to the registered notification destination. The secondary analysis process can be performed according to the state of the user at the time when information is acquired by the sensor device, and the user can automatically receive the result of the secondary analysis process.

2. First Embodiment of the Invention 2-1-1. About System A first embodiment of the present invention will be described. FIG. 1 schematically shows an example of a system configuration according to the first embodiment of the present invention. Sensor devices 11A, 11B, 11C,... Are respectively arranged for predetermined facilities 10A, 10B, 10C,. Similarly, readers 12A, 12B, 12C,... Are arranged corresponding to these sensor devices 11A, 11B, 11C,. These sensor devices 11A, 11B, 11C,... And readers 12A, 12B, 12C,. A server system 14 is further connected to the network 13.

  The sensor devices 11A, 11B, 11C,... Are, for example, camera devices that have an image sensor and are configured to capture still images and / or moving images. The sensor devices 11A, 11B, 11C,... Are identification information for identifying the still image data and / or moving image data obtained by photographing the subject, respectively, from the sensor devices 11A, 11B, 11C,. Are output in association with the sensor ID and time information indicating the time when the image data was captured. The still image data and / or moving image data output from the sensor devices 11A, 11B, 11C,... And the sensor ID are transmitted to the server system 14 via the network 13.

  In the following description, unless otherwise required to distinguish between still image data and moving image data, they are described uniformly as image data as appropriate.

  The readers 12A, 12B, 12C,... Read user IDs, which are identification information for identifying users, stored in the user medium 15 that is a predetermined storage medium. The read user ID is transmitted to the server system 14 via the network 13.

  The network 13 may be a closed network in this system, or may be an open network such as the Internet. When an open network is used as the network 13, the data exchanged between the sensor devices 11A, 11B, 11C,..., The readers 12A, 12B, 12C,. Is preferably encrypted by a predetermined encryption method.

  The server system 14 includes, for example, a server unit 20, an analysis system unit 21, a sensor information database 22, and a user information database 23. The server unit 20 controls and manages the entire system. For example, the server unit 20 communicates with each sensor device 11A, 11B, 11C,... And each reader 12A, 12B, 12C,... Via the network 13, and each sensor device 11A, 11B, 11C,. ... Receives data transmitted from each reader 12A, 12B, 12C,. The received data is stored in the sensor information database 22 and the user information database 23 in a predetermined manner. For example, although details will be described later, image data among the received data is stored in the sensor information database 22 in association with predetermined identification information.

  The analysis system unit 21 performs predetermined analysis processing (primary analysis) on the image data acquired by the sensor devices 11A, 11B, 11C,... And stored in the sensor information database 22, and results of primary analysis If it is determined that the image data is necessary, secondary analysis processing is further performed on the image data.

  The server system 14 is connected to the Internet 30 by the server unit 20. For example, user terminals 31A, 31B, 31C,... Of registered users are connected to the Internet 30. The user terminals 31A, 31B, 31C,... Are not particularly limited as long as they can communicate with the Internet 30. For example, a personal computer or a portable terminal capable of communicating with the Internet 30 using radio waves. Software required for using the Internet 30, such as a mailer for sending and receiving e-mails and a browser for browsing web pages of websites, is installed. The server unit 20 can communicate with each of user terminals 31A, 31B, 31C,... Of registered users connected to the Internet 30 via the Internet 30.

  The sensor information database 22 and the user information database 23 included in the server system 14 will be schematically described. The sensor information database 22 stores image data from each of the sensor devices 11A, 11B, 11C,... In association with predetermined identification information. FIG. 2 shows an exemplary configuration of a sensor information table stored in the sensor information database 22. 2 is an example in which a sensor information table is provided for each of the sensor devices 11A, 11B, 11C,. Each sensor information table is identified by a sensor ID corresponding to each sensor device 11A, 11B, 11C,.

  In the sensor information table, image data is identified by, for example, a unique data ID, and is registered in association with time information and a user ID. For example, the image data is actually stored in a predetermined storage device (not shown) included in the server unit 20, and a pointer indicating the storage location of the image data is registered in the sensor information table. The time information indicates, for example, the time when the image data is captured by the sensor device (for example, the sensor device 11A). The user ID is read from the user medium 15 by a reader (for example, reader 12A) corresponding to the sensor device and transmitted to the server system 14, for example.

  As an example of a method for associating image data with time information and a user ID, a method of interlocking sensor devices 11A, 11B, 11C,... Corresponding to readers 12A, 12B, 12C,. . That is, for example, in the sensor device 11A and the reader 12A corresponding to each other, the user ID read from the user medium 15 by the reader 12A is held in a predetermined manner. Then, when the image data is output from the sensor device 11 </ b> A, the held user ID is associated with the image data and transmitted to the server system 14.

  In addition to this, in each of the sensor devices 11A, 11B, 11C,..., Time information indicating the time at which image data was captured is transmitted in association with the image data, and the readers 12A, 12B, 12C,. In this case, when transmitting the user ID read from the user medium 15, it is also conceivable to transmit time information indicating the time of reading from the user medium 15 in association with the user ID. In the server system 14, the server unit 20 can associate the image data with the user ID based on the time information.

  The user information database 23 has, for example, a user information table and a reference information table, and stores information related to registered users. FIG. 3 shows an example of the configuration of the user information table. In the user information table, the attribute information of the user is associated with the user ID. The attribute information includes, for example, the user's name and contact information. The contact information indicating the user's contact information includes an email address if the system is adapted to distribute data over the Internet. Each user information can be distinguished from each other by a user ID. The information in the user information table is registered in advance, for example, when the user is registered in the system.

  FIG. 4 shows an example of the configuration of the reference information table. The reference information table serves as a reference for determining whether or not to perform secondary analysis processing on image data acquired by the sensor devices 11A, 11B, 11C,... And transmitted to the server system 14. However, each registered user is registered in association with the user ID. Although details will be described later, in the primary analysis process, the reference information is formed as data in the same format as the process result because the image data is compared with the process result obtained by the analysis system unit 21 processing the image data. The information in the reference information table is registered in advance, for example, when the user registers in the system. Not limited to this, depending on the purpose and configuration of the system, it may be registered in the reference information table at a predetermined timing.

  What kind of information is used as the reference information varies depending on the purpose of the system. In this example in which the sensor devices 11A, 11B, 11C,... Are camera devices and the photographed still image and / or moving image data is a processing target, the user corresponding to the user ID has a certain state. Information based on image data obtained by photographing a face can be used as reference information. That is, a face part is detected from image data obtained by photographing a face in a specific state of the user, and feature information of the detected face image is extracted and digitized. A value obtained by digitizing the facial feature information can be used as reference information.

2-1-2. About Schematic Process Flow According to First Embodiment FIG. 5 is a flowchart showing an example of a process according to the first embodiment by the system as described above. Registration of data for the user information table and the reference information table in the user information database 23 is performed in advance prior to the processing of the flowchart of FIG. 5, and the user stores the user medium 15 in which the user ID is stored in advance. Assume that you have it.

  In the first step S10, for example, in the facility 10A, for example, the user performs processing for causing the reader 12A to read the user ID stored in the user medium 15, and the sensor device 11A captures the user as a subject. The server system 14 is connected via the network 13 together with the user ID read by the reader 12A by adding the time information indicating the shooting time to the image data obtained by the shooting and the sensor ID corresponding to the sensor device 11A. Received. Here, it is assumed that the image data output from the sensor device 11A is still image data.

  The server system 14 stores the received user ID, image data, and time information in the sensor information database 22 in a predetermined manner, and supplies the image data to the analysis system unit 21 to perform a predetermined process (step S11). In the next step S12, a primary analysis is performed based on the result of the predetermined processing of the image data. For example, corresponding reference information is extracted from the reference information table in the user information database 23 based on the received user ID, and the extracted reference information is compared with the processing result in step S11.

  Based on the result of the comparison in step S12, a threshold value is determined in step S13. For example, in step S12, a difference between the reference information and the processing result in step S11 is calculated in advance, and it is determined in step S13 whether or not this difference has exceeded a threshold value.

  If it is determined in step S13 that the difference has exceeded the threshold, the process proceeds to step S14. That is, when the difference exceeds the threshold, it can be considered that the predetermined state of the current user is significantly different from the state indicated by the reference information, and corresponding processing is performed in step S14 and subsequent steps.

  In step S14, the analysis system unit 21 further performs an analysis process (secondary analysis process). For example, in step S <b> 14, an analysis that can extract a cause or a factor that greatly differs between a predetermined state of the current user and a state indicated by the reference information is performed as a secondary analysis. As an example, as the secondary analysis, it is conceivable to perform the analysis performed in step S11 in more detail. In addition, as a secondary analysis, it may be possible to select another analysis method based on the analysis result of the analysis performed in step S11.

  In the next step S15, a notification based on the result of the secondary analysis process in step S14 is transmitted to the corresponding user or the like. That is, in the server system 14, the user information table in the user information database 23 is referred to based on the received user ID, contact information is extracted from the attribute information of the user ID, and the contact indicated in the contact information is shown. A notification is sent. If the contact is an e-mail address, the user corresponding to the user ID can access the notification using the user terminal 31A, for example.

  Note that the content of the notification transmitted to the user in step S15 varies depending on the purpose and configuration of the system. For example, a notification of information indicating a cause or a factor that greatly differs between a predetermined state of the current user and a state indicated by reference information, a notification of information for correcting the cause or the factor, a warning notification, or the like can be considered.

  On the other hand, if it is determined in step S13 that the difference does not exceed the threshold value, the process proceeds to step S16. In this case, it can be considered that the predetermined state of the current user and the state indicated by the reference information are not so different as to cause a problem. Therefore, in step S16, it can be considered that no particular processing is performed. In step S16, it may be possible to notify the corresponding user or the like that there is no problem.

  In the user information database 23, by associating the user ID of another user with the user ID, not only the notification in step S15 or step S16 is transmitted to the target user, but also the user of the target user. It can also be transmitted to a user corresponding to another user ID associated with the ID.

2-1-3. Supplement: In the above description, the sensor devices 11A, 11B, 11C,... Are the camera devices that output still image data and / or moving image data, but the sensor devices 11A, 11B, 11C,. If information capable of detecting the predetermined state can be acquired, various types of apparatuses can be applied. For example, even when the camera device is applied as the sensor devices 11A, 11B, 11C,..., The light is not limited to the light in the visible light region, but in the infrared region, the ultraviolet region, and the X-ray region. It is also possible to use an apparatus that uses light as a subject of photographing.

  The sensor devices 11A, 11B, 11C,... Are not limited to devices that output images, such as camera devices and video camera devices. For example, an audio device that outputs sound collected using a microphone can also be used as the sensor devices 11A, 11B, 11C,.

  Further, for example, it is conceivable to use, as the sensor devices 11A, 11B, 11C,..., A receiving device that receives and demodulates a radio wave modulated with a digital signal including a predetermined code. In this case, the user who receives the service is allowed to have a transmitter that modulates and transmits a unique code for each user.

  Various storage media can be used as the user medium 15. For example, a card-type storage medium in which an IC (Integrated Circuit) chip, a non-volatile memory, and an antenna are built in a synthetic resin card and self-generated by radio waves emitted from the reader 14 is operated. 15 can be applied. In addition, an information device that can be carried by the user and can transmit and receive data can be applied as the user medium 15. As such information equipment, a cellular phone terminal or a PDA (Personal Digital Assistant) capable of transmitting and receiving data can be considered.

  Furthermore, in recent years, products that have an IC chip, an antenna mechanism, and a non-volatile memory, in addition to the telephone communication function, and a function equivalent to that of a card-type storage medium have appeared. . It is also possible to use the functional part corresponding to the card type storage medium of such a mobile phone terminal as the user medium 15.

  Further, the data communication between the readers 12A, 12B, 12C,... And the user medium 15 is not limited to wireless communication, and may be performed using electrical contact via a wire, for example, a connector. . In this case, the user medium 15 can be composed of a non-volatile memory and an interface, and so-called USB (Universal Serial Bus) memory, compact flash memory (compact flash is a registered trademark), smart media, etc. can be applied. .

  Furthermore, the user medium 15 may be a rewritable recording medium such as a magnetic disk, a magnetic card, a magneto-optical disk, or an optical disk. In this case, the readers 12A, 12B, 12C,... Have a corresponding drive device.

  Further, the user medium 15 may record data with physical deformation. For example, it is conceivable to make a hole in paper or a plastic plate to record data. If only the user ID and date / time information is recorded, such a recording method can sufficiently cope. In this case, the readers 12A, 12B, 12C,... Read data using an optical method or electrical contact.

  A print medium can be used as the user medium 15. For example, the data is recorded by converting the data into a predetermined code such as a barcode or a QR code (Quick Response Code) and printing it on paper or other predetermined print medium. The readers 12A, 12B, 12C,... Include, for example, a reading device that reads printed information using light.

2-2. First Application Example of First Embodiment Next, a first application example of the first embodiment of the present invention will be described. The first application example is an example in which the first embodiment is applied to a makeup support system that advises a makeup method.

2-2-1. Outline of system The makeup support system will be schematically described. FIG. 6 is a flowchart showing an exemplary flow for explaining the outline of the makeup support system. First, makeup (referred to as ideal makeup) considered to be optimal is applied to the user by the hands of a makeup artist or the like (step S20). Then, the face with the ideal makeup is photographed in a predetermined manner, and image data obtained by photographing is transmitted to the server system (step S21). On the server system side, based on the transmitted image data of the ideal makeup, the facial feature information is extracted by specializing in features suitable for judging the makeup state, and the extracted facial feature information is It is registered in the user information database as user reference information (step S22). At this time, information indicating the cosmetic used for applying the ideal makeup may be registered in a predetermined database in association with the reference information.

  On the other hand, the user later performs makeup (referred to as self-makeup) himself at home, for example (step S23), and photographs the face on which self-makeup has been performed. Then, this self-makeup image data obtained by photographing is transmitted to the server system (step S24). On the server system side, the image data transmitted from the user is subjected to predetermined processing to extract feature information specialized for the facial feature indicating the makeup state, and a primary analysis is performed on the extracted feature information. Process. The primary analysis process is performed, for example, by comparing with feature information registered as reference information of the user (step S25). As a result of the comparison by the primary analysis, if the difference between the two is determined to be greater than or equal to a predetermined value (step S26), it can be considered that there is some problem in the user's makeup method. Therefore, the server system side performs secondary analysis processing on the image data transmitted from the user and / or the analysis result obtained by analyzing the image data so that the problem part can be extracted (step S27).

  The server system side creates advice information for giving advice to the makeup method based on the result of the secondary analysis processing. The created advice information is transmitted to the user (step S28). At this time, corresponding to the advice information, information indicating recommended cosmetics or the like can be added to the advice information and transmitted. While staying at home, the user can obtain a method for bringing his / her make-up closer to the ideal make-up, and can obtain information such as necessary cosmetics.

  If it is determined in step S26 that the difference is less than the predetermined value, a notification indicating that there is no problem in the current makeup method is transmitted from the server system side to the user (step S29). At this time, it is conceivable that the purchase information of cosmetics used when applying the ideal makeup, which is registered in the database in association with the reference information, is added to the notification and transmitted.

2-2-2. About System Applicable to First Application Example FIG. 7 schematically illustrates an example system applicable to the first application example of the first embodiment. The system applicable to the first application example illustrated in FIG. 7 basically has the same configuration as the system applicable to the first embodiment described above with reference to FIG. A camera device 101A and a reader 102A are disposed in the facility 100A and connected to the network 103. The facility 100A is a place where a makeup professional such as a makeup artist applies makeup to a user, such as a makeup class hosted by a cosmetic corner of a department store or a cosmetic manufacturer. Similarly, the camera device 101 </ b> B and the reader 102 </ b> B are disposed, for example, in the user's home 100 </ b> B and connected to the network 103.

  The camera device 101A and the camera device 101B capture a subject and output still image data. In addition to the still image data, the camera devices 101A and 101B include, in the still image data, time information indicating the time when the still image data was captured and a sensor ID for identifying each of the camera devices 101A and 101B. Output in association. Note that a video camera device that outputs moving image data may be further provided for the facility 100A. The camera apparatus 101A outputs moving image data, and still image data may be extracted from the moving image data.

  The readers 102A and 102B read the user ID stored in the user medium 15 (not shown). As described above, various types of user media 15 can be applied. In the first application example, the user media 15 is a non-volatile memory, transmission / reception in the above-described synthetic resin card. And a card-type storage medium having a power generation unit. The user medium 15 is assumed to have a user ID stored in advance and given to the corresponding user.

  Here, it is assumed that the image data output from the camera device 101A and the user ID read by the reader 102A are associated with each other and transmitted to the network 103. For example, the user ID read from the user medium 15 is held in the reader 102A until the user medium 15 is next read by the reader 102A or the user ID held by the reader 102A is erased. It is possible. A user ID held in the reader 102A is associated with image data output from the camera apparatus 101A in a predetermined manner. The same applies to the camera device 101B and the reader 102B.

  In the first application example, the user ID is not limited to being read from the user medium 15 by the reader 102A. For example, the reader 102A may be an input device provided with predetermined input means such as a keyboard, and the user may input the user ID manually.

  The network 103 may be a closed network in this system, or may be an open network such as the Internet. When an open network is used as the network 103, data exchanged between the camera apparatuses 101A and 101B, the readers 102A and 102B, and the server system 104 is encrypted using a predetermined encryption method. It is desirable to do.

  In FIG. 7, the facility where the camera device and the reader connected to the network 103 are arranged is only the facility 100A and the user's home 100B. The provided camera device and reader are connected to the network 103 so that they can be identified.

  The server system 104 corresponds to the server system 14 described with reference to FIG. 1, and includes, for example, a server unit 110, an analysis system unit 111, a sensor information database 112, and a user information database 113. This first application example Then, it has the cosmetics database 114 further. The server unit 110 performs overall control and management of the system. For example, the server unit 110 communicates with the camera device 101A and the reader 102A via the network 103, receives image data and a user ID transmitted in association with each other from the camera device 101A and the reader 102A, and receives a sensor information database. 112 is stored in a predetermined manner. The same applies to data output from the camera device 101B and the reader 102B.

  The analysis system unit 111 performs primary analysis processing as described below on an example of image data acquired by the camera apparatuses 101A and 101B and stored in the sensor information database 112. Further, threshold determination is performed based on the analysis result of the primary analysis, and secondary analysis as described below is performed according to the determination result.

  The server system 104 is connected to the Internet 130 by the server unit 110. For example, a user terminal 131 such as a personal computer that can be used by the user at home 100B or the like is connected to the Internet 130. In FIG. 7, only the user terminal 131 is connected to the Internet 130, but in reality, a larger number of user terminals can be connected.

  Further connected to the Internet 130 is a web server 132 which is a server for providing web services such as web page publishing to the Internet 130. The server system 104 can communicate with the web server 132 via the Internet 130 and upload files to the web server 132.

  In the above description, the camera device 101B disposed in the user's home 100B has been described as being connected to the network 103, but this is not limited to this example. For example, it is assumed that the camera apparatus 101B is connected to a user terminal 131 that is a personal computer, and the user terminal 131 controls shooting by the camera apparatus 101B, and video data output from the camera apparatus 101B is transmitted from the user terminal 131 to the Internet. You may make it transmit to the server system 104 via 130. FIG.

  The configurations of the sensor information database 112 and the user information database 113 will be schematically described. The configuration of the sensor information database 112 is basically the same as the sensor information database 22 described with reference to FIG. In addition, when a video camera device is provided in the facility 100A and a moving image is shot, the video camera device compresses and encodes the moving image data to the network 103, and the server unit 110 The video data supplied after being compressed and encoded is stored in a predetermined manner. Then, the data ID for identifying the data and the information indicating the storage location of the actual data of the compressed and encoded video data are associated with each other and registered in the sensor information table in the sensor information database 112.

  The user information database 113 includes a user information table in which user information is registered and a reference information table. FIG. 8 shows an example of the configuration of the user information table in the user information database 113. In the user information table, for each user ID for identifying a user, user attribute information corresponding to the user ID and information regarding the user's makeup are registered. The user attribute information includes information representing the user, such as the user's name, age, and contact information (e-mail address). The makeup information is information related to the user's makeup, and includes skin type (dry skin or oily, fine or rough texture, background color, etc.), makeup preference (whelming overall, overall , Etc.), and the information such as the product number of the cosmetics that are usually used. These pieces of information are set when the user starts using the system, for example. The configuration of the reference information table will be described later.

  In the system configured as described above, according to the flow described with reference to FIG. 6, first, the user asks a makeup artist or the like to make up makeup (ideal makeup) at the facility 100A (step S20). Further, the user causes the reader 102A to read the user ID stored in the user medium 15, and causes the camera device 101A to photograph the face on which the ideal makeup has been applied. The face is photographed from a plurality of directions such as the front, side, and top of the face so that a stereoscopic image can be formed. The makeup can be captured with a video camera and output as moving image data. In addition, an image is taken so that a reference subject such as color, shading, and brightness is included in the same image data as the user's face so that image correction can be performed later. A plurality of pieces of image data obtained by photographing are added with time information indicating the time of photographing, a sensor ID for identifying the camera device 101A, and a user ID, and transmitted to the server system 104 via the network 103. (Step S21).

  The server system 104 supplies the image data, the time information added to the image data, the sensor ID, and the user ID to the server unit 110. The server unit 110 stores image data, time information, and a user ID in association with each other in a sensor information table corresponding to the sensor ID in the sensor information database 112. The image data stored in the sensor information database 112 is read by the server unit 20 based on the user ID and the like, and is supplied to the analysis system unit 111. The analysis system unit 111 performs a predetermined process on the supplied image data, and generates reference information corresponding to the image data based on the processing result. The reference information is added with a reference information ID for identifying the reference information, is associated with other related information, and is registered in the reference information table in the user information database 113 (step S22).

  The user later performs makeup (self-makeup) himself following the ideal makeup, for example, at the user's home 100B (step S23). In addition, the user causes the reader 102B to read the user ID stored in the user medium 15. Then, the user photographs the face on which self-makeup has been performed with the camera device 101B. On the user side, it is not necessary to shoot so that a three-dimensional image can be configured, and for example, shooting is performed only from the front of the face. Also at this time, an image is taken so that the subject, which is a reference such as color, shading, and brightness, is included in the same image data as the user's face so that image correction can be performed later. A plurality of pieces of image data obtained by photographing are added with time information indicating the time of photographing, a sensor ID for identifying the camera device 101A, and a user ID, and transmitted to the server system 104 via the network 103. (Step S24).

  The server system 104 supplies the image data by self-make and the time information, sensor ID, and user ID added to the image data to the server unit 110. The server unit 110 stores image data, time information, and a user ID in association with each other in a sensor information table corresponding to the sensor ID in the sensor information database 112. The image data stored in the sensor information database 112 is read by the server unit 20 based on the user ID and the like, and is supplied to the analysis system unit 111. The analysis system unit 111 performs a predetermined process on the supplied image data. The server unit 110 extracts the corresponding reference information from the reference information table in the user information database 113 based on the supplied user ID, compares the reference information with the processing result based on the image data by self-make, Analysis is performed (step S25).

  As a result of the primary analysis, if the difference between the reference information and the processing result based on the self-make image data is greater than or equal to a predetermined value (step S26), the server unit 110 performs the secondary analysis process based on the self-make image data ( Step S27). Then, advice information is created based on the secondary analysis result and transmitted to the user. For example, the server unit 110 searches the user information table in the user information database 113 based on the user ID, acquires an email address associated with the user ID, and advises the acquired email address. Send an e-mail with information written or attached.

  The user receives the e-mail attached with the advice information transmitted from the server unit 110 by using the user terminal 131 at the user's home 100B, for example, so that the user can make self-make closer to the ideal makeup based on the advice information. Can be done.

  Here, it is conceivable to transmit image data corresponding to the reference information of the user together with the advice information to the user. That is, the advice information and the image data are packaged together, and the package data is attached to an electronic mail and transmitted. The user receives this e-mail at the user terminal 131 and refers to the advice information while displaying the image data, so that the user can make up according to the advice information while viewing the image by the ideal makeup. Can do self-makeup, close to ideal makeup.

  Further, the server unit 110 can package an advertisement according to the advice information with the advice information and image data based on the ideal makeup and transmit the packaged information to the user. For example, it is conceivable that the server unit 110 searches the cosmetic database 114 for cosmetics that are considered to have a difference obtained as a result of the primary analysis equal to or less than a predetermined value based on the secondary analysis result. Not limited to this, the cosmetic database 114 may be searched for information on cosmetics used in the ideal makeup in step S20. The server unit 110 packages advertisements based on cosmetic information retrieved from the cosmetic database 114, advice information, and image data based on ideal makeup together, and transmits the packaged data to the user. According to this, the user can immediately obtain information on cosmetics suitable for him.

  FIG. 9 shows an exemplary configuration of package data. The package data 140 is, for example, a collection of metadata 141, individual data 142, 142,... And additional information 143 corresponding to each other. As an example, each of these data is configured as a file and archived. One file can be considered. It is also conceivable that these files are stored together in one folder and the folder itself is regarded as package data 140.

  The metadata 141 is an attribute of the packaged package data 140 itself, and includes information that collectively indicates the contents of the packaged data. For example, it is conceivable that ID information for identifying the facility 100A where the user has made an ideal makeup is stored as the metadata 141. As the individual data 142, 142,..., For example, image data, advice information, and advertisement information corresponding to the reference information are stored. The additional information 143 can store predetermined additional information. For example, the additional information 143 includes the date and time when the user performed ideal makeup, the name of the makeup artist in charge of makeup, the name of the facility 100A that performed makeup, and the like. Can be stored as

  In the above description, advice information, image data by ideal makeup, and advertisement information are stored in the package data 140, but this is not limited to this example. When the makeup 100A is photographed as moving image data at the facility 100A, the moving image data can be further stored in the package data 140. The server unit 110 extracts predetermined moving image data from the sensor information database 112, packages it as packaged data 140 and stores it in the package data 140.

  Note that moving image data included as individual data in the package data 140 is preferably subjected to so-called streaming distribution, which reduces the burden of browsing at the distribution destination. In streaming distribution, compressed moving image data and audio data are distributed in predetermined units, and the receiving side accumulates and decompresses the data in the buffer, receives the subsequent data and stores it in the buffer. The stream data such as can be played while being downloaded.

2-2-3. Example of Analysis Processing According to First Application Example Next, examples of primary analysis processing and secondary analysis processing applicable to the first application example of the first embodiment will be described. First, an example of the primary analysis process will be described. In the primary analysis process, for example, a plurality of predetermined points on the face are set as measurement points, and predetermined measurement is performed at each measurement point on a face subjected to ideal makeup and a face subjected to self-makeup. A difference is calculated by comparing the value measured for the face with makeup and the value measured for the face with self-makeup. It is conceivable to compare this difference with a predetermined threshold.

  An example of measurement points will be described with reference to FIG. For example, the server system 104 sets measurement points based on image data that was captured when the user applied ideal makeup at the facility 100A. First, based on image data obtained by photographing the face 150 from the front, the face outline 152 is specified, and the hair outline 151 is specified. Based on the specified face outline 152, the position of the measurement point inside the face 150 is specified.

  As measurement points inside the face 150, left and right eyebrows 153A and 153B, left and right eyes 154A and 154B, and a lip 155 can be considered. The positions and shapes of the eyebrows 153A and 153B, the eyes 154A and 154B, and the lips 155 are measured. Regarding the shape of the lip 155, it is conceivable to make a judgment based on a change in color by measuring the color. The shape of the lip 155 can also be determined based on the stereoscopic image. For the eyebrows 153A and 153B, the color is also measured. For eyes 154A and 154B, the color and shape of eye lines, wrinkles, etc. are measured in more detail.

  As measurement points inside the face 150, it may be possible to specify predetermined positions 156A to 156J to which attention is paid during makeup. As an example, center position 156A of the forehead, positions 156D and position 156G corresponding to the nose, left and right eyelid positions 156B and 156C, left and right cheekbone positions 156E and 156F, left and right cheek positions 156H and 156I, and jaws Position 156J is considered. For these positions 156A-156J, the position and color are measured.

  Of these positions 156A to 156J, cheekbone positions 156E and 156F, nose apex position 156G, chin position 156J, and the like are configured based on image data taken from a plurality of directions after ideal makeup. It is thought that it can be specified by using a 3D face image.

  Further, although not shown, as a measurement point inside the face 150, for example, a line connecting two predetermined points may be specified. A color gradation is measured by a line connecting the two predetermined points. It is conceivable that the predetermined two points for measuring the posterior foundation are set, for example, in the vicinity of the eyes or in the vicinity of the cheekbones or cheeks.

  Evaluation values are obtained for the image data based on the ideal makeup based on the measurement values at these measurement points. For shape data of eyebrows, eyes, lips, etc., a shape dictionary in which various shapes for these eyebrows, eyes, and lips are digitized and registered is prepared, and the measured values measured from the face and registered in the shape dictionary It can be considered that a difference is obtained by comparing the value and an evaluation value is obtained based on the difference. Regarding colors, it is conceivable to use the values of the three primary colors themselves obtained from the image data as evaluation values. Not limited to this, the values of the three primary colors may be converted into chromaticity coordinates (x, y, Y) defined by CIE (commission internationale de l'eclairage) and used as evaluation values. Regarding gradation, it is conceivable to measure the colors of a plurality of points on a line connecting two predetermined points.

  In this way, the position information of each measurement point obtained from the image data by ideal makeup and the evaluation value are registered in the reference information table as reference information. FIG. 11 shows a configuration example of the reference information table. Here, as illustrated in FIGS. 11A to 11D, the reference information table is composed of a plurality of tables. The configuration of the reference information table is not limited to this, and a plurality of tables may be combined into one, or further divided.

  FIG. 11A shows an example of a definition table that defines each measurement point. This definition table is provided for each reference information, for example, and stores the position information and the types of measurement points in association with the identification information PoID for identifying the measurement points. When a measurement point is defined using a plurality of position information, it is conceivable to list the plurality of position information. The type indicates the type of data measured at the measurement point, such as color and shape.

  FIG. 11B shows a configuration of an example of an evaluation value table in which evaluation values at each measurement point are stored for each reference information. The measurement value at each measurement point may be stored instead of the evaluation value. The evaluation value is obtained from the measured value during the analysis process.

  FIG. 11C shows a configuration of an example of an image data table in which image data in each reference information is stored. For each reference information, each of images obtained by photographing the user from a plurality of directions and a stereoscopic image formed from the plurality of images are stored. Actually, the image data itself is stored in a predetermined storage device (not shown) included in the server unit 110, and a pointer indicating the storage location of the image data is registered in the image data table.

  FIG. 11D shows a configuration of an example of an attribute information table in which attribute information is stored for each reference information. For example, for each reference information, it may be possible to register a corresponding user ID, an acquired date, an arbitrary description, a name of the facility 100A, a person in charge of the user at the facility 100A, and the like.

  The primary analysis may be performed as follows as an example. For the image data by self-make, the position of each measurement point is obtained based on the reference information, and at the position of each obtained measurement point, the same measurement as that performed when obtaining the reference information is performed. The evaluation value is obtained based on Then, for each measurement point, each evaluation value obtained for the reference information and each evaluation value obtained based on the image data by self-make is weighted according to the measurement point, and weighted. For the evaluation value, the absolute value of the difference is calculated for each corresponding set of evaluation values. The absolute values of the differences calculated for each set of evaluation values are added, and the addition result is compared with a predetermined threshold value. If it is determined that the addition result is larger than the threshold value (step S26 in FIG. 6), secondary analysis is performed.

  The secondary analysis may be performed as follows as an example. First, the difference for each set of evaluation values calculated at the time of the primary analysis is sorted in units of measurement points, and the contents of the differences are analyzed in descending order for the measurement points having a difference greater than or equal to a predetermined value. To do. The analysis is performed according to the type of measurement point and the measurement content. For example, in the case of a shape measurement value, that is, eyebrows, eyes (eyelids, eyelines), lips (boundary of lipstick), it is considered that there is a deviation in the position of the contour between ideal makeup and self makeup. Therefore, the magnitude and direction of the deviation are analyzed. In the case of a color measurement value, that is, a lip or a gradation between two predetermined positions 156A to 156J inside the face 150, the color difference is analyzed or in which direction the chromaticity coordinates are shifted. The

  Based on the secondary analysis result, advice information for giving makeup advice to the user may be generated and presented to the user. In the server system 104, for example, text data for advice is prepared in advance for each measurement point at which the secondary analysis is performed and for each analysis result for which the difference information of the evaluation value is predicted. Then, based on the actual secondary analysis result, the text data for advice is combined in a predetermined manner into one text data, which is presented as general makeup advice information for the user. The advice information is provided to the user by being included in the package data 140, for example.

  In the above description, the advice information and the like are described as being attached to the e-mail, but this is not limited to this example. For example, it is conceivable that advice information, image data based on ideal makeup, and advertisement information are presented to a user as a web page on a website so as to be viewable on the Internet 130.

  As an example, a storage device capable of defining a URL (Uniform Resource Locator) on the Internet 130 is provided for the server unit 110, and package data 140 in which advice information, image data by ideal makeup, and advertisement information are collected together is stored in this storage device. To store. In this case, the package data 140 is a folder in which, for example, advice information and image data by ideal makeup are stored. A predetermined URL is defined for the image data stored in the package data 140.

  The server unit 110 uses image data included in the package data 140 held in the storage device using a format that can be presented as a web page on the Internet 130 such as HTML (Hyper Text Markup Language) or XML (Extensible Markup Language). A file (hereinafter referred to as a web file) in which link data, layout information, text data based on advice information, link information based on advertisement information, and the like are described is created. The link information based on the advertisement information is, for example, link information to a web page that sells cosmetics indicated in the advertisement information. This web file is transferred to the web server 132 on the Internet 130.

  The web server 132 defines a URL for the web file and allows browsing on the Internet 130. The user can browse the image data, advice information, and advertisement information included in the package data 140 by accessing the URL using the user terminal 131 connected to the Internet 130.

  As an example, the server unit 110 transfers the web file to the web server 132 and notifies the user that the web file can be viewed on the Internet 130. More specifically, the server unit 110 searches the user information table in the user information database 113 and acquires the user's e-mail address. Then, the server unit 110 creates an e-mail in which the URL of the web file is described in a predetermined manner, and transmits it to the acquired e-mail address. The user who has received this e-mail accesses the URL based on the description of the e-mail.

  FIG. 12 shows a screen configuration of an example of a web page for browsing image data, advice information, and advertisement information included in the package data 140. The web page is generally displayed by software called a browser installed in the user terminal 131. For example, the browser accesses a file on the Internet 130 according to the input URL. If the accessed file is described in a predetermined format such as an HTML format, text data display, link to image data, layout of each display, and the like are performed according to the description of the file. Further, the browser can transmit data to the web server 132 by means designated in the read file.

  In addition to personal computers and the like, such browser software has been generally installed in mobile phone terminals and home game devices that can be connected to the Internet. In the following description, it is assumed that the browser is mounted on the personal computer. Each function of the browser can be controlled by a pointing device such as a mouse and a character input device such as a keyboard.

  Referring to FIG. 12, a web page display unit 302 for displaying a web page is provided on the browser screen 300 below the display unit 301 for performing control of the browser itself and control by an OS (Operating System). . On the web page display unit 302, image data by ideal makeup, text data as advice information, and advertisement information included in the package data 140 are arranged and displayed in a predetermined manner. In the example of FIG. 12, ideal makeup image data is displayed in the display area 310 in the web page display unit 302, text data is displayed in the display area 311, and advertisement information is displayed in the display area 312. The display areas 311 and 312 are each provided with a scroll bar at the right end, and when the data cannot be completely displayed in the area, the scroll bar is hidden in the area 311 and the area 312 by operating the scroll bar. The data can be displayed in the area.

  FIG. 13 shows another example of display by a web page. In the example of FIG. 13, a range 340 </ b> A to 340 </ b> G is predetermined in accordance with each measurement point in a region 330 where image data by ideal makeup is displayed on the browser screen 320 as illustrated by a dotted line in the drawing. Is set. For example, in the user terminal 131 on which the browser screen 320 is displayed, a pointing device such as a mouse is operated in a predetermined manner, and the cursor 341 on the screen is positioned in a desired range among the ranges 340A to 340G. On the other hand, advice information corresponding to the range is selectively displayed. Further, cosmetic advertisement information corresponding to the range is selectively displayed in the area 332.

  Furthermore, in the first application example of the first embodiment, cosmetics can be introduced to the user based on the result of the secondary analysis. For example, for a measurement point that is an analysis target in the secondary analysis, the cosmetic database 114 is referred to according to the position and measurement type, and a cosmetic that is determined to be appropriate based on the result of the secondary analysis is selected. The cosmetic information is included in the package data 140 and provided to the user. In the cosmetic database 114, for example, it is conceivable to classify cosmetics according to types or parts of the face used mainly, and register color information, price information, cosmetic product numbers, and the like.

  FIG. 14 shows an exemplary configuration of the cosmetic database 114. The type is, for example, a cosmetic product roughly classified according to its use, and is associated with a part in the face. Although the corresponding part has a specific name in FIG. 14, it is considered that it is actually preferable to indicate the corresponding part by a measurement point. For example, the identification information PoID of the measurement point described with reference to FIG. 11A is used. it can. Data is registered by color and product number for each type or corresponding part, and price information is associated with each. In the example of FIG. 14, the color information is stored based on a naming method by a cosmetic manufacturer. In this case, it is preferable to separately prepare a table for associating the color name based on the cosmetic manufacturer's naming method with the coordinates on the chromaticity diagram. The series is, for example, a product classification by a cosmetic manufacturer.

  As an example of a cosmetic information search method, it is assumed that the color of the lip by self-makeup (lip 155 in the example of FIG. 10) is too dark compared to the color by ideal makeup based on the secondary analysis result. In the case where “the color is darker than a certain color”, it is actually necessary to indicate the direction on the chromaticity diagram, for example. Here, in order to avoid complications, it is simply expressed as “dark and light”.

  In such a case, for example, referring to the cosmetic database 114, cosmetics corresponding to the color of the lip 155 in the reference information are extracted from the cosmetics (in this case, lipstick) corresponding to the part, and information on the cosmetics is extracted. Can be included in the package data 140 and provided to the user. At this time, it is more preferable that information indicating the method of purchasing the cosmetic is also provided to the user together with the cosmetic information. For example, the URL of a website where the cosmetic can be purchased via the Internet 130 is provided to the user together with the cosmetic information. The user can place an order for cosmetics immediately after receiving the secondary analysis result.

  In the above description, the cosmetic product is introduced when the difference between the feature information based on self-makeup and the feature information based on ideal makeup is greater than or equal to a predetermined value in step S26 in the flowchart of FIG. It is not limited. That is, even when it is determined in step S26 that these differences are less than a predetermined value, cosmetics can be introduced. In this case, for example, based on the used cosmetic information in the user information table in the user information database 113, other cosmetics having effects equivalent to or better than the cosmetics used by the user, or having the same effects and lower price. It is possible to introduce cosmetics.

2-2-4. Supplement Since makeup often uses subtle shades, in order to perform an accurate analysis in the primary and secondary analysis, the image data obtained by photographing the face with ideal makeup and the face by self-makeup are photographed. It is necessary to make the color appearance the same with the processed image data. At the same time, it is necessary to be able to compare the color information obtained by measuring the image data and the color information of each cosmetic registered in the cosmetic database 114 with a common display system.

  As an example, as described above, when photographing a face with ideal makeup and photographing a face with self-makeup, the reference object such as color, lightness and brightness is included in the same image data as the user's face Take a photo as shown. The reference subject includes, for example, a white portion for performing correction based on white balance and a color sample portion for performing color correction for each color. The reference subject is placed at a predetermined position in the screen and photographing is performed.

  When performing a primary analysis based on image data, or when image data obtained by the camera devices 101A and 101B is stored in the sensor information database 112, the image data is first arranged at a predetermined position in the screen. The reference subject described above is measured in a predetermined manner, and the color information of the white portion and the color information of the color sample portion are acquired based on the measurement result. Then, it is conceivable to normalize the image data obtained from the camera devices 101A and 101B based on the color information and the color information of the reference subject under the standard light source.

  In addition, a shadow portion may occur in a predetermined portion of the face in the image data depending on the unevenness in the three-dimensional shape of the face and the direction of the light source at the time of shooting, and this shadow portion affects the color measurement. This effect can be avoided by, for example, simulating a light source on a stereoscopic face image registered as reference information and predicting the occurrence of shadows. For example, for the image data by self-make, measure the part where shadows are predicted to occur such as around the nose, extract the part that appears to be shadowed, perform light source simulation based on this extracted shadow part information, Predict the direction of the light source. Based on the predicted light source direction, the occurrence of another shadow portion is predicted.

  As described above, according to the first application example of the first embodiment, the user can easily perform makeup that is closer to the ideal makeup while at home, and obtain necessary cosmetic information. And purchases can also be made on the spot.

2-3. Second Application Example of the First Embodiment Next, a second application example of the first embodiment of the present invention will be described. The second application example is an example in which the first embodiment is applied to a sports learning system that provides advice on body forms such as sports.

2-3-1. System Overview A sports learning system according to the second application example will be schematically described. FIG. 15 is a flowchart showing an exemplary flow for explaining the outline of this sports learning system. First, a form ideal for the user is photographed (step S30). For example, a trainer or an instructor (hereinafter referred to as an instructor) gives an instruction to the user to operate in the correct form, and a picture of the user operating according to the instruction is taken as a moving image. The form photographed here is called an ideal form. It is assumed that the user's operation at the time of ideal form shooting may be slower than the operation based on the actual form.

  The video data in which the ideal form is photographed is transmitted to the server system (step S31). On the server system side, feature information indicating the feature of the form is extracted based on the transmitted video data, and the extracted feature information of the form is registered in the user information database as user reference information (step S32).

  On the other hand, the user later performs a form reproduction operation at a training facility or at home, and shoots the state as a moving image (step S33). The form photographed here is called a self-form. And the video data which image | photographed the self-form is transmitted to a server system (step S34). On the server system side, the video data transmitted from the user is analyzed, the feature information indicating the feature of the form is extracted, and the extracted feature information and the feature information registered as the reference information of the user are included. Compare (step S35). As a result of the comparison, if the difference between the two is greater than or equal to a predetermined value (step S36), it can be considered that there is some problem in the user's form. Therefore, the server system performs secondary analysis processing on the video data transmitted from the user and / or the analysis result obtained by analyzing the video data so that the problem part can be extracted (step S37).

  The server system side creates advice information for giving advice of the form to the user based on the result of the secondary analysis processing. The created advice information is transmitted to the user (step S38). At this time, corresponding to the advice information, information indicating recommended sportswear and sports equipment can be added to the advice information and transmitted. The user can bring his / her own form closer to the ideal form through self-training and self-learning, and obtain necessary information.

  If it is determined in step S36 that the difference is less than the predetermined value, a notification indicating that there is no problem in the current form is transmitted from the server system side to the user (step S39). At this time, based on the information of the user registered in the user information database and the like, it is conceivable that purchase information of related products is added to the notification and transmitted.

2-3-2. System applicable to second application example FIG. 16 schematically shows an example system applicable to the second application example of the first embodiment. In the following description, the second application example will be described assuming that the first embodiment is applied to a golf learning system that advises a user of a golf swing form.

  The system applicable to the second application example illustrated in FIG. 16 is basically the system applicable to the first embodiment described above with reference to FIG. 1 and the implementation described above with reference to FIG. It has the same configuration as the system applicable to the first application example of the first form. In FIG. 16, the same reference numerals are given to the same parts as those in FIG. 7 described above, and detailed description thereof is omitted.

  The facility 160 is, for example, a place where an instructor is instructed to give a form to a user. In the example of FIG. 16, two video camera devices 161A and 162A and a reader 163A are arranged in a facility 160 and are connected to the network 103 that is opened or closed by this system. One of the two video camera devices 161A and 162A (referred to as the video camera device 161A) is for photographing the entire swing of the user, and the other (the video camera device 162A) is used for the golf club. This is for photographing the impact of the head. The reader 163A reads, for example, a user ID from a user medium 15 (not shown).

  On the other hand, two video camera devices 161B and 162B and a reader 163B are arranged in a place where the user performs his / her own form training such as the user's home 164 or a golf driving range lane not shown, Connected to the network 103. Also in this case, like the facility 160, the two video camera apparatuses 161B and 162B are for photographing the entire swing and the state of impact, respectively.

  Each of the video camera devices 161A, 162A, 161B, and 162B captures a subject, encodes and outputs the obtained video data in a predetermined compression format, outputs time information indicating the time when the video data is captured, and the video camera device 161A. , 162A, 161B, and 162B are output in association with the video data. The time information is generated in each of the video camera devices 161A, 162A, 161B, and 162B, and a time code that allows the time designation in units of frames can be used.

  Note that the video camera devices 161A, 162A, 161B, and 162B need to capture high-speed motion, and therefore preferably have a higher frame rate than a normal video camera. Furthermore, it is more preferable that the video camera devices 161A, 162A, 162A, and 162B are interlocked with the flashlight system. Depending on the system configuration and the like, the video camera devices 162A and 162B that capture the impact state may be omitted.

  The readers 163A and 163B read the user ID stored in the user medium 15 (not shown). As described above, various types of user media 15 can be applied. In the second application example, the user media 15 is a non-volatile memory, transmission / reception in the above-described synthetic resin card. And a card-type storage medium having a power generation unit. The user medium 15 is assumed to have a user ID stored in advance and given to the corresponding user.

  Similar to the first application example described above, the video data output from the video camera devices 161A and 162A and the user ID read by the reader 163A are associated with each other and transmitted to the network 103. Shall be. The same applies to the video camera devices 161B and 162B and the reader 163B. Also in the second application example, as in the first application example described above, the user ID is not limited to being read from the user medium 15 by the reader 163A. For example, the reader 163A may be an input device provided with predetermined input means such as a keyboard, and the user may input the user ID manually.

  In FIG. 16, the facility where the video camera device and the reader connected to the network 103 are arranged is only the facility 160 and the user's home 164, but in reality, a larger number of facilities, etc. Are connected to the network 103 so that they can be identified.

  The server system 104 ′ corresponds to the server system 104 described with reference to FIG. 7, and for example, the server unit 110 ′, the analysis system unit 111 ′, the sensor information database 112 ′, the user information database 113 ′, and the golf information database 166. Have In the golf information database 166, information related to golf such as information on golf equipment and information on a driving range is registered. The server unit 110 ′ controls and manages the entire system, and stores the video data, user ID, and time information transmitted from the video camera devices 161A, 162A, 161B, and 162B in the sensor information database 112 ′. To do.

  The analysis system unit 111 ′ performs primary analysis processing as described later on an example of video data acquired by the video camera apparatuses 161 </ b> A, 162 </ b> A, 161 </ b> B, and 162 </ b> B and stored in the sensor information database 112 ′. Further, threshold determination is performed based on the analysis result of the primary analysis, and secondary analysis as described below is performed according to the determination result.

  The server system 104 'is connected to the Internet 130 by a server unit 110'. For example, a user terminal 165 such as a personal computer that can be used by a user at home 164 is connected to the Internet 130. In the example of FIG. 16, the user terminal connected to the Internet 130 is only the user terminal 165, but in reality, a larger number of user terminals can be connected. Also connected to the Internet 130 is a web server 132, which is a server that provides web services such as web page publishing to the Internet 130.

  In the above description, the video camera devices 161B and 162B arranged in the user's home 164 have been described as being connected to the network 103, but this is not limited to this example. For example, it is assumed that the video camera devices 161B and 162B are connected to a user terminal 165 that is a personal computer, and shooting control by the video camera devices 161B and 162B is performed by the user terminal 165 and output from the video camera devices 161B and 162B. The video data may be transmitted from the user terminal 165 to the server system 104 ′ via the Internet 130.

  The configurations of the sensor information database 112 'and the user information database 113' will be schematically described. The configuration of the sensor information database 112 ′ is the same as that of the sensor information database 112 described with reference to FIG. 2. For the data ID for identifying the video data, pointer information indicating the storage location of the video data, the user ID, The time information is associated with each other and registered in the sensor information table. In this case, the time information is, for example, the time at which the video data was captured.

  The user information database 113 'includes a user information table in which user information is registered and a reference information table. FIG. 17 shows an example of the configuration of the user information table in the user information database 113 '. In the user information table, for each user ID for identifying a user, user attribute information corresponding to the user ID and information regarding the user's golf are registered. The user attribute information includes, for example, information for identifying the user such as the user's name and contact information (e-mail address), and information indicating the physical characteristics of the user such as age, height, and weight. The golf-related information is information related to the user's golf, for example, information related to the user's golf technology such as a handicap and a driver's flight distance, and information related to golf equipment such as a golf club used by the user. Consists of. These pieces of information are set when the user starts using the system, for example. The configuration of the reference information table will be described later.

  In the system configured as described above, according to the flow described with reference to FIG. 15, the user first performs a swing operation using an ideal form under the guidance of an instructor at the facility 160, and the form is displayed using, for example, the video camera device 161 </ b> A. A picture is taken (step S30). In addition, the user causes the reader 163A to read the user ID stored in the user medium 15. The video data on which the form is photographed is added with time information indicating the time at which the form was photographed, a sensor ID for identifying the video camera device 161A, and a user ID, and is transmitted to the server system 104 ′ via the network 103. (Step S31).

  The server system 104 ′ supplies the video data and time information, sensor ID, and user ID added to the video data to the server unit 110 ′. For example, the video data is stored in a predetermined storage device in the server unit 110 ′. The server unit 110 'stores pointer information indicating a storage location of video data, time information, and a user ID in a predetermined correspondence with the sensor information table corresponding to the sensor ID of the sensor information database 112'.

  The video data is read to the server unit 110 'based on the pointer information, the user ID, etc., and supplied to the analysis system unit 111'. The analysis system unit 111 ′ performs predetermined processing on the supplied video data, and generates reference information for the video data. The reference information is added with a reference information ID for identifying the reference information, is associated with other related information, and is registered in the reference information table in the user information database 113 '(step S32).

  The user later performs a swing operation by himself / herself, for example, at the user's home 164, and photographs the form using the video camera device 161B (step S33). Further, the user causes the reader 163B to read the user ID stored in the user medium 15. The video data in which the self-form is photographed is added with time information indicating the photographing time, a sensor ID for identifying the video camera device 161B, and a user ID, and transmitted to the server system 104 ′ via the network 103. (Step S34).

  The server system 104 'supplies the video data in the self-form and the time information, sensor ID, and user ID added to the video data to the server unit 110'. For example, the video data is stored in a predetermined storage device in the server unit 110 ′. The video data is read to the server unit 110 'based on the pointer information, the user ID, etc., and supplied to the analysis system unit 111'. The analysis system unit 111 ′ performs a predetermined process on the supplied video data. Then, the server unit 110 ′ extracts the corresponding reference information from the reference information table in the user information database 113 ′ based on the supplied user ID, and compares the reference information with the processing result based on the video data by the self form. Then, the primary analysis is performed (step S35).

  As a result of the comparison by the primary analysis, if the difference between the reference information and the processing result based on the video data based on the self-form is greater than or equal to a predetermined value (step S36), the server unit 110 ' A secondary analysis process is performed on the processing result based on the data (step S37). Then, advice information is generated based on the secondary analysis result, and transmitted to the user using e-mail, for example.

  The user receives the e-mail attached with the advice information transmitted from the server unit 110 ′, for example, by using the user terminal 165 at the user's home 164, so that the form closer to the ideal form is based on the advice information. It becomes possible to learn swing.

  Here, it is conceivable to transmit video data corresponding to the reference information of the user together with the advice information to the user. That is, the advice information and the video data are packaged together, and the package data is attached to an electronic mail and transmitted. Since the configuration of the package data is the same as that described with reference to FIG. 9, the description thereof is omitted here. The user can learn the swing more effectively by receiving the e-mail at the user terminal 165 and referring to the advice information while reproducing the video data.

  Note that video data is preferably packaged with advice information in the form of stream distribution. As an example, a predetermined URL is given to video data in the server unit 110 ′, and transmission control is performed so that streaming distribution is possible for access to the URL. On the user side, the video data can be reproduced without waiting for reception of the entire length of the video data. It is also possible to package a decomposed image of the ideal form and / or self-form in the time series direction together with advice information. For example, the server unit 110 ′ extracts video data frame images in an ideal form and / or self-form at predetermined time intervals and packages them together with advice information and the like.

  Further, in the second application example, advice information or the like is not limited to being attached to an electronic mail and transmitted. That is, as in the first application example described above, the advice data, the ideal form and / or the self-form video data and the decomposed image in the time series direction of the video data as a web page on the website, It is conceivable to present it to the user as being viewable on the Internet 130. It is also conceivable to display the associated advice information in association with the decomposed image at a predetermined timing.

2-3-3. Example of Analysis Processing According to Second Application Example Next, examples of primary analysis processing and secondary analysis processing applicable to the second application example of the first embodiment will be described. First, an example of the primary analysis process will be described. In the primary analysis process, for example, a plurality of predetermined points on the user's body are set as measurement points, the positions of the measurement points are measured for each of the ideal form and the self-form, and the values measured for the ideal form, Compare the measured value for the self-form and calculate the difference. It is conceivable to compare this difference with a predetermined threshold.

  An example of measurement points will be described with reference to FIG. Note that FIG. 18 is a frame image that captures the state at the start of swing by the ideal form. When analyzing body movements, it is conceivable to observe major joints of the body as measurement points. In the example of FIG. 18, shoulders 401A and 401B, elbows 401C and 401D, wrists 401E and 401F, hip joints 401G and 401H, knees 401I and 401J, and ankles 401K for the left and right sides of the user's body 400, respectively. And 401L are used as measurement points, and the crown portion 402 is added to the measurement points. In addition, these measurement points are examples, and are not limited to this example.

  In addition, as illustrated by a thick line in FIG. 18, the body 400 can be expressed by a straight line connecting the measurement points 401 </ b> A to 401 </ b> L and 402 and a center line 403 of the trunk. Thus, it is expected that the primary analysis process and the secondary analysis process described later are facilitated by expressing the structure of the body 400 in a simplified manner.

  A plurality of predetermined frames are extracted from the video data obtained by photographing the swing state of the ideal form and the self-form, and the positions of the measurement points 401A to 401L and the measurement points 402 are determined for each of the extracted frame images. taking measurement. For example, each frame of video data obtained by shooting a swing state is analyzed, and a frame where a swing start point, a moment when a ball hits a club head, and a swing end point are extracted is extracted. It is also conceivable to extract a frame at a time-series elapsed point based on these frames.

  Here, it is necessary to extract the measurement points 401A to 401L and 402 from each frame image. As an example, a method is conceivable in which a mark whose pattern information is registered in advance in the analysis system unit 111 ′ is attached to each of the measurement points 401 </ b> A to 401 </ b> L and 402 of the user's body 400 when photographing a form. The mark is not particularly limited as long as appearance features can be easily extracted and the influence on the user's movement is small. For example, it is conceivable to use as a mark an object having a predetermined shape or displaying a predetermined pattern, or a light emitter such as an LED (Light Emitting Diode). The analysis system unit 111 ′ detects this mark from the frame image at the time of analysis, and specifies each measurement point 401 </ b> A to 401 </ b> L, 402.

  However, the measurement points 401A to 401L and 402 may be directly extracted from the frame image. For example, it is conceivable to analyze each frame image in time series and extract a joint portion and the like.

  Thus, the position information of each measurement point obtained from the video data in the ideal form is registered in the reference information table as reference information. FIG. 19 shows a configuration example of a reference table in the user information database 113 '. Here, as illustrated in FIGS. 19A and 19B, the reference information table is composed of a plurality of tables. The configuration of the reference information table is not limited to this, and a plurality of tables may be combined into one, or further divided. In the reference information table, each of the measurement points 401A to 401L and 402 is represented by unique identification information PoID.

  FIG. 19A is an example table in which one piece of reference information is registered. The reference information itself is identified from other reference information using the reference information ID. The reference information includes, for example, information on a plurality of frames extracted from one video data, and position information indicating the positions of the measurement points 401A to 401L and 402 is registered for each of the plurality of frames. Each of the frames is expressed in units of frames using, for example, a time code. In the table, register only the above-mentioned three frames: the frame where the swing start point was shot, the frame where the ball hits the club head, and the frame where the swing end point was shot. Can be considered.

  However, the present invention is not limited to this, and all frames of video data obtained by photographing an ideal form may be registered. When all the frames are registered, the above-described three frames are made identifiable. In addition, when all the frames are registered, it is also conceivable that measurement points are extracted only for the three frames described above, and measurement point extraction is omitted for the other frames.

  FIG. 19B shows an exemplary configuration of an attribute information table in which attribute information is stored for each piece of reference information. For example, for each reference information, a corresponding user ID, an acquired date, identification information of video data corresponding to the reference information are associated with predetermined attribute information. As the predetermined attribute information, for example, an arbitrary description, the name of the facility 160, the person in charge of the user in the facility 160, and the like can be registered.

  The primary analysis may be performed as follows as an example. First, the positions of the measurement points 401A to 401L and 402 are obtained for a predetermined frame (for example, a frame at the start of swing) of video data obtained by photographing the self-form. Position information indicating the positions of the respective measurement points 401A to 401L and 402 in a predetermined frame of video data based on these self-forms, and the positions of the respective measurement points 401A to 401L and 402 extracted from the corresponding frames registered in the reference information table For each measurement point, each measurement point is weighted according to the measurement point, and for the weighted position information, the absolute value of the difference between the position information is calculated for each corresponding set of measurement points. . For example, the difference may be obtained as a vector size. Then, the absolute value of the difference calculated for each set of measurement points is added, and the addition result is compared with a predetermined threshold value. As a result of the comparison, if it is determined that the addition result is larger than the threshold (step S36 in FIG. 13), a secondary analysis is performed.

  The secondary analysis may be performed as follows as an example. Regarding the position information indicating the positions of the measurement points 401A to 401L and 402 based on the reference information and the position information indicating the positions of the measurement points 401A to 401L and 402 based on the self-form used in the primary analysis, A difference in position information is obtained as vector information for a set of measurement points corresponding to positions. And the difference information of each measurement point 401A-401L and 402 is analyzed comprehensively. The analysis content may be a twist of the body 400, an arm swing, a movement of the center of gravity, or the like.

  For example, as described above, a plurality of frames such as a frame in which the swing start point is shot, a frame in which the ball hits the club head, and a frame in which the swing end point is shot are analyzed. In such a case, the determination based on the result of the primary analysis may be performed for each analysis target frame, or may be performed using the analysis results of all the analysis target frames. When the determination based on the result of the primary analysis is performed for each frame, the secondary analysis may be performed for each corresponding frame.

  Based on the secondary analysis result, it is preferable to generate advice information for giving advice on swing to the user and present it to the user. In the server system 104 ′, for example, text data for advice corresponding to the predicted secondary analysis result is prepared in advance. Based on the result of the actual secondary analysis, advice text data is selected in advance, or a plurality of advice text data are combined as needed to form one text data. Present as advice information. The advice information is provided to the user by being included in the package data 140, for example.

  Also in the second application example of the first embodiment, as in the first application example described above, the product can be introduced to the user according to the result of the secondary analysis. . In this example in which the second application example is applied to the golf learning system, it is conceivable to introduce golf equipment according to the secondary analysis result. For example, referring to the golf information database 166 based on the secondary analysis result, a golf equipment which is considered appropriate is selected. As an example, information on the golf club is registered in the golf information database 166 in association with information indicating the characteristics of the golf club such as length and weight, and based on the result of the secondary analysis, A golf club suitable for bringing the form closer to the ideal form is searched, and product information obtained as a result of the search is presented to the user.

  At this time, the URL of a website where the product indicated in the search result can be purchased via the Internet 130 is provided to the user together with the product information. The user orders the product immediately after receiving the secondary analysis result. It can be performed.

  In the above description, the product is introduced when the difference based on the comparison result by the primary analysis is greater than or equal to the predetermined value in step S36 in the flowchart of FIG. 13, but this is not limited to this example. That is, as in the case of the first application example described above, even when these differences are determined to be less than a predetermined value in step S36, products and the like can be introduced.

  In the above description, the ideal form is shot by the user performing a swing operation under the guidance of an instructor or the like. However, this is not limited to this example. For example, it is conceivable to take a swing action based on an ideal form by an instructor and create user reference information based on the obtained video data. In this case, if there is a physique difference between the user and the instructor, there is a possibility that an accurate determination cannot be made in the primary analysis. Therefore, it is conceivable to normalize position information indicating the positions of the measurement points 401A to 401L and 402 of the instructor according to the user's physique based on the physique difference between the instructor and the user.

  In addition to this, by appropriately setting a threshold value when making a determination in the primary analysis, the ideal form of the instructor can be used as reference information without being normalized.

2-3-4. Supplement Although the case where the second application example of the first embodiment is applied to the golf learning system has been described above, this is not limited to this example. That is, the second application example can be applied to other sports as long as the user's form is compared with the ideal form for determination. For example, the second application example can be applied to baseball pitching, batting training systems, various athletics training systems, various martial arts training systems, and the like.

  Furthermore, the reference information does not necessarily have to be an ideal form. For example, of course, the user's best form may be used as reference information.

  Furthermore, the second application example is not limited to being applied to a sports learning system. For example, the second application example can be applied to a learning system for calligraphy such as brush calligraphy and pen calligraphy. In this case, it is conceivable to take a written character on a still image and compare it with a model character.

  As described above, according to the second application example of the first embodiment, the user can easily learn a form closer to the ideal form while at home, and obtain necessary tools and information. And purchases can also be made on the spot.

2-4. Third Application Example of the First Embodiment Next, a third application example of the first embodiment of the present invention will be described. The third application example is an example in which the first embodiment is applied to a foreign language conversation learning system used in a foreign language conversation class that teaches a conversation in a foreign language. In the third application example, primary analysis and secondary analysis are performed using audio information together with image information.

2-4-1. System Overview A foreign language conversation learning system according to the third application example will be schematically described. FIG. 20 shows an example flow for explaining the outline of this foreign language conversation learning system. In the foreign language conversation learning system described here, a lecturer gives a lesson of foreign language conversation to a student (user) based on a text in a specific classroom. At the same time, the user practices conversation outside the classroom, for example, at the user's home, based on the text and lesson records in the classroom. At this time, the user can perform conversation practice using communication via the Internet using a user terminal such as a personal computer connected to the Internet.

  First, in a classroom or the like, the instructor is shooting a sound as a moving picture based on the text, and the sound that is pronounced is recorded (step S40). The movement or state of the instructor's mouth and face accompanying pronunciation is called an ideal pronunciation form, and the corresponding voice is called an ideal pronunciation voice. The video data obtained by photographing the ideal pronunciation form and the audio data obtained by recording the ideal pronunciation sound are transmitted to the server system (step S41). On the server system side, based on the transmitted video data and audio data, feature information indicating the characteristics of the ideal pronunciation form and the ideal pronunciation sound is extracted, respectively, and the extracted characteristic information of the ideal pronunciation form and the characteristics of the ideal pronunciation sound are extracted. The information is registered in the user information database as user reference information (step S42).

  On the other hand, the user later reviews the pronunciation based on the text outside the classroom, for example, at home, etc., shoots the situation and records the sound that has been pronounced (step S43). The movement of the mouth of the user and the movement of the face or the state of the face, which are taken here, are called a self-speaking form, and the corresponding sound is called a self-sounding sound. The video data obtained by photographing the self-pronunciation form and the audio data recording the self-pronunciation voice are transmitted to the server system (step S44).

  On the server system side, the video data and the audio data transmitted from the user are analyzed, respectively, and feature information indicating the features of the self-speech form and the self-speech sound is extracted, respectively. Is compared with the feature information registered as the reference information, and a primary analysis is performed (step S45). As a result of the primary analysis, if the difference between the two is greater than or equal to a predetermined value (step S46), it can be considered that there is some problem in the pronunciation of the foreign language by the user. Therefore, the server system performs a secondary analysis process on the video data and audio data transmitted from the user and the analysis result obtained by analyzing the video data and the audio data so that the problem part can be extracted (step). S37).

  The server system side creates advice information for giving advice to the user based on the result of the secondary analysis processing. The created advice information is transmitted to the user (step S48). At this time, appropriate advertisement information can be added to the advice information and transmitted. In the case of a foreign language conversation learning system, for example, introduction of other courses can be considered as advertisement information.

  If it is determined in step S46 that the difference is less than the predetermined value, a notification indicating that there is no problem with the current pronunciation, for example, is transmitted from the server system side to the user.

2-4-2. About System Applicable to Third Application Example FIG. 21 schematically shows an example system applicable to the third application example of the first embodiment. A system applicable to the third application example illustrated in FIG. 21 basically has the same configuration as each system described with reference to FIGS. 1, 7, and 16. In FIG. 21, the same reference numerals are given to the portions common to 16 described above, and detailed description thereof will be omitted.

  In the system illustrated in FIG. 21, a user goes to a specific classroom 170 and takes a lesson of a foreign language conversation under a lecturer. In addition, the user uses the user terminal 175 such as a personal computer connected to the Internet 130 outside the classroom 170, for example, the user's home 174, etc., to practice conversation based on the lesson received in the classroom 170, etc. It can be performed using communication via the network.

  The classroom 170 is a place where the instructor gives a lesson of foreign language conversation to the user. In the example of FIG. 21, a video camera device 171A, a microphone (MIC) 172A, and a reader 173A are arranged in a classroom 170 and are connected to the network 103 that is open or closed by this system. The video camera device 171A is arranged so that, for example, the state of pronunciation of a lecturer can be photographed. Similarly, the microphone 172A is arranged so that, for example, the voice of the lecturer can be clearly collected.

  Note that the microphone 172A actually performs a predetermined signal processing such as amplification processing on a microphone unit that collects sound and an audio signal output from the microphone unit, and further converts it into a digital audio signal and outputs it. A signal processing unit;

  On the other hand, a video camera device 171B, a microphone 172B, and a reader 173B are arranged outside the classroom 170, for example, at a user's home 174, and are connected to the network 103, respectively. The video camera device 171 </ b> B and the microphone 172 </ b> B are arranged so as to record a user's pronunciation.

  Each of the video camera devices 171A and 171B outputs video data obtained by photographing a subject by compressing and encoding the video data in a predetermined manner, time information indicating the time when the video data was photographed, and video camera devices 171A and 171B. Each sensor ID to be identified is output in association with the video data. Similarly, in the microphones 172A and 172B, the audio data obtained and collected respectively is compressed and encoded in a predetermined manner, and time information indicating the time when the audio data is collected and the microphones 172A and 172B are identified. The sensor ID to be output is output in association with the audio data. Here, the video data and the audio data are preferably output synchronously.

  The readers 173A and 173B read a user ID and the like from a user medium 15 (not shown). As described above, various types of user media 15 can be applied. In the third application example, the user media 15 is a non-volatile memory, transmission / reception in the above-described synthetic resin card. And a card-type storage medium having a power generation unit. The user medium 15 is assumed to have a user ID stored in advance and given to the corresponding user.

  Similar to the first and second application examples described above, video data output from the video camera device 171A, time information and sensor ID, audio data output from the microphone 172A, time information and sensor ID, and reader 173A The user ID read out in step S1 is associated with a predetermined one and transmitted to the server system 104 ″ described later via the network 103. Similarly, each data output from the video camera device 171B and output from the microphone 172B are transmitted. Each data and the user ID read by the reader 173B are associated with each other and transmitted to the server system 104 ′ via the network 103.

  Also in the third application example, as in the first and second application examples described above, the user ID is not limited to being read from the user medium 15 by the reader 173A or the reader 173B. For example, the reader 173A or the reader 173B may be an input device provided with predetermined input means such as a keyboard, and the user may input the user ID manually.

  In FIG. 21, only the classroom 170 and the user's home 174 have a video camera device, a microphone, and a reader that are connected to the network 103, but in reality, more classrooms and A video camera device, a microphone, and a reader provided at the user's home are connected to the network 103 so that they can be identified.

  The server system 104 ″ corresponds to the server system 104 in FIG. 7 or the server system 104 ′ in FIG. 16, and includes, for example, a server unit 110 ″, an analysis system unit 111 ″, a sensor information database 112 ″, and a user information database 113 ″. The server unit 110 ″ controls and manages the entire system. The video camera device 171A, the microphone 172A, and the reader 173A, the video camera device 171B, the microphone 172B, and the reader 173B transmit video data, audio data, The user ID and time information are stored in the sensor information database 112 ″ in a predetermined manner.

  The analysis system unit 111 ″ receives video data acquired by the video camera devices 171A and 171B and stored in the sensor information database 112 ″, and audio data acquired by the microphones 172A and 172B and stored in the sensor information database 112 ″. Then, a primary analysis process as described below is performed, a threshold is determined based on the analysis result of the primary analysis, and a secondary analysis as described below is performed according to the determination result. The analysis and / or the secondary analysis may be performed by using the video data and the audio data independently, respectively, or the video data and the audio data may be performed in association with each other.

  The server system 104 ″ is connected to the Internet 130 by the server unit 110 ″. For example, a user terminal 175 such as a personal computer that a user can use at home 174 is connected to the Internet 130. In the example of FIG. 21, the user terminal connected to the Internet 130 is only the user terminal 175, but in reality, a larger number of user terminals can be connected. Also connected to the Internet 130 is a web server 132, which is a server that provides web services such as web page publishing to the Internet 130.

  In the above description, the video camera device 171B, the microphone 172B, and the reader 173B disposed in the user's home 174 have been described as being connected to the network 103, but this is not limited to this example. For example, the video camera device 171B, the microphone 172B, and the reader 173B are connected to a user terminal 175 that is a personal computer. The video camera device 171B controls shooting, the microphone 172B controls sound collection, and the reader 173B transmits data. The user terminal 175 may perform reading control, and the video data output from the video camera device 171B, the microphone 172B, and the reader 173B may be transmitted from the user terminal 175 to the server system 104 ″ via the Internet 130.

  The configurations of the sensor information database 112 ″ and the user information database 113 ″ will be schematically described. First, the configuration of an example of the sensor information database 112 ″ will be described with reference to FIG. 22. The video data output from the video camera device and the audio data output from the microphone are as exemplified in FIG. 22A, for example. 22A, the video data and the audio data are managed as files, and are associated with each other by time information.

  FIG. 22B shows an example of a table for managing video data in a video data file and audio data in an audio data file in association with each other synchronously. In the example of FIG. 22B, audio data is managed in association with each frame of video data in units of samples. As an example, when the frame frequency is 30 Hz and the sampling frequency of audio data is 48000 Hz, 1600 samples of monaural audio data are associated with each frame.

  The user information database 113 ″ includes, for example, a user information table and a reference information table, and stores information related to registered users. In the third application example, the configuration of the user information table is shown in FIG. Since the same configuration as that described above can be applied, detailed description is omitted, for example, the attribute information of the user is associated with the user ID for identifying the user in the user information table. The user name, contact information (e-mail address, etc.), information indicating the course the user is attending, the name of the instructor in charge of the user, etc. The configuration of the reference information table will be described later.

  In the system having such a configuration, according to the flow described with reference to FIG. 20, first, the user causes the reader 173 </ b> A to read the user ID stored in the user medium 15 in the classroom 170. Based on the given text, students will take a foreign language conversation lesson under the guidance of the instructor. The video camera device 171A and the microphone 172A arranged in the classroom 170 capture the ideal pronunciation form and pick up the ideal pronunciation sound by the lecturer (step S40). Video data and audio data are respectively output from the video camera device 171A and the microphone 172A. The video data and audio data are the sensor ID for identifying the video camera device 171A and the microphone 172A, the time when shooting and / or sound collection is started. Is added to the server system 104 ″ via the network 103 (step S41).

  The server system 104 ″ supplies the video data, audio data, time information added to the video data and audio data, sensor ID, and user ID to the server unit 110 ″. The video data and the audio data are respectively stored as files in the storage device of the server unit 110 ″. The server unit 110 ″ stores the video data file and the sensor of the video camera device 171A with respect to the sensor information database 112 ″. The ID, the audio data file, the sensor ID of the microphone 172A, and the time information are stored in the sensor information table in association with each other, and the video data and the audio data are associated with a frame of the video data, for example, in units of audio data samples. And stored in the sensor information table.

  The video data file and the audio data file are read from the sensor information database 112 ″ based on the user ID and the like and supplied to the analysis system unit 111 ″. The analysis system unit 111 ″ performs predetermined processing on the video data and / or audio data stored in the audio data file stored in the supplied video data file, and refers to the video data and / or audio data. The reference information is added with a reference information ID for identifying the reference information, is associated with other related information, and is registered in the reference information table in the user information database 113 ″ in advance. (Step S42).

  The user later reviews the lesson according to the text used in the classroom 170, for example, at the user's home 174. The self-pronunciation form by this review is photographed by the video camera device 171B, and the self-pronunciation sound is collected by the microphone 172B. Further, the user causes the reader 173B to read the user ID stored in the user medium 15. The video data in which the self-speech form is photographed and the audio data in which the self-speech sound is collected identify time information indicating the time when the photographing and / or sound collection is started, and the video camera device 171B and the microphone 172B, respectively. The sensor ID and the user ID are added and transmitted to the server system 104 ″ via the network 103 (step S44).

  The server system 104 ″ supplies the video data by the self-sounding form and the audio data by the self-sounding voice and the time information, sensor ID and user ID added to these data to the server unit 110 ″. The video data and the audio data are respectively stored as files in the storage device of the server unit 110 ″. The video data and the audio data are read to the server unit 110 ″ based on the user ID and the like, and the analysis system unit. 111 ″. The analysis system unit 111 ″ performs predetermined processing on the supplied video data and audio data. Then, the server unit 110 ″ extracts the corresponding reference information from the reference information table in the user information database 113 ″ based on the supplied user ID, and the reference information, the video data by the self-speech form, and the audio by the self-speech sound The processing result based on the data is compared, and the primary analysis is performed (step S45).

  If the difference based on the reference information, the video data based on the self-speech form and the audio data based on the self-speech sound is greater than or equal to a predetermined value as a result of the primary analysis (step S46), the server unit 110 " A secondary analysis process is further performed based on the audio data by voice (step S47), and advice information is generated based on the secondary analysis result and transmitted to the user using e-mail, for example.

  The user receives an e-mail attached with advice information transmitted from the server unit 110 ″ using the user terminal 175 at the user's home 174, for example, so that the ideal pronunciation form and the ideal pronunciation are based on the advice information. It is possible to learn pronunciations close to speech.

  In the third application example, similarly to the first and second application examples described above, it is considered that video data and audio data corresponding to the reference information of the user are transmitted to the user together with the advice information. It is done. That is, the advice information and the video data and audio data are packaged together, and the package data is attached to an e-mail and transmitted. It is more preferable that the video data and the audio data can be streamed respectively. The user can learn the foreign language conversation more effectively by receiving the e-mail at the user terminal 175 and referring to the advice information while reproducing the video data and the audio data.

  Of course, in the third application example, the advice information is not limited to being transmitted by attaching to an e-mail, but the advice information, the ideal pronunciation form, the video data and the audio data using the ideal pronunciation voice, etc. As the upper web page, it is possible to present it to the user as being viewable on the Internet 130.

2-4-3. Example of Analysis Processing According to Third Application Example Next, examples of primary analysis processing and secondary analysis processing applicable to the third application example of the first embodiment will be described. First, an example of reference information applicable to the third application example will be described. The video data in the ideal pronunciation form and the audio data in the ideal pronunciation sound are analyzed in a predetermined manner to obtain a reference method.

  An example method for obtaining reference information is shown below. First, the pronunciation content is analyzed based on the audio data, and the portion of the text that is pronounced is analyzed. A technique for converting pronunciation sound into text is known. Then, the video data portion corresponding to the audio data portion associated with the text portion is extracted based on the registered contents of the table of the sensor information database 112 ″. Data and audio data can be associated with each other.

  Next, face detection is performed on the video data, the lip contour in the detected face region is specified, and time-series changes of the lip contour are obtained in association with the audio data, thereby generating pronunciation and lip Movement can be associated. Further, the lip shape corresponding to each pronunciation is digitized. For example, the shape of the lip may be converted into an evaluation value by using one or a plurality of numerical values based on the width of the lip contour and the height of the lip contour.

On the other hand, for audio data, for example, as shown in FIG. 23, for example, an envelope composed of changes in the signal axis in the time axis direction is obtained, and based on the envelope, each sound generation time interval t ₂ -t ₁ , time interval t ₃ -T ₂ and time interval t ₄ -t ₃ are measured. As a result, a pronunciation rhythm is obtained when a predetermined text is pronounced. It is conceivable that the time interval of each pronunciation is relatively determined and digitized to obtain an evaluation value. In addition, the portion where the sound is generated based on the envelope is analyzed, and the pitch and volume of the sound are obtained for each sound generation portion. The intonation information can be known based on the pitch and volume of the sound for each sounding portion. It is conceivable that the pitch and level of each pronunciation are relatively obtained and digitized to obtain an evaluation value.

  Each evaluation value analyzed as described above based on the video data of the ideal pronunciation form and the audio data of the ideal pronunciation sound can be used as reference information. FIG. 24 shows an example of the configuration of the reference information table in the user information database 113 ″ for registering the reference information obtained in this way. In the example of FIG. 24, a video data file associated with the user ID is shown. In the audio data file, a portion corresponding to the text “This is a pen” constituting the reference information is specified, and each evaluation value obtained for the specified portion is registered.

  In the example of FIG. 24, as reference information, evaluation values of pronunciation timing, time interval and pitch are registered for each pronunciation in the text based on the analysis result for the envelope of the audio data. At the same time, lip shape evaluation values are registered as reference information based on the analysis results for audio data and video data.

  The primary analysis can be performed as follows as an example. Based on the audio data based on the self-speech voice, the text portion registered in the reference information table is specified for the audio data and the video data based on the self-speech form. For the part specified as the text part, an analysis similar to that performed when obtaining the reference information is performed, and an evaluation value is obtained based on each analysis result. Then, each evaluation value obtained for the reference information and each evaluation value obtained based on the self-speaking form and the self-speech voice are weighted in a predetermined manner for each corresponding evaluation value. Then, with respect to the weighted evaluation value, the absolute value of the difference is calculated for each set of evaluation values corresponding to the reference information, the self-speaking form, and the self-spoken voice, and the addition result is compared with a predetermined threshold value. If it is determined that the addition result is larger than the threshold value (step S46 in FIG. 20), secondary analysis is performed.

  The secondary analysis may be performed as follows as an example. First, the differences for each set of evaluation values calculated during the primary analysis are sorted in a predetermined manner, and items with poor results are extracted from the pronunciation timing, lip shape, time interval, and pitch. And about each extracted item, each evaluation value calculated | required about the reference information and each evaluation value calculated | required based on the self-voice form and the self-voiced voice are analyzed in more detail. Not limited to this, it is conceivable to comprehensively analyze the sound generation timing, lip shape, time interval, and pitch. For example, registering trends, causes, improvement methods, etc. in advance as a table for combinations of various evaluation values of pronunciation timing, lip shape, time interval, and pitch, and comparing them with actual evaluation values Is also possible.

  Based on the secondary analysis result, advice information for giving advice such as utterance to the user may be generated and presented to the user. As an example, in the server system 104 ″, text data for advice is prepared in advance for each analysis result predicted by the secondary analysis. Then, based on the result of the actual secondary analysis, the text data for advice is prepared. The text data is combined in a predetermined manner into one piece of text data, which is presented as advice information for the user, and the advice information is included in the package data 140 and provided to the user.

  In the third application example of the first embodiment, as in the first and second application examples described above, products can be introduced to the user according to the result of the secondary analysis. Has been. In this example in which the third application example is applied to a foreign language conversation learning system, it is conceivable to introduce another course of the conversation classroom, for example, according to the secondary analysis result. At this time, it is more preferable to apply for the change to the introduced course via the Internet 130.

3. First Modified Example of First Embodiment of the Invention Next, a first modified example of the first embodiment of the present invention will be described. FIG. 25 schematically shows an example of a system configuration according to the first modification of the first embodiment. Note that, in FIG. 25, the same reference numerals are given to the portions common to FIG. 1 described above, and detailed description thereof is omitted. A first modification of the first embodiment illustrated in FIG. 25 is a reader device for reading data of the user medium 15 from the configuration of FIG. 1, as is apparent from comparison with FIG. Is omitted. That is, in the first modification of the first embodiment, the user is identified based only on the output data of the sensor devices 11A, 11B,.

3-1-1. About System In FIG. 25, sensor devices 11A, 11B,... Are arranged at predetermined facilities 10A, 10B,. The sensor devices 11A, 11B, 11C,... Are, for example, camera devices that have an image sensor and are configured to capture still images and / or moving images, and are still image data obtained by capturing a subject. And / or moving image data is output in association with sensor ID, which is identification information for identifying each of the sensor devices 11A, 11B, 11C,..., And time information indicating the time of shooting. These sensor devices 11A, 11B, 11C,... Are connected to the network 13, respectively. A server system 14 </ b> A is further connected to the network 13. The still image data and / or moving image data output from the sensor devices 11A, 11B, 11C,... And the sensor ID are transmitted to the server system 14A via the network 13.

  In the following, in order to avoid complexity, still image data and / or moving image data are described as “image data” in a unified manner unless otherwise required.

  The server system 14A schematically includes a server unit 20, an analysis system unit 21, a user information database 22A, and a sensor information database 23, as in the first embodiment described above. The sensor information database 23 has a sensor information table that stores image data transmitted from the sensor devices 11A, 11B,... In association with the sensor ID and time information. Since the same configuration as the sensor information table according to the first embodiment described above with reference to FIG. 2 can be applied to the sensor information table, detailed description thereof is omitted here.

  The user information database 22A is based on the user information table that has already been described with reference to FIG. 3 and manages the user ID and attribute information in association with each other and the image data transmitted from the sensor devices 11A, 11B,. A personal identification information table for managing the personal identification information for identifying the user ID in association with the user ID.

  The personal identification information table will be described. In the first modification of the first embodiment, the server unit 20 performs predetermined face detection on the image data transmitted from the sensor devices 11A, 11B,. The user is identified based on the image. For example, the user's face is photographed in advance, face detection is performed on the obtained image data, and facial features are extracted from the detected face image. The personal identification information table manages facial feature values obtained by digitizing the extracted facial features as face information in association with the user ID. FIG. 26 shows an example of the configuration of an identification information table for managing face information and user ID in association with each other. In FIG. 26, a figure is described as the face information. Actually, a predetermined numerical value indicating the feature value of the face is stored in this portion.

3-1-2. Example of User Identification Method An example of a user identification method according to the first modification of the first embodiment will be described with reference to the flowchart of FIG. For example, the user's face is photographed by the sensor device (referred to as sensor device 11A), and the obtained image data is transmitted to the server system 14A via the network 103 (step S100). The image data is supplied to the server unit 20 and a process for identifying the user is performed (step S101).

  In the user identification process of step S101, first, in step S110, the server unit 20 performs face detection on the image data transmitted from the sensor device 11A, and extracts face feature information from the detected face image.

  Various techniques for identifying a face part from image data and extracting features of the specified face part have been proposed, and many techniques have already been put into practical use. For example, non-patent literature “KCYow and R. Cipolla,“ Feature-based human face detection, ”“ Image and Vision Computing, Vol. 15, No. 9, pp 713-735 (1997) ”includes eyes, nose, A method for detecting a mouth and detecting a face from the position is described. For example, the eye, nose, and mouth are detected by performing a predetermined filtering process on the image data to extract the contour line, and comparing each with the pupil, lip, and nostril contour model to identify each position. Conceivable.

  In addition, the digitization of face information, for example, prepared in advance a face information dictionary in which various face patterns are registered, and the face pattern registered in the face information dictionary and the face image data obtained by photographing the face by the user, It is conceivable to perform the comparison with the position information of the nose and mouth. As an example, a face pattern registered in the face information dictionary is obtained by comparing the face pattern registered in the face information dictionary with the position information of the eyes, nose, and mouth extracted from the user's face image data to obtain similarity. Among them, the face pattern having the highest similarity is extracted, and information indicating the face pattern and similarity information are used as facial feature data of the user. As the similarity, for example, a difference between information based on the face information dictionary and information based on the user's face image data can be used.

  When the facial feature information is extracted from the transmitted image data, the personal identification information table is referred to in the next step S111, and the facial information corresponding to the facial feature information extracted in step S110 is searched and searched. The user ID associated with the face information is identified.

3-1-3. Schematic processing flow In the first modification of the first embodiment of the present invention, processing other than the above-described user identification processing, that is, secondary analysis based on threshold determination based on the results of primary analysis and primary analysis. The determination according to whether or not the analysis is performed, the secondary analysis, and the notification based on the secondary analysis result can be directly applied to the method according to the first embodiment described above. For example, referring to the flowchart of FIG. 5, the user's face information is registered in advance in the personal identification information table prior to processing, and in the first step S10, image data captured by the sensor device 11A is transmitted to the server system 14A. To do. The process for reading the user ID from the user medium is omitted. On the server system 14A side, the user ID is specified based on the transmitted image data as described in step S101 of FIG.

  The processing after the user ID is specified, that is, the processing after step S11 in FIG. 5, can apply the same processing as in the first embodiment described above. That is, referring to FIG. 5, the server system 14A stores the specified data in the sensor information database 22 in association with the image data and the time information to which the identified user ID is assigned, and the analysis system unit 21 stores the image data in advance. (Step S11). In the next step S12, corresponding reference information is extracted from the reference information table in the user information database 22A based on the specified user ID, and the primary analysis process is performed based on the extracted reference information and the processing result in step S11. For example, the difference between the reference information and the processing result in step S11 is obtained. A threshold is determined based on the primary analysis result (step S13), and if it is determined that the threshold is exceeded, further secondary analysis is performed (step S14), and a notification based on the secondary analysis result is transmitted to the corresponding user. (Step S15). As the notification destination, contact information acquired by referring to the user information table in the user information database 22A based on the user ID specified from the image data as described above is used.

3-2. Application Example of First Embodiment to First Modification Example The first, second, and third application examples of the first embodiment described above are the first modification of the first embodiment. Can be applied to examples. In the case of the first, second, and third application examples of the first embodiment, as in the case described above, the processes other than the identification of the user ID are left as they are, as in the first embodiment of the first embodiment. It can be applied to a modified example. Hereinafter, a first application example of the first embodiment described above, that is, an example in which a makeup support system is applied to the first modification of the first embodiment will be described.

  FIG. 28 schematically shows an example of a system configuration in which a makeup support system, which is a first application example of the first embodiment, is applied to the first modification of the first embodiment. In FIG. 28, parts common to those in FIG. 7 described above are denoted by the same reference numerals, and detailed description thereof is omitted. As apparent from the comparison with FIG. 7 described above, the system according to the application example of the first modification of the first embodiment includes a reader device for reading the data of the user medium 15 from the configuration of FIG. It is omitted. That is, in the application example in the first modification of the first embodiment, the user is identified based only on the output data of the camera devices 101A and 101B.

3-3-1. About System Applicable to Application Example In FIG. 28, a camera device 101A is arranged for a facility 100A such as a cosmetic corner or a makeup studio of a department store, which is an object of service of this system. The camera apparatus 101A is connected to the server system 104A via the network 103. Similarly, a camera device 101B is arranged at the user's home 100B or the like and connected to the server system 104A via the network 103. The server system 104A has a server unit 110 and an analysis system unit 111 as well as a user information database 112A, a sensor information database 113, and a cosmetics database 114, as in the first application example of the first embodiment described above. . Here, the user information database 112A includes the user identification information table described with reference to FIG. 26 together with the user information table.

3-3-2. Schematic Process Flow FIG. 29 is a flowchart showing a process flow of an example of a makeup support system according to an application example in the first modification of the first embodiment. In the flowchart of FIG. 29, the same reference numerals are given to the processes common to the flowchart of FIG. 6 described above, and detailed description thereof is omitted. In the flowchart of FIG. 29, processing (step S101A and step S101B) for performing user identification based on image data is added to the flowchart of FIG. 6 described above.

  Prior to the processing of this flowchart, it is assumed that the user's face is photographed in a predetermined manner, face detection is performed on the obtained image data, and facial feature information is extracted. It is assumed that the facial feature information and the user ID for identifying the user are associated with each other and registered in the personal identification information table in the user information database 112A.

  The user has the ideal makeup applied at the facility 100A (step S20), and the camera device 101A captures a predetermined face with the ideal makeup. The image data obtained by photographing is transmitted to the server system 104A via the network 103 together with time information indicating the time when photographing was performed and a sensor ID for identifying the camera device 101A (step S21).

  The server system 104A supplies the image data transmitted from the camera device 101A and the time information and sensor ID added to the image data to the server unit 110. The server unit 110 performs user identification processing based on the supplied image data (step S101A). That is, as described with reference to FIG. 27, the server unit 110 performs face detection processing on the supplied image data, and extracts facial feature information from the detected face image (step S110 in FIG. 27). ). Then, the user ID corresponding to the extracted facial feature information is specified with reference to the identification information table (step S111 in FIG. 27).

  When the user ID is specified, the server unit 110 stores the image data, the time information, and the user ID in a predetermined manner in the sensor information table corresponding to the sensor ID in the sensor information database 112. The server unit 110 reads out the image data stored in the sensor information database 112 based on the user ID and supplies the read image data to the analysis system unit 111. The analysis system unit 111 performs a predetermined process on the supplied image data. Based on the analysis result, reference information corresponding to the image data is generated. As the process for generating the reference information, the process in the first application example of the first embodiment described above can be applied as it is. The generated reference information is added with a reference information ID, associated with other related information, and registered in the reference information table in the user information database 113A (step S22).

  The user later performs self-makeup, for example, at the user's home 100B (step S23), and the camera device 101B captures a predetermined face by the camera device 101B. The image data obtained by shooting is added with time information indicating the time of shooting and a sensor ID for identifying the camera device 101A, and is transmitted to the server system 104 via the network 103 (step S24). Supplied to the unit 110. Based on the supplied image data, the server unit 110 performs user identification processing in the same manner as in step S101A described above (step S101B), and identifies the corresponding user ID.

  When the user ID corresponding to the image data by self-make is specified, the server unit 110 detects the image data by self-make, the time information and sensor ID added to the image data, and the specified user ID. The image data, time information, and user ID are stored in association with each other in the sensor information table corresponding to the sensor ID in the information database 112.

  The image data stored in the sensor information database 112 is read by the server unit 110 based on the user ID and the like, supplied to the analysis system unit 111, and subjected to predetermined processing. The server unit 110 extracts corresponding reference information from the reference information table in the user information database 113 based on the specified user ID, compares the reference information with the processing result based on the image data by self-make, Analysis is performed (step S25).

  The subsequent processing is the same as the processing after step S25 described with reference to FIG. That is, as a result of the primary analysis in step S25, if the difference based on the reference information and the image data by self-make is greater than or equal to a predetermined value (step S26), the server unit 110 performs further secondary analysis processing based on the image data by self-make. This is performed (step S27). As the secondary analysis processing, the secondary analysis processing in the first application example of the first embodiment described above can be applied as it is. Advice information is created based on the secondary analysis result and transmitted to the user.

  The sending of advice information to the user and the presentation to the user using the website can be performed in the same manner as in the first application example of the first embodiment described above. Description is omitted.

  Also in the application example of the first modification of the first embodiment, the user can use the advice information transmitted from the server unit 110 as in the first application example of the first embodiment described above. Based on this, it is possible to perform self-makeup that is closer to ideal makeup while at home.

  In the above description, the user identification information is registered in advance in the user information database 112A, but this is not limited to this example. For example, after image data of ideal makeup is transmitted to the server in step S21, the process proceeds to step S22 without going through step S101A, and the feature information obtained based on the image data is registered in the personal identification information table in step S22. To do. At the time of registration, the user ID can be associated with the feature information and registered in the personal identification information table. In this case, it is conceivable to generate a user ID on the system side. You may make it input user ID using a predetermined input means.

3-3-3. Application of the first embodiment to the second and third application examples As is apparent from the description using the flowchart of FIG. 29, the first modification example of the first embodiment and the above-described example are also described. The first embodiment is different from the first embodiment only in the user identification method, and is based on other processing related to the subject of the present invention, that is, based on the results of primary analysis and primary analysis based on image data obtained by photographing the user. The determination on whether or not to perform the secondary analysis by the threshold determination, and the notification based on the secondary analysis and the secondary analysis result are common. Therefore, the second and third application examples of the first embodiment described above are the first of the first embodiment of the first embodiment, in the same manner as the application example of the first modification of the first embodiment. It can be applied to the modified example.

  For example, when the second application example of the first embodiment is applied to the first modification of the first embodiment, the readers 163A and 163B are omitted in the system configuration of FIG. At the same time, in the flowchart of FIG. 15, after the video data transmission processing to the server in steps S31 and S34, the user identification processing in step S101 described with reference to the flowchart of FIG. 27 is performed. Similarly, when the third application example of the first embodiment is applied to the first modification of the first embodiment, the readers 173A and 173B are omitted in the system configuration of FIG. In the flowchart of FIG. 20, after the audio data and video data transmission processing to the server in steps S41 and S44, the user identification processing in step S101 described with reference to the flowchart of FIG. 27 is performed. .

4). Second Modification of First Embodiment of Invention Next, a second modification of the first embodiment of the present invention will be described. FIG. 30 shows an example of the system configuration according to the second modification of the first embodiment. In FIG. 30, the same parts as those in FIG. 1 described above are denoted by the same reference numerals, and detailed description thereof is omitted. 30, the system according to the second modification of the first embodiment illustrated in FIG. 30 has the facilities 10A, 10B,... From the configuration of FIG. , The reader for reading the data of the user medium 15 is omitted, and the writers 16A, 16B,... For writing the data to the user medium 15 ′ and the data stored as a history in the storage medium are read. A reader device 17 is added.

  The writer devices 16A, 16B,... Write the system usage history to the user medium 15 'that is a storage medium held by the user. At this time, the writer devices 16A, 16B,... Need only write sensor IDs for identifying the corresponding sensors 11A, 11B,... And information indicating the use date and time of the facility, for example. It is not necessary to connect to the server system 14B, and it can be configured as an independent device.

4-1-1. About System In FIG. 30, sensor devices 11A, 11B,... Are respectively arranged at predetermined facilities 10A, 10B,. The sensor devices 11A, 11B,... Are, for example, camera devices that have an image sensor and are configured to capture still images and / or moving images, and still image data obtained by capturing a subject and / or Alternatively, the moving image data is output in association with a sensor ID that is identification information for identifying each of the sensor devices 11A, 11B,. These sensor devices 11A, 11B,... Are connected to the network 13, respectively.

  A server system 14B is further connected to the network 13. Still image data and / or moving image data output from the sensor devices 11A, 11B,... (Hereinafter referred to as “image data” unless otherwise specified) and the sensor ID are transmitted via the network 13 to the server system. 14A is transmitted. The reader device 17 connected to the network 103 reads the history information stored in the user medium 15 ′ and transmits it to the server system 104 ″ via the network 103.

  The server system 14B schematically includes a server unit 20, an analysis system unit 21 ″, a user information database 22, and a sensor information database 23, as in the first embodiment described above. 3 has a user information table for managing user IDs and attribute information in association with each other, as described above with reference to Fig. 3. The sensor information database 23 stores image data transmitted from the sensor devices 11A, 11B,. 2 has a sensor information table that is stored in association with the sensor ID and time information described with reference to FIG.

  FIG. 31 shows an exemplary configuration of history information stored in the user medium 15 '. The history information includes a sensor ID for identifying the sensor devices 11A, 11B,... And a time indicating a time at which data is written to the user medium 15 ′ by a writer device corresponding to the sensor device indicated by the sensor ID. Information is stored in association with each other. In the example of FIG. 31, the time information includes date information.

4-1-2. Specification of data based on history information In the second modification of the first embodiment, the data corresponding to the user is determined from the data stored in the sensor information database 23 based on the history information stored in the user medium 15 '. Identify the data. FIG. 32 is a flowchart showing an example of a method for specifying data based on the history information. Prior to the processing of this flowchart, it is assumed that a user ID for identifying the user is written in the user medium 15 ′ in advance.

  For example, when the user uses the facility 10A to acquire user data by the sensor device 11A, the writer 16A indicates the sensor ID for identifying the sensor device 11A with respect to the user medium 15 ′ and the current time. The time information is written in association with each other (step S200). The user writes data to the user medium 15 ′ by the writer 16A each time user data is acquired by the sensor devices 11A, 11B,..., And the time information is stored in the user medium 15 ′. And the sensor ID are associated and written as history information.

  The data acquired by the sensor device 11A is added with the sensor ID corresponding to the sensor device 11A and time information indicating the current time, and transmitted to the server system 104 ″ via the network 103 (step S201). The transmitted data, sensor ID, and time information are supplied to the server unit 20 and stored accumulatively in the sensor information database 23. The transmitted data is stored in a predetermined storage of the server unit 20, and the storage location. May be stored in the sensor information database 23.

  In the next step S202, the data stored in the sensor information database 23 is specified based on the history information and the user ID written in the user medium 15 '. More specifically, the user performs a reading operation of the history information stored in the user medium 15 'by the reader 17 (step S210). The history information read from the user medium 15 ′ to the reader 17 is transmitted to the server system 104 ″ via the network 103 and supplied to the server unit 20. The server unit 20 detects the sensor based on the supplied history information. With reference to the sensor information table in the information database 23, data corresponding to the sensor ID and time information is specified (step S211).

4-1-3. Schematic processing flow In the second modification of the first embodiment of the present invention, whether or not to perform the secondary analysis based on the threshold judgment based on the results of the primary analysis and the primary analysis, which is the subject of the present invention. For the determination and the notification based on the secondary analysis and the secondary analysis result, the method according to the first embodiment described above can be applied as it is. For example, referring to the flowchart of FIG. 5, prior to processing, the user's face information is registered in the personal identification information table in advance, and in the first step S10, for example, image data photographed by the sensor device 11A is used for photographing. The server ID is transmitted to the server system 14B together with the sensor ID for identifying the sensor device 11A and the time information indicating the time when the photographing was performed. At this time, for example, the user performs a process of writing the sensor ID corresponding to the sensor 11A and the time information indicating the current time as history information to the user medium 15 ′ by the writer 16A (step S200 in FIG. 32). .

  The server system 14B performs a predetermined process on the data transmitted from the sensor 11A (step S11). At this time, for example, the user performs a process of reading the history information and the user ID stored in the user medium 15 'by the reader 17 (step S210). The history information and user ID read by the reader 17 from the user medium 15 ′ are transmitted to the server system 14 </ b> B via the network 103. The server system 14B side refers to the sensor information database 23 based on the history information, identifies the corresponding data, and associates the user ID with the identified data (step S211). The predetermined process in step S11 is performed using the data specified and associated with the user ID.

  The subsequent processing is the same as the processing after step S12 in FIG. That is, in step S12, the corresponding reference information is extracted from the reference information table in the user information database 22 based on the user ID read from the user medium 15 ′, and the extracted reference information and the predetermined processing in step S11 are performed. The results are compared and a primary analysis is performed. Then, a threshold determination is made based on the result of the primary analysis (step S13), and if it is determined that the threshold has been exceeded, further secondary analysis is performed (step S14), and the user corresponding to the notification based on the secondary analysis result (Step S15). As a notification destination, contact information acquired by referring to the user information table in the user information database 22 based on the user ID specified from the image data as described above is used.

4-2. Application Example of Second Embodiment of First Embodiment The first, second, and third application examples of the first embodiment described above are the second modification of the first embodiment. Can be applied to examples. In the case of the first, second, and third application examples of the first embodiment, similarly to the above, the processes other than the specification of data are performed as they are, and the second modification of the first embodiment is performed. Can be applied to examples. Hereinafter, a first application example of the first embodiment described above, that is, an example in which the makeup support system is applied to the second modification of the first embodiment will be described.

4-2-1. System applicable to application example FIG. 33 shows an example system in which a makeup support system, which is the first application example of the first embodiment, is applied to the second modification of the first embodiment. A structure is shown schematically. In FIG. 33, parts common to those in FIG. 7 described above are denoted by the same reference numerals, and detailed description thereof is omitted. The system according to the application example of the second modification of the first embodiment is a reader device for reading the data of the user medium 15 ′ from the configuration of FIG. Are added, and writers 140A and 140B for writing time information and sensor ID to the user medium 15 ′ and a reader 141 for reading information written on the user medium 15 ′ are added.

  In FIG. 33, a camera apparatus 101A is arranged for a facility 100A such as a cosmetic corner or a makeup studio of a department store, which is a service target of this system. The camera apparatus 101A is connected to the server system 104A via the network 103. Similarly, a camera device 101B is arranged at the user's home 100B or the like and connected to the server system 104B via the network 103. In addition, writers 140A and 140B are arranged in facility 100A and user home 100B, respectively. These writers 140A and 140B can be configured as independent devices that are not connected to the network 103.

  Further, a reader 141 for reading data from the user medium 15 'is provided in a predetermined manner. The reader is provided in the user's home 100B, for example, and connected to the network 103. The data stored in the user medium 15 ′ is read by the user using the reader 141, for example, and transmitted to the server system 104 </ b> B via the network 103.

  The server system 104B includes a server unit 110 and an analysis system unit 111 as well as a user information database 112, a sensor information database 113, and a cosmetics database 114, as in the first application example of the first embodiment described above. . The tables included in the databases 112, 113, and 114 are the same as the tables described with reference to FIGS. 2, 8, and 11 in the first embodiment and the first application example. Since the configuration can be applied, a detailed description thereof is omitted here.

The server system 104B is photographed by the camera device 101A or the camera device 101B based on the user ID and history information read from the user medium 15 ′ by the reader 141 and transmitted via the network 103, and transmitted via the network 103. Data corresponding to a predetermined user or data corresponding to a predetermined situation can be identified from the image data thus obtained.

4-2-2. Schematic Process Flow FIG. 34 is a flowchart showing a process flow of an example of the makeup support system according to the second modification of the first embodiment. In the flowchart of FIG. 34, the same reference numerals are assigned to the processes common to the flowchart of FIG. 6 described above, and detailed description thereof is omitted. The flowchart according to FIG. 34 is different from the flowchart of FIG. 6 described above in that a data specifying process based on history information and user ID read from the user medium 15 ′ is added, and accordingly, a slight change is added. Yes.

  The user has the ideal makeup applied at the facility 100A (step S20), and the camera device 101A captures a predetermined face with the ideal makeup. The image data obtained by photographing is transmitted to the server system 104B via the network 103 together with time information indicating the time when photographing is performed and a sensor ID for identifying the camera device 101A (step S21). At this time, for example, the user writes time information indicating the current time and a sensor ID corresponding to the camera device 101A as history information to the user medium 15 'by the writer 140A.

  The user later performs self-makeup, for example, at the user's home 100B (step S23), and the camera device 101B captures a predetermined face by the camera device 101B. The image data obtained by shooting is added with time information indicating the time of shooting and a sensor ID for identifying the camera device 101A, and is transmitted to the server system 104 via the network 103 (step S24). Supplied to the unit 110. At this time, for example, the user writes time information indicating the current time and a sensor ID corresponding to the camera device 101B as history information to the user medium 15 'by the writer 140B.

  The server system 104B supplies the transmitted image data to the server unit 110, and the server unit 110 stores the image data in the sensor information database 113 based on the sensor ID and time information.

  Next, based on the history information and the user ID stored in the user medium 15 ′, data specifying processing corresponding to the user ID is performed on the data stored in the sensor information database 113 (step S 202). That is, for example, the user performs processing for causing the reader 141 to read the history information and the user ID stored in the user medium 15 '(step S210 in FIG. 32). The history information and user ID read by the reader 141 are transmitted to the server system 104B via the network 103. The server system 104B supplies the history information and the user ID to the server unit 110. The server unit 110 specifies data corresponding to the user ID from the data stored in the sensor information database 113 based on the supplied history information and user ID (step S211 in FIG. 32).

  More specifically, the user performs a reading operation of the history information stored in the user medium 15 'by the reader 141 (step S210). The history information read from the user medium 15 ′ to the reader 141 is transmitted to the server system 104 </ b> B via the network 103 and supplied to the server unit 110. The server unit 110 refers to the sensor information table in the sensor information database 113 based on the supplied history information, and identifies data corresponding to the sensor ID and time information (step S211).

  In the data specifying process in step S202 in FIG. 34, image data by ideal makeup and image data by self-make are specified. The distinction between image data based on ideal makeup and image data based on self-makeup can be determined based on, for example, a sensor ID.

  When the image data by ideal makeup and the image data by self-make are specified in step S202A, the process proceeds to step S251. In step S251, face detection is performed on the image data based on the ideal makeup specified in step S202A, and the detected face image is characterized by features appropriate for determining the makeup state. Extraction is performed to generate reference information. The generated reference information is registered in the reference information table in association with the user ID. Further, face detection is performed on image data obtained by self-makeup, and feature extraction is performed by specializing in features suitable for judging the makeup state. Then, the extracted feature information and the corresponding reference information are compared to perform primary analysis.

  The subsequent processing is the same as the processing after step S25 described with reference to FIG. That is, as a result of the primary analysis in step S25, if the difference based on the reference information and the image data by self-make is greater than or equal to a predetermined value (step S26), the server unit 110 performs further secondary analysis processing based on the image data by self-make. This is performed (step S27). As the secondary analysis processing, the secondary analysis processing in the first application example of the first embodiment described above can be applied as it is. Advice information is created based on the secondary analysis result and transmitted to the user.

  The sending of advice information to the user and the presentation to the user using the website can be performed in the same manner as in the first application example of the first embodiment described above. Description is omitted.

  Also in the application example of the first modification of the first embodiment, the user can use the advice information transmitted from the server unit 110 as in the first application example of the first embodiment described above. Based on this, it is possible to perform self-makeup that is closer to ideal makeup while at home.

  In the above description, the user ID is stored in advance for the user medium 15 ′. However, this is not limited to this example. For example, when the user medium 15 ′ is read by the reader 141, a predetermined ID is used. You may make it input user ID using an input means. In the above description, reading of data from the user medium 15 ′ by the reader 141 is performed at the stage of the primary analysis process, but this is not limited to this example. For example, data can be read from the user medium 15 ′ by the reader 141 every time the camera devices 101 </ b> A and 101 </ b> B are photographed.

4-2-3. Application of First Embodiment to Second and Third Application Examples As is apparent from the description using the flowchart of FIG. 34, the second modification example of the first embodiment and the above-described example The first embodiment is different from the first embodiment only in the process related to data identification. Other processes related to the subject of the present invention, that is, the results of the primary analysis and the primary analysis based on the image data obtained by photographing the user. The determination on whether or not to perform the secondary analysis by the threshold determination based on, and the notification based on the secondary analysis and the secondary analysis result are common. Accordingly, in the same manner as the application example according to the second embodiment of the first embodiment, the second and third application examples of the first embodiment described above and the first application of the first embodiment. It can be applied to the second modification.

  For example, when the second application example of the first embodiment is applied to the first modification of the first embodiment, the readers 163A and 163B are omitted in the system configuration of FIG. A writer is added to facilities 160 and 164, respectively. In addition, a reader for reading history information and user ID from the user medium 15 ′ is provided and connected to the network 103. In the flowchart of FIG. 15, the processing of step S32 and step S35 is omitted, and the flowchart of FIG. 32 is used for the video data of the ideal form and the self-form taken in step S31 and step S34 after step S34. The data specifying process in step S202 described above is performed.

  Similarly, when the third application example of the first embodiment is applied to the second modification of the first embodiment, the readers 173A and 173B are omitted in the system configuration of FIG. In addition, a writer is added to the facility 170 and the user's home 174, respectively. In addition, a reader for reading history information and user ID from the user medium 15 ′ is provided and connected to the network 103. In the flowchart of FIG. 20, the processing of step S42 and step S45 is omitted, and the video data of the ideal utterance form and the self utterance form photographed in step S41 and step S44 after step S44 is shown in the flowchart of FIG. The data specifying process in step S202 described with reference to FIG.

5. Second Embodiment of the Invention Next, a second embodiment of the present invention will be described with reference to the drawings. As shown in FIG. 35, the second embodiment is a system according to the present invention on one computer device (PC) 200 such as a personal computer that can operate independently without depending on the network. Is realized.

5-1. System Overview The service provider side creates predetermined reference information based on image data photographed by the camera device 201, for example, records it on the recording medium 203, and provides it to the user. The recording medium 203 displays a program for creating data corresponding to the reference information on the user side, a program for performing predetermined primary analysis and secondary analysis, and a message corresponding to the result of the secondary analysis. The software 202 including the program is recorded together with the reference information. In addition to these, the software 202 can include other programs necessary for services such as a camera control program and a data management program. Note that the reference information and the software 202 may be recorded on different recording media 203 and provided to the user.

  The recording medium 203 may be of any type as long as it can be read by the computer apparatus 200. For example, a read-only type DVD (Digital Versatile Disc) or CD (Compact Disc), or a recordable type DVD or CD can be used as the recording medium 203. A non-volatile semiconductor memory that can be attached to and detached from the computer device 200 such as a flash memory may be used as the recording medium 203, or a hard disk that is removable from the computer device 200 may be used as the recording medium 203. Hereinafter, the recording medium 203 is a disc-shaped recording medium such as a DVD or a CD.

  On the user side, the recording medium 203 is loaded into the computer apparatus 200 in a predetermined manner, and the software 202 is activated. For example, the camera device 204 connected to the computer device 200 shoots itself. Image data obtained by photographing is supplied to the computer apparatus 200.

  In the computer device 200, predetermined processing is performed on the image data supplied from the camera device 204 by an image data processing program included in the software 202 to create data corresponding to the reference information. Using the created data and the reference information read from the recording medium 203, a primary analysis process is performed by a primary analysis program. Then, the result of the primary analysis process is determined as a threshold, and if it is determined that the threshold has been exceeded, a secondary analysis is performed and a message based on the analysis result is displayed. If it is determined that the result of the primary analysis process does not exceed the threshold value, a message indicating that is displayed.

  FIG. 36 shows an exemplary configuration of the computer apparatus 200. As illustrated in FIG. 36, the computer apparatus 200 can apply a general configuration, and a CPU (Central Processing Unit) 211, a RAM (Random Access Memory) 212, a ROM (Read Only Memory) are connected to the bus 210. ) 213, a display control unit 214, and a communication I / F (interface) 216, and a hard disk (HD) 217, a drive device 218, an input device I / F 219, and a stream input I / F 220 are connected to the bus 210. Each is connected.

  The CPU 211 uses the RAM 212 as a work memory, for example, and controls each part of the computer device 200 according to a program stored in advance in the ROM 213 or a program stored in the hard disk 217. The display control unit 214 converts the display control signal generated by the CPU 211 into a signal in a format that can be displayed on the display 215, and supplies the signal to the display 215. The communication I / F 216 implements, for example, TCP / IP (Transmission Control Protocol / Internet Protocol) as a communication protocol, and controls communication with the Internet according to a command from the CPU 211.

  The drive device 218 is loaded with a detachable recording medium 203 in a predetermined manner, and writes data to the recording medium 203 and reads data from the recording medium 203 in accordance with instructions from the CPU 211. The input device I / F 219 is connected to input devices such as a mouse 219A and a keyboard 219B, and supplies signals output from these input devices to the CPU 211 in a predetermined manner. For example, a microphone 221 and a camera device 204 are connected to the stream input I / F 220, and audio data based on the output of the microphone 221 and image data output from the camera device 204 are supplied to the CPU 211 in a predetermined manner.

  Note that the input device I / F 219 and the stream input I / F 220 can be configured by, for example, a USB (Universal Serial Bus) interface. As the stream input I / F, an interface conforming to the regulations of IEEE 1394 (Institute Electrical and Electronics Engineers 1394) can be used. Of course, other standard interfaces may be used as the input device I / F 219 and the stream input I / F 220.

  The hard disk 217 stores various data, programs that run on the CPU 211, and the like. The program is provided by being recorded in, for example, the recording medium 203, and the program read from the recording medium 203 loaded in the drive device 218 is stored in the hard disk 217 in a predetermined manner and installed to be executable. However, the present invention is not limited to this, and the program can be executed from the recording medium 203. Audio data based on the output of the microphone 221 and image data output from the camera device 204 are temporarily stored in the hard disk 217.

  For example, a video camera that can capture a moving image and output video data can be applied to the camera device 204 under the control of the CPU 211. Not limited to this, a still camera that outputs a still image may be used as the camera device 204. Further, the microphone 221 can output the collected sound as digital audio data. However, the present invention is not limited thereto, and an analog audio signal may be output from the microphone 221, converted into digital audio data by an A / D converter (not shown), and supplied to the stream input I / F 220.

  The camera device 204 may be a digital video camera that is independent of the computer device 200 and can operate independently. In this case, the image data is supplied to the computer device 200 via a predetermined recording medium and stored in the hard disk 217 in a predetermined manner.

5-2. FIG. 37 is a flowchart showing an example of processing according to the second embodiment. Prior to the processing of this flowchart, it is assumed that reference information and software 202 are recorded in advance on the recording medium 203 and this recording medium 203 is loaded in the drive device 218 of the computer apparatus 200. Furthermore, it is assumed that the software 202 is read from the recording medium 203 by the drive device 218 and stored and installed in the hard disk 217 so as to be executable.

  In step S50, the software 202 is activated in response to, for example, a user operation on the computer apparatus 200. The following processing is executed based on the control of the software 202. In the next step S51, the reference information recorded on the recording medium 203 is read by the software 202. The read reference information is temporarily stored in the hard disk 217, for example.

  Next, in step S52, user data is captured. For example, the camera device 204 captures the user as a subject so as to correspond to the reference information, and the image data output from the camera device 204 by the capture is supplied to the computer device 200 and stored in the hard disk 217 in a predetermined manner. When the user data is captured, in step S53, a predetermined process is performed on the captured user data, and data corresponding to the reference information is obtained.

  In the next step S54, a primary analysis process is performed on the data obtained in step S53. For example, the reference information read from the recording medium 203 in step S51 is compared with the data obtained in step S53, and the primary analysis process is performed. Based on the result of the primary analysis process, a threshold value is determined in step S55. For example, in step S54, a difference between the reference information and the data obtained in step S53 is calculated in a predetermined manner, and it is determined in step S55 whether this difference has exceeded a threshold value.

  If it is determined in step S55 that the difference has exceeded the threshold, the process proceeds to step S56, and further a secondary analysis process is performed. In this secondary analysis process, for example, the analysis process may be further performed on the data obtained in step S53, or the analysis process may be performed on the original data of the data.

  In step S57, a predetermined message based on the result of the secondary analysis in step S56 is displayed on the display 215. For example, the CPU 211 creates message data in accordance with the result of the secondary analysis, and generates a display control signal for causing the display 215 to display this message data. This display control signal is supplied to the display control unit 214, converted into a signal in a format that can be displayed on the display 215, and supplied to the display 215.

  On the other hand, if it is determined in step S55 that the difference does not exceed the threshold value, the process proceeds to step S58. In step S58, for example, a message indicating that there is no problem with the captured user data is displayed on the display 215.

  In the above description, user data is captured in step S52 by supplying image data captured using the camera device 204 connected to the computer device 200 to the computer device 200. It is not limited to. For example, image data captured by a digital camera device or a digital video camera device that is independent of the computer device 200 and used by itself may be taken into the computer device 200.

  Also, the order of the steps in the flowchart of FIG. 37 is not limited to the order described above. For example, the user data acquisition in step S52 may be performed at any timing before the processing of step S53, and may be performed before the activation of the software 202 in step S50 as an example. In this case, the captured image data is temporarily stored in the hard disk 217, and the image data is read from the hard disk 217 during the predetermined processing in step S53. Similarly, reading of the reference information from the recording medium 203 is not limited to the order described above. That is, the reference information only needs to be read from the recording medium 203 before the comparison process between the processing result for the user data and the reference information in step S54, for example.

  Further, in the above description, the reference information that has been subjected to the predetermined processing is recorded on the recording medium 203, but this is not limited to this example. For example, the original data for obtaining the reference information may be recorded in the recording medium 203, and the reference information may be generated by the software 202 based on the original data after the software 202 is started in the computer apparatus 200.

5-3. Application Example of Second Embodiment 5-3-1. First application example (example applied to makeup support system)
This second embodiment can also be applied to various services in the same manner as the first embodiment described above. As a first application example, consider a case where the second embodiment is applied to the makeup support system described in the first embodiment and the first application example in each modification described above.

  When the second embodiment is applied to a makeup support system, for example, an ideal makeup is applied to a user at a cosmetic corner of a department store, a makeup studio, or the like, and the face with the ideal makeup is photographed by the camera device 201. . Then, a predetermined process is performed on image data by ideal makeup obtained by photographing using a computer device (not shown) to create reference information corresponding to the user. The reference information and the software 202 described above are packaged, recorded on the recording medium 203, and provided to the user.

  Here, a program included in the software 202 for creating data corresponding to the reference information on the user side, a program for performing predetermined primary analysis and secondary analysis, and a message corresponding to the result of the secondary analysis are displayed. Each program based on the program to be displayed is the first application example of the first embodiment described above, and the same processing as that described with reference to FIGS. It is assumed that execution is possible.

  It is also conceivable to further package advertisement information such as cosmetics on the recording medium 203 together with the software 202 and reference information. Further, when the computer device 200 can be connected to the Internet, link information to a cosmetics sales site on the Internet can be packaged together with the software 202 and reference information.

  By using the reference information recorded on the recording medium 203 and the software 202 when the user performs self-makeup, the user can easily make makeup that is closer to the ideal makeup while at home.

5-3-2. Second application example (example applied to a sports learning system)
As a second application example, consider a case where the second embodiment is applied to the sports learning system described in the first embodiment described above and the second application example in each modification. Also in this case, the second embodiment can be applied based on the same concept as the first application example described above. That is, the user's ideal form is photographed by the camera device 201 which is a video camera. Then, the video data obtained by shooting is subjected to a predetermined process by a computer device (not shown) to create reference information corresponding to the user. The reference information and the software 202 described above are packaged, recorded on the recording medium 203, and provided to the user.

  Here, a program included in the software 202 for creating data corresponding to the reference information on the user side, a program for performing predetermined primary analysis and secondary analysis, and a message corresponding to the result of the secondary analysis are displayed. It is assumed that each program based on the program to be displayed can execute on the computer device 200 the same processing as the processing described in the second application example of the first embodiment described above.

  It is also conceivable that an ideal form by an instructor, not a user, is taken to create reference information. In this case, if there is a physique difference between the user and the instructor, there is a possibility that an accurate determination cannot be made in the primary analysis. Therefore, based on the physique difference between the instructor and the user, a program for normalizing position information indicating the position of each measurement point of the instructor according to the user's physique is added to the software 202. However, the instructor's ideal form may be used as reference information as it is without normalization by appropriately setting a threshold value for making a determination in the primary analysis. The recording medium 203 on which the reference information based on the ideal form by the instructor and the software 202 including the program for performing the normalization process or the threshold adjustment according to the physique difference is limited to a specific user. Instead, it can be a general-purpose application.

  It is also conceivable to further package advertisement information such as sports equipment on the recording medium 203 together with the software 202 and reference information. Further, when the computer device 200 can be connected to the Internet, link information to a sports equipment sales site on the Internet can be packaged together with the software 202 and reference information.

  The user can easily wear a form closer to the ideal form by using the reference information recorded on the recording medium 203 and the software 202 at the time of training in a home or other personal use facility. .

5-3-3. Third application example (example applied to a learning system for foreign language conversation)
As a third application example, a case is considered in which the second embodiment is applied to the foreign language conversation learning system described in the third application example of each modification of the first embodiment described above. In this case, an ideal utterance form by the instructor is captured by the camera device 201 which is a video camera to obtain video data, and the instructor's pronunciation is collected as an ideal utterance pronunciation by a microphone (not shown) in synchronism with the capturing and audio data is collected. Get. Then, predetermined processing is performed on the video data and the audio data by a computer device (not shown) to create reference information for each of the ideal utterance form (video data) and the ideal utterance pronunciation (audio data). The reference information and the software 202 described above are packaged, recorded on the recording medium 203, and provided to the user.

  Here, a program included in the software 202 for creating data corresponding to the reference information on the user side, a program for performing predetermined primary analysis and secondary analysis, and a message corresponding to the result of the secondary analysis are displayed. Each program based on the program to be displayed is assumed to be able to execute on the computer device 200 the same processing as the processing described in the third application example of the first embodiment described above.

  It is also conceivable to further package advertisement information such as other course guidance such as the foreign language conversation school together with the software 202 and the reference information in the recording medium 203. Further, when the computer device 200 can be connected to the Internet, link information for a site for applying for a course change on the Internet can be packaged together with the software 202 and reference information.

  The user can easily learn the ideal utterance form and the ideal utterance pronunciation by performing self-lesson of foreign language conversation using the reference information recorded on the recording medium 203 and the software 202 at home or the like. .

6). Third Embodiment of Invention 6-1. Outline Next, a third embodiment of the present invention will be described. The third embodiment of the present invention is an example in which the present invention is applied to a security system. A third embodiment of the present invention will be schematically described with reference to FIGS. First, a plurality of camera devices C ₁ , C ₂ ,... Are installed in a city, a residential area, etc. so as to be able to take a picture of the street without any trouble, and connected to a server system (not shown) via a wired or wireless network. . The camera devices C ₁ , C ₂ ,... Shoot images at predetermined time intervals, and the captured image data is sensor IDs for identifying the camera devices C ₁ , C ₂ ,. It is transmitted to the server system together with time information indicating the time when photographing was performed. The server system extracts an image of the target user from the transmitted image data using a technique such as pattern detection or face detection.

Here, it is assumed that a user who is a security target (hereinafter referred to as a “target user”) is walking along a route indicated by a route A in FIG. When the target user moves without staying in one place, the image data output from each camera device C ₁ , C ₂ ,... An image is considered to be included. That is, in the example of FIG. 39, the appearance of the target user is first captured by the camera device C ₁ (time t _{a to} t _b ) and then captured by the camera device C ₂ (time t _{c to} t _d ). Finally, it is captured by the camera device C ₅ (time t _{i to} t _j ). In the portion where the shooting range overlaps, the shooting is performed overlappingly. In this example, the camera devices C ₃ and C ₄ also capture the target user for a short time (time t _{e to} t _f and time t _{g to} t _h ).

Here, for example, as illustrated in FIG. 40, when the target user is captured for a very long time by a specific camera device (camera device C _{5 in} the example of FIG. 40), there is some accident in the target user. It can be considered that it may have occurred. Therefore, in the third embodiment of the present invention, the behavior of the target user is analyzed based on the image data (primary analysis), and if it is determined that there is no movement for a predetermined time or more, the secondary processing is performed on the image data. Analyze and analyze the situation of the target user. As a result of analysis, if it is determined that there is a high possibility that some accident has occurred in the target user, a message indicating that is transmitted to other users related to the target user.

6-2. Applicable System FIG. 41 schematically shows an example system applicable to the third embodiment. A plurality of camera devices 201 </ b> A, 201 </ b> B,... Are installed at predetermined positions in a service target area such as a city or a residential area. The installation locations and orientations of the camera devices 201A, 201B,... In addition, it is more preferable that the camera devices 201A, 201B,... Can capture not only light in the visible light region but also light in the infrared region.

  Each of the camera devices 201A, 201B,... Is connected to the network 200. Each of the camera devices 201A, 201B,... Shoots at predetermined time intervals, and transmits image data obtained by the shooting to a server system 202 described later via the network 200. At this time, a sensor ID for identifying each of the camera devices 201A, 201B,..., And time information indicating the time at which shooting was performed are transmitted in association with the image data.

  The network 200 may be a network closed by this system as described above, or may be an open network such as the Internet.

  A server system 202 is further connected to the network 200. The server system 202 includes a server unit 210 and an analysis system unit 211, and also includes a user information database 212 and a sensor information database 213. The server unit 210 controls and manages the entire system. For example, the server unit 210 receives data transmitted from the camera devices 201A, 201B,... Via the network 200, and stores the data in the sensor information database 213 and the user information database 212 in a predetermined manner. For example, image data output from the camera devices 201A, 201B,... And received via the network 200 is stored in the sensor information database 213 in association with the sensor ID and time information by the server unit 210. .

  The analysis system unit 211 performs predetermined analysis processing on the image data stored in the sensor information database 213 (primary analysis processing), and if it is determined that the analysis is necessary based on the analysis result, Further analysis processing is performed (secondary analysis processing).

  The server system 202 is connected to the Internet 203 by the server unit 210. The Internet 203 includes, for example, user terminals 220A, 220B,... Used by other users related to the target user, for example, a guardian of the target user, a person in charge such as a facility, a reporting organization, etc., a portable terminal 221A, 221B,... Are connected. The user terminals 220A, 220B,... Can be of any type as long as communication over the Internet 203 is possible. For example, the user terminals 220A, 220B,. Software necessary for using the Internet 203, such as a browser, is installed. The mobile terminals 221A, 221B,... Are, for example, mobile phone terminals or PDAs (Personal Digital Assistants) capable of communication via the Internet 203, and are similar to the user terminals 220A, 220B,. , Equipped with mailer and browser. The server unit 210 can communicate with the user terminals 220A, 220B,... And the mobile terminals 221A, 221B,.

  The user information database 212 and the sensor information database 213 will be schematically described. The user information database 212 is, for example, a user information table that stores information about the target user, a user related information table that defines the relationship between the target user and other users, and information for identifying the target user based on image data Is stored in the personal identification information table.

  In the user information table, user attributes such as the user name, address, and contact address (e-mail address) are associated with the user ID for identifying the user, as already described with reference to FIG. . Also, in the identity identification information table, as already described with reference to FIG. 26, information indicating the facial features of the user is associated with the user ID. In the third embodiment, the features of the user's face may be extracted not only from the front of the face but also from the side of the face or from an oblique direction. Furthermore, it is also conceivable to extract features as a whole for the whole body of the target user. In this third embodiment, it is only necessary that the personal identification information is registered only for the target user.

  FIG. 42 shows an example of the configuration of the user related information table. In the third embodiment, a group is formed with respect to one target user, and a relation to the target user of the user related to the target user is registered in the user related information table, and a level at which notification is performed. Is registered. In the example of FIG. 42, the level “1” and the higher level “2” are set. The relationship with the target user may be a guardian of the target user, a person in charge at a facility related to the target user, a reporting organization, or the like. The reporting organization can be the police or an organization established to protect children and the elderly in the community. The level for performing the notification will be described later.

  The sensor information database 213 has a sensor information table. FIG. 43 shows an exemplary configuration of the sensor information table. In the sensor information table, for each sensor ID, image data is identified by a unique data ID, and is registered in association with time information. The image data is, for example, a pointer that indicates a storage location of actual image data. In the following description, for the sake of explanation, it is assumed that the image data main body has image data registered directly in the sensor information table.

6-3. Schematic Process Flow Next, an example of the process according to the third embodiment is schematically shown using the flowcharts of FIGS. 44 to 46. These processes of FIGS. 44 to 46 are executed in parallel.

  FIG. 44 shows an example of processing in each of the camera devices 201A, 201B,. As described above, each of the camera devices 201A, 201B,... Shoots (step S300), and the captured image data is sensor IDs for identifying the camera devices 201A, 201B,. Along with time information indicating the time at which the image was taken, the information is transmitted to the server system 202 via the network 200 (step S301). The processes in step S300 and step S301 are repeated at predetermined time intervals. The transmitted data is received by the server system 202, supplied to the server unit 210, and stored in the sensor information database 213 in a predetermined manner.

  FIG. 45 shows an example of processing in the server unit 210. The server unit 210 sequentially reads image data from the sensor information database 213 (step S310), and extracts an image of the target user for each read image data (step S311). For example, the server unit 210 performs face detection on the image data and extracts features of the detected face image. Of the facial feature information extracted from the image data, the facial feature information corresponding to the target user stored in the personal identification information table in the user information database 212 has a similarity higher than a predetermined value. The target user is identified as an image.

  The method of extracting the image of the target user from the image data is not limited to this. For example, a human figure may be extracted from the image data by pattern detection or the like and compared with features based on the whole image of the target user. It is also conceivable to specify the target user using both the face image and the whole body image.

  When the image of the target user is extracted from each image data, the image data from which the image of the target user is extracted is arranged in time order in the next step S312.

  In step S310 described above, the server unit 210 temporarily stores the image data read from the sensor information database 213 in a predetermined storage device (not shown) included in the server unit 210. Then, for the image data stored in the predetermined storage device, the target user image extraction process in step S311 and the image data rearrangement process in step S312 are performed.

  FIG. 46 shows an example of processing by the analysis system unit 211. In the first step S320, the analysis system unit 211 sets a predetermined temporally continuous image data among the image data arranged in time order by the process of step S312 in the flowchart of FIG. The primary analysis is performed on the image data of the sheet, and the movement of the target user is detected.

For example, as illustrated in FIG. 47, the image t _11, t _12, t ₁₃ is an image captured by the camera device C _1, ..., t _18, is the image captured by ... and the camera device C ₂ image _{t 21, t 22, t 23} , ···, t 28, out of the ..., and from the image t ₁₁ ~t ₁₄ and the image t ₂₄ ~t ₂₈ as an image of the target user is extracted ( In FIG. 46, it is shown with hatching. These images are rearranged in the order of images t ₁₁ , t ₁₂ , t ₁₃ , t ₂₄ , t ₁₄ , t ₂₅ , t ₁₅ , t ₂₆ , t ₂₇ , t ₂₈ to determine the moving speed of the target user. It should be noted that position correction in the image due to the difference in installation positions of the camera devices C ₁ and C ₂ is performed in advance.

A predetermined number of images that are continuous in time among the images t ₁₁ , t ₁₂ , t ₁₃ , t ₂₄ , t ₁₄ , t ₂₅ , t ₁₅ , t ₂₆ , t ₂₇ , and t ₂₈ from which the image of the target user has been extracted. This image data is the target of analysis. Here, it is assumed that the predetermined number is 5, and the moving speed of the target user is obtained using the images t ₁₁ , t ₁₂ , t ₁₃ , t ₂₄ , and t ₁₄ . As the moving speed, for example, it is conceivable to use an average value of values obtained by dividing the moving distance of the target user between the images by time and performing this on all the images to be analyzed. The moving speed may be obtained using only the first and last images in terms of time.

  When the moving speed of the target user is obtained, it is determined in next step S321 whether or not the moving speed exceeds a predetermined threshold value. If it is determined that the threshold value has been exceeded, the process returns to step S320, assuming that the target user has no abnormality. On the other hand, if it is determined that the moving speed does not exceed the predetermined threshold, the process proceeds to step S322, and the secondary analysis process is performed. That is, when the moving speed is equal to or lower than the predetermined threshold, it is considered that some accident may have occurred in the target user as described with reference to FIG.

  Various processes can be considered as the secondary analysis process in step S322. As an example, it is conceivable to analyze the posture of the target user. That is, the attitude of the target user with respect to each image to be analyzed is analyzed. For example, when an analysis result such as the target user crouching or falling is obtained, it can be said that there is an extremely high possibility that some accident has occurred in the target user.

  As another example, it is conceivable to analyze the color of the target user. For example, a color is detected for each portion of the target user for each image to be analyzed. And when the part which shows a red color generate | occur | produces at a certain time, it can be considered that the object user is shed blood.

  As yet another example, when each of the camera devices 201A, 201B,... Is capable of photographing not only the light in the visible light region but also the light in the infrared region, a value corresponding to the temperature of the target user, that is, the body temperature. Can be analyzed. If, for example, a decrease in body temperature is detected at a certain point in time, it can be considered that some accident may have occurred in the target user.

  As a result of the analysis in step S322, if it is determined that there is a possibility that some accident has occurred in the target user (step S323), the process proceeds to step S324. If it is determined that no accident has occurred, the process returns to step S320. In step S324, a message indicating that some accident has occurred to the target user is notified to other users registered in association with the target user in the user related information table. For example, information on other users is retrieved from the user information table, and an e-mail in which a message is described is transmitted to an e-mail address registered as a contact address.

  At this time, it is more preferable to attach the image data as the basis for determining that an accident has occurred to the target user in the secondary analysis to the e-mail in which the message is described, and package the message and the image data. . In other words, the recipient of this e-mail can immediately know what accident occurred to the target user and whether or not the accident really occurred by looking at the image data attached to the message. it can.

  For example, in the predetermined storage device included in the server unit 210, the above-described processing of step S320 to step S324 is repeated each time an image of the target user is extracted from one or more image data and arranged in time order.

  In addition, as a result of the secondary analysis, there may be a case where it is not clear whether an accident has occurred for the target user. In such a case, for example, when a message is transmitted to a reporting organization or the like, the burden on the reporting organization may increase unnecessarily. Therefore, it is preferable that a partner to which a message is transmitted can be selected according to the result of the secondary analysis.

  For example, if it is determined as a result of the secondary analysis that there is an extremely high possibility that an accident has occurred in the target user, a message etc. is sent to the user set at a higher level in the user related information table shown in FIG. Is sent. In the example of FIG. 42, a message or the like is transmitted to the guardians 1 and 2, the person in charge, and the reporting organization set to the level "1" or "2". On the other hand, if the result of the secondary analysis cannot clearly determine whether or not an accident has occurred in the target user, a message or the like is transmitted only to the user set at a low level in the user related information table. In the example of FIG. 42, the message is transmitted only to the parents 1 and 2 set at the level “1”. Parents 1 and 2 who have received the message can determine whether or not an accident has occurred in the target user by checking the image data attached to the message.

1 is a schematic diagram schematically showing an example system configuration according to a first embodiment of the invention; It is a basic diagram which shows the structure of an example of a sensor information table. It is a basic diagram which shows the structure of an example of a user information table. It is a basic diagram which shows the structure of an example of a reference information table. It is a flowchart which shows the process of an example by the 1st Embodiment. It is a flowchart which shows the flow of an example for demonstrating the outline | summary of a makeup | decoration support system. It is an approximate line figure showing roughly an example system applicable to the 1st example of application of a 1st embodiment. It is a basic diagram which shows the structure of an example of a user information table. It is a basic diagram which shows the structure of an example of package data. It is a basic diagram for demonstrating the example of a measurement point. It is a basic diagram which shows the structural example of a reference information table. It is a basic diagram which shows the screen structure of an example of the web page which browses the image data, advice information, and advertisement information contained in package data. It is a basic diagram which shows another example of the display by a web page. It is a basic diagram which shows the structure of an example of a cosmetics database. It is a flowchart which shows the flow of an example for demonstrating the outline | summary of the learning system for sports. It is a basic diagram which shows roughly an example system applicable to the 2nd application example of 1st Embodiment. It is a basic diagram which shows the structure of an example of a user information table. It is a basic diagram for demonstrating the example of a measurement point. It is a basic diagram which shows the structural example of a reference table. It is a flowchart which shows the flow of an example for demonstrating the outline | summary of a foreign language conversation learning system. It is a basic diagram which shows roughly an example system applicable to the 3rd application example of 1st Embodiment. It is a basic diagram for demonstrating the structure of an example of a sensor information database. It is a basic diagram which shows the example of the analysis method of audio data. It is a basic diagram which shows the structure of an example of a reference information table. It is an approximate line figure showing roughly the system configuration of an example by the 1st modification of a 1st embodiment. It is a basic diagram which shows the structure of an example of a principal identification information table. It is a flowchart which shows the identification method of an example of the user by the 1st modification of the 1st Embodiment. It is a basic diagram which shows roughly the system configuration of an example which applied the makeup | decoration support system which is the 1st application example of 1st Embodiment to the 1st modification of 1st Embodiment. It is a flowchart which shows the flow of a process of an example of the makeup | decoration support system by the application example in the 1st modification of 1st Embodiment. It is a basic diagram which shows the system configuration | structure of an example by the 2nd modification of the 1st Embodiment. It is a basic diagram which shows the structure of an example of the historical information memorize | stored in a user medium. It is a flowchart which shows the method of an example of the data specification based on historical information. It is a basic diagram which shows roughly the system configuration | structure of an example which applied the makeup | decoration support system which is the 1st application example of 1st Embodiment to the 2nd modification of 1st Embodiment. It is a flowchart which shows the flow of a process of an example of the makeup | decoration support system in the 2nd modification of 1st Embodiment. It is a basic diagram for demonstrating schematically the 2nd Embodiment of invention. It is a block diagram which shows the structure of an example of a computer apparatus. It is a flowchart which shows the process of an example by the 2nd Embodiment. It is a basic diagram for demonstrating schematically the 3rd Embodiment of invention. It is a basic diagram for demonstrating schematically the 3rd Embodiment of invention. It is a basic diagram for demonstrating schematically the 3rd Embodiment of invention. It is a basic diagram which shows roughly an example system applicable to the 3rd Embodiment. It is a basic diagram which shows the structure of an example of a user related information table. It is a basic diagram which shows the structure of an example of a sensor information table. It is a flowchart which shows an example process in each camera apparatus. It is a flowchart which shows an example of a process in a server part. It is a flowchart which shows an example process by an analysis system part. It is an approximate line figure for explaining an example analysis method by an analysis system part.

Explanation of symbols

11A, 11B, 11C, 11D Sensor device 12A, 12B, 12C, 12D Reader 13 Network 14 Server system 15 User medium 16A, 16B, 16C, 16B Writer 17 Reader 20 Server unit 21 Analysis system unit 22 User information database 23 Sensor information database 23 30 Internet 31A, 31B, 31C User terminal 100A Facility 100B User home 101A, 101B Camera device 102A, 102B Reader 104 Server system 110 Server unit 111 Analysis system unit 112 User information database 113 Sensor information database 114 Cosmetics database 130 Internet 132 Web server 140 Package data 161A, 161B, 162A, 162B Video cameras 172A, 172B Ecrophone 200 Computer device 202 Software 203 Recording medium

Claims (35)

In an information analysis device that outputs information according to the result of analyzing image data,
An image detection unit for detecting an image related to the user's body from the image data;
A first analysis unit that performs a first predetermined analysis process based on an image related to the user's body detected by the image detection unit;
A comparison unit that compares the analysis result by the first analysis unit with a threshold;
A second analysis unit that performs a second predetermined analysis process based on an image related to the user's body detected by the image detection unit;
An information analysis apparatus that determines whether or not to perform the second predetermined analysis process by the second analysis unit based on a comparison result by the comparison unit. In the information analysis device according to claim 1,
The image detection unit
An information analysis apparatus for detecting an image of a user's face from the image data. In the information analysis device according to claim 1,
The image detection unit
An information analysis apparatus for detecting an overall image of a user's body from the image data. In the information analysis device according to claim 1,
The first predetermined analysis process by the first analysis unit is:
An information analysis apparatus characterized by being performed based on a difference between a value based on a reference image corresponding to an image related to the user's body and a value based on an image related to the user's body. In the information analysis device according to claim 1,
The first predetermined analysis process by the first analysis unit is:
An information analysis apparatus, which is performed based on temporal changes in an image related to the user's body. In the information analysis device according to claim 1,
The first predetermined analysis process by the first analysis unit is:
An information analysis apparatus based on a value obtained by measuring a predetermined position in an image related to the user's body. In the information analysis device according to claim 6,
The first predetermined analysis process by the first analysis unit is:
An information analysis apparatus based on a color at a predetermined position in an image related to the user's body. In the information analysis device according to claim 6,
The first predetermined analysis process by the first analysis unit is:
An information analysis apparatus based on a movement of a predetermined position in an image related to the user's body. In the information analysis device according to claim 1,
The second predetermined analysis process by the second analysis unit is:
An information analysis apparatus, which is performed based on the first predetermined analysis processing result by the first analysis unit. In the information analysis device according to claim 9,
The second predetermined analysis process by the second analysis unit is:
An information analysis apparatus characterized in that, based on a comparison result by the comparison unit, the analysis process by the first analysis unit is a process of analyzing a cause of exceeding the threshold value. In the information analysis device according to claim 1,
The second predetermined analysis process by the second analysis unit is:
An information analysis apparatus, wherein the first analysis unit is performed on data obtained by performing the first analysis processing on an image related to the user's body. In the information analysis device according to claim 1,
The second predetermined analysis process by the second analysis unit is:
An information analysis apparatus, which is performed on an image related to the user's body detected from the image data by the image detection unit. In the information analysis device according to claim 1,
The second predetermined analysis process by the second analysis unit is:
An information analysis apparatus based on a value obtained by measuring a predetermined position in an image related to the user's body. The information analysis apparatus according to claim 13,
The second predetermined analysis process by the second analysis unit is:
An information analysis apparatus based on a color at a predetermined position in an image related to the user's body. The information analysis apparatus according to claim 13,
The second predetermined analysis process by the second analysis unit is:
An information analysis apparatus based on a movement of a predetermined position in an image related to the user's body. In the information analysis device according to claim 1,
An information analysis apparatus, further comprising: a notification creation unit that creates a notification based on a result of the second predetermined analysis process by the second analysis unit. The information analysis apparatus according to claim 16, wherein
The notification created by the notification creation unit is
An information analysis apparatus comprising a message based on the second predetermined analysis processing result. The information analysis apparatus according to claim 16, wherein
The notification created by the notification creation unit is
An information analysis apparatus comprising image data corresponding to the second predetermined analysis processing result among the image data. The information analysis apparatus according to claim 16, wherein
The notification created by the notification creation unit is
An information analysis apparatus comprising advertisement information based on the second predetermined analysis processing result. The information analysis apparatus according to claim 16, wherein
The notification created by the notification creation unit is
An information analysis apparatus comprising: a message based on the second predetermined analysis processing result; and image data corresponding to the second predetermined analysis processing result among the image data. The information analysis device according to claim 20,
The notification created by the notification creation unit is
An information analysis apparatus further comprising advertisement information based on the second predetermined analysis processing result. In the information analysis device according to claim 1,
A voice detector for detecting voice uttered by the user from the voice data;
The first and / or the second analyzing unit performs the first and / or the second analyzing process by further using the voice generated by the user detected by the voice detecting unit. Information analysis device. The information analysis apparatus according to claim 22,
The information analysis apparatus according to claim 1, wherein the first and / or second analysis unit analyzes a rhythm of utterance based on the voice uttered by the user detected by the voice detection unit. The information analysis apparatus according to claim 22,
The information analysis apparatus according to claim 1, wherein the first and / or second analysis unit analyzes the pitch of the utterance for each sound generation portion based on the voice uttered by the user detected by the voice detection unit. The information analysis apparatus according to claim 22,
The information analysis apparatus according to claim 1, wherein the first and / or the second analysis unit analyzes the volume of the utterance for each sound generation portion based on the voice uttered by the user detected by the voice detection unit. In the information analysis device according to claim 1,
A communication unit that communicates with an external device via a network;
The information analysis apparatus, wherein the image data is captured by an external imaging apparatus and supplied via the network. In the information analysis device according to claim 1,
A communication unit that communicates with an external device via a network;
The first predetermined analysis process by the first analysis unit is based on a difference between a value based on a reference image corresponding to an image related to the user's body and a value based on an image related to the user's body. To be done based on
The information analysis apparatus characterized in that the reference image is supplied via the network. In the information analysis device according to claim 1,
A notification creation unit for creating a notification based on a result of the second predetermined analysis process by the second analysis unit;
An information analysis apparatus, further comprising: a notification transmission unit that transmits the notification created by the notification creation unit toward the Internet. The information analysis apparatus according to claim 28, wherein
The notification sending unit
An information analysis apparatus, wherein the notification is transmitted via a website on the Internet. The information analysis apparatus according to claim 28, wherein
A related information management unit that manages the user and information of other users related to the user in association with each other;
The information analysis apparatus, wherein the notification transmission unit transmits the notification to the other user managed by the related information management unit. In an information analysis method for outputting information according to the result of analyzing image data,
An image detection step of detecting an image related to the user's body from the image data;
A first analysis step for performing a first predetermined analysis process based on an image related to the user's body detected by the image detection step;
A comparison step for comparing the analysis result of the first analysis step with a threshold;
A second analysis step of performing a second predetermined analysis process based on an image related to the user's body detected by the image detection step;
An information analysis method comprising: determining whether or not to perform the second predetermined analysis process in the second analysis step based on a comparison result in the comparison step. In an information analysis program for causing a computer to execute an information analysis method for outputting information according to the result of analyzing image data,
The above information analysis method is
An image detection step of detecting an image related to the user's body from the image data;
A first analysis step for performing a first predetermined analysis process based on an image related to the user's body detected by the image detection step;
A comparison step for comparing the analysis result of the first analysis step with a threshold;
A second analysis step of performing a second predetermined analysis process based on an image related to the user's body detected by the image detection step;
An information analysis program for determining whether or not to perform the second predetermined analysis process in the second analysis step based on a comparison result in the comparison step. In a computer-readable recording medium on which an information analysis program for causing a computer to execute an information analysis method for outputting information according to the result of analyzing image data is recorded,
An image detection step of detecting an image related to the user's body from the image data;
A first predetermined analysis process is performed based on a difference between an image related to the user's body detected by the image detection step and a value based on a reference image corresponding to the image related to the user's body. 1 analysis step;
A comparison step for comparing the analysis result of the first analysis step with a threshold;
A second analysis step of performing a second predetermined analysis process based on an image related to the user's body detected by the image detection step;
An information analysis program for causing a computer to execute an information analysis method for determining whether or not to perform the second predetermined analysis process in the second analysis step based on the comparison result in the comparison step;
A computer-readable recording medium on which the reference image and / or a value based on the reference image is recorded. In an information analysis system that outputs information according to the result of analyzing image data,
An imaging device that images a subject and outputs image data;
An image detection unit for detecting an image related to the user's body from the image data;
A first analysis unit that performs a first predetermined analysis process based on an image related to the user's body detected by the image detection unit;
A comparison unit that compares the analysis result by the first analysis unit with a threshold;
A second analysis unit that performs a second predetermined analysis process based on an image related to the user's body detected by the image detection unit;
An information analysis device configured to determine whether to perform the second predetermined analysis process by the second analysis unit based on a comparison result by the comparison unit;
The image data output from the imaging device is transmitted to the information analysis device via a network, and the image detection unit relates to the user's body from the image data transmitted via the network. An information analysis system characterized in that an image to be detected is detected. The information analysis system according to claim 34,
The first predetermined analysis process by the first analysis unit is based on a difference between a value based on a reference image corresponding to an image related to the user's body and a value based on an image related to the user's body. To be done based on
The information analysis system, wherein the reference image is supplied to the information analysis device via the network.

Images