JP6670677B2 - Technical support apparatus, method, program and system - Google Patents

Description

  Embodiments of the present invention relate to a technical support device, a method, a program, and a system.

  A user may perform work and technology while referring to an application or web content on a terminal such as a smartphone or a tablet.

For example, when a user performs makeup, a user often refers to a video or illustration on the Internet using a terminal such as a smartphone or a tablet.
As a conventional technique, there is a technique for acquiring information selected by a user's operation of a terminal, and a technique for acquiring information when a user performs an explicit gesture such as raising or shaking a hand (for example, Non-Patent Document 1).

Takuya Sugimoto et al., Cooking Support Using Speech Recognition and Gesture Functions, IPSJ 76th Annual Convention, p. 4-879, (2014)

  However, it is difficult for the user to operate the terminal in order to select a theme or a part to be made in parallel with the user carrying the makeup with the makeup tool.

  In addition, as described above, it is difficult for the user to operate the terminal and perform the gesture as described above in order to acquire information in parallel with the user holding the makeup tool and performing makeup.

  Therefore, in the related art, it is not possible for the user to provide appropriate technical support when performing the work.

  The present invention has been made in view of the above circumstances, and an object of the present invention is to provide a technical support device, a method, a program, and a system capable of performing appropriate technical support when a user performs work. It is in.

In order to achieve the above object, a technical support apparatus according to a first aspect of the present invention includes an image acquisition unit that acquires a target image that is an image relating to a processing target including a user, and feature point information from the acquired target image. a feature point detection means for detecting a feature point information storage means for storing the detected feature point information, and the difference extracting means for extracting the difference information between frames of said stored the feature point information, the relative said processed Theme selection means for selecting theme information indicating the theme of the work of the user, presentation information storage means for storing presentation information for the user, and before the work by the user based on the difference information and the selected theme information. of the situation according to each of a plurality of steps up to and including post-work from the situation recognition means for recognizing a current situation according to the working, the difference information And based on the selected theme information, and recognizing the target region recognizing means an object region which is a part for the user to work on the processing target, based on the difference information, motion recognition recognizes an operation of the user Means, when the difference information is extracted by the difference extracting means, a result of recognition by the situation recognizing means, information of the target area recognized by the target area recognizing means, and a result of recognition by the action recognizing means. A presentation information acquisition unit that acquires presentation information to the user from presentation information stored in the presentation information storage unit, and a presentation that generates information obtained by performing predetermined processing on the acquired presentation information based on Information generation means, presentation timing storage means for storing information on the timing of presenting the presentation information for each presentation content, and the presentation information Based on the information generated by the generating means, obtains information on the timing of presenting the presentation information from the presentation timing storage means, and based on this information, presents the presentation information subjected to predetermined processing to the output device. Output timing control means.

  Further, in the technical support device according to a second aspect of the present invention, in the first aspect, the feature point detection unit extracts feature point information of the target image and determines a work situation referred to by the user. The information processing apparatus further includes a reference state storage unit that stores the indicated situation information, and the situation recognition unit includes a situation determination unit that determines a work situation of the user based on the feature point information and the situation information.

Further, in the technical support device according to a third aspect of the present invention, in the first aspect, the processing target includes a tool used by the user for work, and the feature point detecting unit includes a feature point of the target image. Information, and the difference extraction unit extracts the difference information based on the feature point information, and indicates a relationship between a tool used by the user for work and a target area that is a target of work using the tool. Tool information storing means for storing tool information, wherein the target area recognizing means obtains a target area that is a target of work using the tool based on the feature point information; based on the difference information and the tool information based on the point information, and tools recognition means for obtaining information of the target area and the tool is a work object with the tool, the target region acquisition hand And a target area determining unit that determines information of the target area and the tool obtained by the target area and the tool recognizing unit obtained by the above, and outputs a determination result as a result of recognition by the target area recognizing unit. .

  Further, in a technical support apparatus according to a fourth aspect of the present invention, in the second aspect, the reference state storage means stores the status information for each type of the theme, and the status determination means includes Based on the theme information selected by the selection means, the difference information based on the feature point information, and the situation information, a status of the user's work related to the theme information selected by the theme selection means is determined.

  Further, in a technical support apparatus according to a fifth aspect of the present invention, in the third aspect, the tool information storage unit stores the tool information for each type of the theme, and the target area obtaining unit includes The target area is obtained based on the theme information selected by the theme selecting means and the difference information based on the feature point information, and the tool recognizing means is based on the theme information and the feature point information selected by the theme selecting means. The information of the target area and the tool is acquired based on the difference information.

  That is, according to the present invention, it is possible to provide appropriate technical support when a user performs an operation.

FIG. 1 is a block diagram illustrating an example of a technical support device according to an embodiment of the present invention. FIG. 6 is a diagram showing, in a table format, a configuration of feature point information extracted by the technical support device according to the embodiment of the present invention. FIG. 4 is a diagram showing, in a table form, a configuration of face feature point information extracted by the technical support device according to the embodiment of the present invention. FIG. 6 is a diagram showing, in a table form, an example of difference information obtained from face feature point information by the technical support device according to the embodiment of the present invention. The figure which shows an example of the difference information which the technical support apparatus in embodiment of this invention obtained from the characteristic point information in table format. FIG. 2 is a block diagram showing an example of a detailed configuration of a situation recognition unit of the technical support device according to the embodiment of the present invention. The figure which shows an example of the presentation information stored in the presentation information storage part of the technical support apparatus in embodiment of this invention in a table format. The figure which shows an example of the status information stored in the reference state storage part of the technical support apparatus in embodiment of this invention in a table format. FIG. 2 is a block diagram showing an example of a detailed configuration of a target area recognition unit of the technical support device according to the embodiment of the present invention. The figure which shows an example of the tool information stored in the tool information storage part of the technical support apparatus in embodiment of this invention in table format. The figure which shows an example of the presentation timing information stored in the presentation timing storage part of the technical support apparatus in embodiment of this invention in a table form. 9 is a flowchart illustrating an example of a procedure for presenting a greeting accompanying the person authentication by the technical support device according to the embodiment of the present invention. The figure which shows an example of the display screen in the presentation of the greeting accompanying the person authentication by the technical support apparatus in embodiment of this invention. 9 is a flowchart illustrating an example of a procedure for presenting makeup information by the technical support device according to the embodiment of the present invention. FIG. 6 is a diagram showing an example of a display screen in a region confirmation by the technical support device according to the embodiment of the present invention. FIG. 4 is a diagram illustrating an example of a display screen in presenting makeup information by the technical support device according to the embodiment of the present invention. 5 is a flowchart illustrating a first example of a procedure for presenting living information by the technical support device according to the embodiment of the present invention. 9 is a flowchart illustrating a second example of a procedure for presenting living information by the technical support device according to the embodiment of the present invention. The figure which shows an example of the display screen in presentation of the living information by the technical support apparatus in embodiment of this invention. 9 is a flowchart illustrating an example of a procedure for storing information and presenting previous information by the technical support device according to the embodiment of the present invention. FIG. 4 is a diagram showing an example of a display screen in information storage by the technical support device according to the embodiment of the present invention. The figure which shows an example of the display screen in presentation of previous information by the technical support apparatus in embodiment of this invention.

Hereinafter, a technical support device, a method, a program, and a system according to an embodiment of the present disclosure will be described in detail with reference to the drawings. In the following embodiments, the same operation is performed for the parts with the same numbers, and the overlapping description will be omitted.
The present invention recognizes a situation where a user is making makeup by using image information and presents information according to the situation of the user.

  ADVANTAGE OF THE INVENTION According to this invention, a user's situation can be recognized (or predicted) using image information from a user's natural makeup operation, without an explicit gesture by a user.

Further, the user can acquire information according to the above-mentioned situation without operating the terminal or performing a gesture.
In addition, the user can actively receive information necessary for a life other than makeup without operating the terminal or performing a gesture.

FIG. 1 is a block diagram illustrating an example of a technical support device according to an embodiment of the present invention.
As shown in FIG. 1, the technical support device 10 includes an image acquisition unit 11, a feature point detection unit 12, a difference extraction unit 13, a feature point information storage unit 14, a theme selection unit 15, a situation recognition unit 16, and a target area recognition unit. 17, a motion recognition unit 18, a presentation information acquisition unit 19, a presentation information storage unit 20, a presentation information generation unit 21, a presentation timing control unit 22, and a presentation timing storage unit 23.

  The image obtaining unit 11 obtains an image (also referred to as a target image) related to a processing target obtained by shooting with the camera 30. The camera 30 may exist outside the technical support device 10 as shown in FIG. 1, or may be built in the technical support device 10. The image acquisition unit 11 performs pre-processing such as a correction process on the acquired image. The image acquisition unit 11 outputs the image subjected to this processing to the feature point detection unit 12.

  Specifically, the correction process includes a process of adjusting the resolution of the image, a process of removing a shadow in the image, and a process of removing another object (such as a hand or makeup) from a target region (a user's part or part) in the image. (A tool, etc.), a process of removing an object when they overlap, a process of limiting an area to be detected in an image by background extraction, a process of inverting left and right, and the like. Here, it is assumed that the camera 30 continues to capture the progress of the work on the processing target, and that the image acquisition unit 11 acquires each frame of the image at a fixed interval, for example, an interval of about 30 to 60 ms.

  The feature point detection unit 12 receives a target image (or a preprocessed target image) from the image acquisition unit 11. The feature point detection unit 12 detects feature point information from each frame of the target image. The feature point information includes a target image, and information on a positional relationship and a color of the feature point in the target image.

  The feature point detection unit 12 outputs feature point information detected from the same target image to the difference extraction unit 13.

Next, details of the feature point information will be described.
In the case of makeup, the feature point information may be general feature point information or face feature point information. In this case, general feature point information is distinguished from face feature point information, and does not overlap with face feature point information. The face feature point information includes information on a positional relationship between a target image and a processing region in the target image, and color. The processing area is a part where the user works on the processing target, and is also called a part or a part. The parts of the face include parts that impress the face such as left eyebrow, right eyebrow, left eye, right eye, nose, mouth, left cheek, and right cheek.
FIG. 2 is a diagram illustrating, in a table form, a configuration of feature point information extracted by the technical support device according to the embodiment of the present invention.
The feature point detection unit 12 derives feature point information from the acquired image using a feature point detection method such as the Harrisa method or the KLT method. As shown in FIG. 2, in the feature point information, feature point coordinates p_a are associated with colors (actually measured values) c_a. In addition, the feature point coordinates p_a are assumed to be not only two-dimensional but also one-dimensional coordinates to three-dimensional coordinates, but higher-dimensional coordinates can be used similarly. The color (actually measured value) c_a may be an RGB value, an RGBA value, an HSV value for each pixel, or an average value of the RGB values, the RGBA values, and the HSV values of all the pixels in the area. Any information may be used as long as it is a value. The same applies to other color information such as a color (actually measured value) c_b to be described later.

FEATURES point detecting unit 12 uses the face detection method of a method such as a Viola-Jones, acquires the face feature point information in the image. FIG. 3 is a diagram showing, in a table form, a configuration of face feature point information extracted by the technical support device according to the embodiment of the present invention. As shown in FIG. 3, the face feature point information is associated with a part ID, feature point coordinates p_b, and a color (actually measured value) c_b. The part ID is, for example, an identifier for uniquely identifying a part of the user's face.

  The feature point detection unit 12 may perform a correction process in feature point detection. More specifically, processing to bring the face to the front, processing to remove the object when an object such as a hand or shadow overlaps the target area in the image, processing to erase moles and scratches, processing other than the face There is a trimming process.

The difference extraction unit 13 acquires feature point information from the feature point detection unit 12. Upon extracting the difference, the difference extracting unit 13 outputs a work start signal to the presentation information acquisition unit 19 and the presentation information storage unit 20. The difference extraction unit 13 stores the acquired feature point information in the feature point information storage unit 14.
The difference extracting unit 13 acquires difference information from the feature point information of each frame adjacent in time series stored in the feature point information storage unit 14. The difference information includes a difference value and information indicating the position. The feature point information storage unit 14 stores, for example, about 10 consecutive frames in time series, and overwrites the oldest frames sequentially.

  The difference extraction unit 13 outputs the acquired difference information to the target area recognition unit 17. Further, the difference extracting unit 13 outputs the acquired difference information to the situation recognizing unit 16 and the target area recognizing unit 17.

  Next, details of the difference information will be described.

In the case of makeup, the difference information includes difference information extracted based on general feature point information and face difference information extracted based on face feature point information. FIG. 4 is a diagram illustrating, in a table format, an example of face difference information obtained from the face feature point information by the technical support device according to the embodiment of the present invention. As shown in FIG. 4, the face difference information obtained from the face feature point information by the difference extraction unit 13 is associated with a part ID, feature point coordinates (actual measurement value) p_b, color (actual measurement value) c_b, and difference value d_b. .
The difference extracting unit 13 compares the color information on the latest frame with the color information on a frame earlier than the latest frame, and uses the binary difference, the least squares method, or the like to calculate the above-mentioned face difference information. Calculate the difference value d_b. The difference value d_b may be an average value of a region which is also a unit of one pixel. Further, the face difference information and the difference information are not limited to the color difference value between one frame, and may be, for example, a color difference value between a plurality of frames so as to be able to cope with a change in an image state (temporary reflection of light, etc.). Or the average value of

  FIG. 5 is a diagram illustrating, in a table form, an example of difference information obtained from the feature point information by the technical support device according to the embodiment of the present invention. As shown in FIG. 5, the difference information obtained by the difference extraction unit 13 from the feature point information does not include the part ID of the face difference information obtained from the face feature point shown in FIG. p_a, color (actually measured value) c_a, and difference value d_a are associated with each other.

  The feature point information storage unit 14 is a so-called buffer, and can be realized by a storage device such as a nonvolatile memory, and can store face feature point information and feature point information for each frame.

  The external content DB 50 can be realized by, for example, a storage device such as a non-volatile memory, and stores contents related to makeup (a website related to cosmetics and beauty, a database of a cosmetics company or a company related to beauty, and an application that stores a past history of makeup by a user). The user selects in advance the external content 70 such as content that is not related to makeup (train operation status, weather forecast application, etc.) and which information of the external content 70 is to be used. It is stored. For example, when the external content 70 is a cosmetics web site, when the information that a new product has been released is updated, information that presents detailed information of the product or a video on how to use the product as makeup advice information is stored in the external content DB 50. It is thought that it is done. In the case where the external content 70 is a weather forecast application, information indicating that the weather forecast information is presented together with makeup advice when the weather forecast information is updated may be stored. A URL for accessing the external content 70 may be registered in the external content DB 50.

  The theme selection unit 15 acquires theme information specified by the user via the operation terminal 40 such as a tablet. The operation terminal 40 may be integrated with the technical support device 10. The theme information indicates a category of work, and in the case of makeup, the theme information indicates a category of makeup that is referred to by the user, such as a cute system or a mature system. The category of the makeup theme may be changed according to the set value, and an identifier for uniquely identifying the theme is set as the theme ID. The theme information may be set based on information obtained from the operation terminal 40 or the external content DB 50. For example, a photo managed by an application for storing media such as a photo incorporated in the operation terminal 40, a trend page on a cosmetics company's web page or SNS of the external content 70 registered in the external content DB 50, a user's favorite Acquire theme information from makeup etc.

FIG. 6 is a block diagram illustrating an example of a detailed configuration of the situation recognition unit of the technical support device according to the embodiment of the present invention.
As shown in FIG. 6, the situation recognition section 16 includes a situation determination section 16a and a reference state storage section 16b.

  The theme selection unit 15 outputs the theme information to the situation determination unit 16a in the situation recognition unit 16, the tool recognition unit 17b in the target area recognition unit 17, and the target area acquisition unit 17a in the target area recognition unit 17. The target area acquisition unit 17a and the tool recognition unit 17b will be described later.

  The situation determination unit 16a of the situation recognition unit 16 acquires difference information from the difference extraction unit 13 and theme information from the theme selection unit 15, respectively. The situation determining unit 16a determines the situation of the user's work based on the difference information and the theme information. The situation determination unit 16a outputs the situation determination result to the presentation information acquisition unit 19.

  The target area recognition unit 17 obtains difference information from the difference extraction unit 13. The target area recognition unit 17 recognizes a target area (a part or part of the user) in the image and a tool to be used based on the acquired difference information. The target area recognition unit 17 outputs information indicating the recognized target area or tool to the presentation information acquisition unit 19. Details of the configuration and operation of the target area recognition unit 17 will be described later.

  The motion recognition unit 18 obtains difference information from the difference extraction unit 13 and recognizes a broadly-defined motion such as a smile or gesture of the user in the image based on the difference information, and determines a result of the recognition. Output to the presentation information acquisition unit 19.

  The presentation information storage unit 20 has a storage device such as a nonvolatile memory, for example, and stores the presentation information described below in this storage device. The presentation information acquisition unit 19 indicates a situation determination result from the situation recognition unit 16, information indicating a target area or a tool from the target area recognition unit 17, a work start signal from the difference extraction unit 13, and an operation from the action recognition unit 18. Get each information.

  The presentation information acquisition unit 19 obtains the situation determination result, the target area, the tool, and the information indicating the operation based on the acquired situation determination result, the information indicating the target area and the tool, and the work start signal and the information indicating the action. The presentation information stored in the presentation information storage unit 20 is compared, and these situation determination results are associated with a situation ID, a theme ID, a part ID, a tool ID, and a motion ID corresponding to information indicating a target area, a tool, and a motion. Obtained presentation information. As shown in FIG. 7, the presentation information includes information on the content to be presented to the user and the presentation format. The presentation information acquisition unit 19 outputs the acquired presentation information to the presentation information generation unit 21.

FIG. 7 is a diagram illustrating an example of the presentation information stored in the presentation information storage unit of the technical support device according to the embodiment of the present invention in a table format.
As illustrated in FIG. 7, the presentation information stored in the presentation information storage unit 20 is associated with a situation ID, a theme ID, a site ID, a tool ID, an operation ID, a presentation content, a content detail, and a presentation format.

  The situation ID is a state of work performed by the user such as “application start = 0 ′”, “before work = 0”, “working = 1”, “after work = 2”, “application running = 3”, and the like ( This is an identifier for uniquely identifying the situation. The tool ID is, for example, an identifier for uniquely identifying a tool used by the user for makeup. Here, the identifier is made up of the maker, color, color, and product number of the makeup tool. The action ID is an identifier for uniquely identifying an action such as “smile of user = 0”, “raise hand = 1”. "-" Of various IDs in the presentation information indicates that the type of the ID is irrelevant.

The presentation content includes, in addition to greetings and makeup advice to the user, information other than makeup and beauty, for example, information necessary for daily life, chat, and the like.
The presentation format includes (1) outputting a sentence, illustration, moving image, or audio, (2) generating the content to be presented as an image, and superimposing the image on the target image, and (3) registering the content in the external content DB 50. Displaying in the form of a chat in cooperation with the external content 70, presenting a link to the user, and the like are exemplified. Further, a plurality of these presentation formats may be combined for the same presentation content.

  When the work start signal is acquired from the difference extracting unit 13, the presentation information storage unit 20 stores information (eg, information such as train delay) in an application (eg, train operation status) registered in the external content DB 50. It is confirmed whether or not the information has been updated. When the information has been updated, the updated information is acquired from the external content DB 50 in order to present the information to the user, and overwritten and stored. The presentation information storage unit 20 acquires the updated information from the external content DB 50 in order to present the updated information to the user when the information is updated in the application on the external content DB 50 during the activation of the application. And overwrite it. Then, the presentation information acquisition unit 19 acquires the presentation information corresponding to “Situation ID = 3” and outputs the presentation information to the presentation information generation unit 21.

  The presentation information generation unit 21 receives the presentation information from the presentation information acquisition unit 19. The presentation information generating unit 21 performs a process of correcting the received presentation information and a process of outputting the presentation information (for example, a drawing process when the output format is an image), and displays the presentation information that has been subjected to this process. Output to the presentation timing control unit 22.

  When the presentation information is an image, the correction process of the presentation information may be a process of adjusting the balance between the color of the target image and the presentation position.

Next, details of the situation recognition unit 16 will be described.
The situation determination unit 16a of the situation recognition unit 16 acquires difference information from the difference extraction unit 13 and theme information from the theme selection unit 15, respectively.

The reference state storage unit 16b is a storage device such as a non-volatile memory, for example, and stores in advance status information indicating the status of work to which the user refers for each theme.
FIG. 8 is a diagram illustrating an example of the status information stored in the reference status storage unit of the technical support device according to the embodiment of the present invention in a table format. As shown in FIG. 8, the situation information stored in the reference state storage unit 16b includes a theme ID, a part ID, feature point coordinates (theoretical value) p'_b, a color (theoretical value) c'_b, and a difference value d '. _b and a situation ID are associated with each other.

The feature point coordinates (theoretical value) p′_b, the color (theoretical value) c′_b, the difference value d′ _b, and the situation ID are set values, and can be changed by, for example, setting by operating the operation terminal 40. Further, the color (theoretical value) c′_b can be changed according to the theme information acquired from the external content DB 50 or the operation terminal 40.
The situation determination unit 16a acquires situation information from the reference state storage unit 16b using the acquired difference information and theme information.
The situation determination unit 16a outputs the acquired situation information to the presentation information acquisition unit 19.

Next, a specific example of acquiring situation information in the case of makeup will be described. The situation determination unit 16a determines a difference value d_b defined by one or more lines on the face difference information, which satisfies a predetermined condition (can be changed by setting), a site ID associated with the difference value d_b, and a feature point coordinate ( The specific face difference information associated with the (measured value) p_b and the color (measured value) c_b is extracted from the face difference information acquired as described above.
The situation determination unit 16a includes (1) a combination of the theme ID, which is the theme information acquired from the theme selection unit 15, and the part ID in the specific face difference information extracted from the face difference information, and (2) a combination of the reference state storage unit 16b. The obtained situation information is compared with the combination of the theme ID and the part ID.

  When these compared various IDs have the same value among the same type of IDs, the situation determination unit 16a determines (1) one element of the specific face difference information, and the color (actual measurement) corresponding to each of the compared IDs. The difference value between the (value) c_b and the color (theoretical value) c′_b corresponding to each of the compared IDs, which is one element of the situation information stored in the (2) reference state storage unit 16b, is calculated.

  The situation determination unit 16a compares the calculated difference value with the range of the difference value d'_b of each row of the situation information stored in the reference state storage unit 16b, and the calculated difference value is included in this range. If it is included, the situation ID (0, 1, or 2), theme ID, and part ID corresponding to the difference value d'_b in this range are obtained from the situation information.

Next, details of the target area recognition unit 17 will be described. FIG. 9 is a block diagram illustrating an example of a detailed configuration of the target area recognition unit (of the technical support apparatus according to the embodiment of the present invention).
As shown in FIG. 9, the target area recognition unit 17 includes a target area acquisition unit 17a, a tool recognition unit 17b, a target area determination unit 17c, and a tool information storage unit 17d. The target area recognition unit 17 may include only the target area acquisition unit 17a and the target area determination unit 17c, or may include only the tool recognition unit 17b, the target area determination unit 17c, and the tool information storage unit 17d. Alternatively, the configuration may include only the target area acquisition unit 17a, or may include only the tool recognition unit 17b and the tool information storage unit 17d.

  The target area acquisition unit 17a acquires face difference information from the difference extraction unit 13 and theme information from the theme selection unit 15, respectively. The target area obtaining unit 17a obtains information on the target area from the obtained difference information and theme information. The target area acquisition unit 17a outputs information on the determined target area to the target area determination unit 17c. Specifically, the target area acquisition unit 17a color-codes the acquired face difference information for each part at coordinate points that are equal to or greater than a set value (can be changed by setting) with a difference value in each part. For example, a coordinate point having a difference value equal to or more than a set value in the eyebrow area is red, a coordinate point having a difference value equal to or more than a set value in the eye area is blue, and a coordinate value having a difference value in the cheek area is equal to or more than a set value. Is a green coordinate point, and a coordinate point having a difference value equal to or more than a set value in the lip region is yellow. When there are a plurality of adjacent coordinate points among the colored coordinate points in each part, the target area obtaining unit 17a extracts the contour of the area formed by the adjacent coordinate points. At this time, the target area acquiring unit 17a changes the condition of the area generated from the coordinate points by setting, for example, when there are a plurality of adjacent coordinate points, when the area of the area formed by the adjacent coordinate points is X or more. I can do it. The target area acquisition unit 17a determines a part having the smallest y coordinate (upper part of the screen) among the extracted contours as the target area.

  The tool recognition unit 17b acquires difference information from the difference extraction unit 13 and theme information from the theme selection unit. The tool recognizing unit 17b uses the obtained difference information and theme information to obtain information on the target area and the tool to be used from the tool information storage unit 17d. The tool recognition unit 17b outputs the acquired information on the target area to the target area determination unit 17c.

  The acquisition of the information on the target area and the tool by the tool recognizing unit 17b is a pattern matching method by statistical machine learning or general object recognition, and the method is not limited.

In the tool information storage unit 17d, tool information indicating the relationship between the tool used by the user for work and the processing area is stored in advance.
FIG. 10 is a table showing an example of tool information stored in the tool information storage unit of the technical support device according to the embodiment of the present invention. As shown in FIG. 10, the tool information is associated with a theme ID, a part ID, a tool ID, feature point coordinates (theoretical value) p'_a, and a color (theoretical value) c'_a.

The tool recognition unit 17b compares the theme ID acquired from the theme selection unit 15 with the theme ID in the tool information stored in the tool information storage unit 17d. The tool recognizing unit 17b sets the feature point coordinates (theoretical value) p′_a and color corresponding to the corresponding theme ID in the tool information stored in the tool information storage unit 17d when the compared theme IDs have the same value. The difference (or similarity) between the (theoretical value) c′_a and the feature point coordinates p_a and the color (actually measured value) c_a in each row of the difference information acquired from the difference extracting unit 13 is calculated.
When the value of the calculated difference is within a certain set value (can be changed by setting) (in the case of similarity, the value is equal to or more than a certain set value), the tool recognizing unit 17b determines the value of the difference. The part ID, theme ID, and tool ID corresponding to the feature point coordinates (theoretical value) p′_a and the color (theoretical value) c′_a used for the calculation are acquired from the tool information.

  For example, the tool recognition unit 17b recognizes the makeup tool from the difference information using object recognition or the like. The tool recognizing unit 17b compares the recognized combination of the feature point coordinates p_a and the color (actually measured value) c_a of the makeup tool with the feature point coordinates (theoretical value) p′_a in the tool information stored in the tool information storage unit 17d. The combination of the color (theoretical value) c'_a is compared, and when the compared values are the same or close to satisfying the predetermined condition, the compared feature point coordinates (theoretical value) p'_a and the color (theoretical value) The part ID, theme ID, and tool ID associated with the combination of c′_a are acquired from the tool information as information of the target area.

  When the user sets information on a product (for example, a makeup tool) purchased at the time of purchasing the product by operating the operation terminal 40, the external content DB 50 stores information on the product on the external content 70 such as the web content of a cosmetic maker. It is also conceivable that the feature point coordinates and color information are obtained and stored as tool information in the tool information storage unit 17d. The feature point coordinates (theoretical value) p'_a and the color (theoretical value) c'_a in the tool information may be values obtained from machine learning.

  The target area determination unit 17c receives information on the target area and the tool from the target area acquisition unit 17a and the tool recognition unit 17b. The target area determination unit 17c determines the received target area or tool, and outputs the determination result to the presentation information acquisition unit 19 as a recognition result by the target area recognition unit 17.

  The determination method can be changed by setting by operating the operation terminal 40. For example, if the information of the target area acquired from both the target area acquiring unit 17a and the tool recognizing unit 17b is the same, the target area determining unit 17c sends the information of the target area to the presentation information acquiring unit 19 as the information of the target area. May be output. Alternatively, when the target area determining unit 17c receives the information of the target area or the tool from only one of the target area obtaining unit 17a and the tool recognizing unit 17b, the information of the target area or the tool is transmitted to the presentation information obtaining unit 19 as it is. It is possible to output.

  The presentation timing control unit 22 receives the processed presentation information from the presentation information generation unit 21. The presentation timing control unit 22 acquires the presentation timing information from the presentation timing storage unit 23 based on the received presentation information, and outputs the processed presentation information to the output device 60 based on the presentation timing information. . The output device 60 may be integrated with the technical support device 10. The presentation timing storage unit 23 is realized by, for example, a storage device such as a non-volatile memory, and stores information related to the timing of presenting presentation information in advance for each presentation content. The presentation timing information is a condition when the presentation information is output to the output device 60, and enables the presentation timing control unit 22 to refer to the information.

Here, a specific example of control of the presentation timing of the presentation information will be described. FIG. 11 is a table showing an example of presentation timing information stored in the presentation timing storage unit 23 of the technical support device according to the embodiment of the present invention.
As shown in FIG. 11, the presentation timing information stored in the presentation timing storage unit 23 includes a presentation number (equivalent to “No.” of the presentation information shown in FIG. 7), a presentation content (the presentation information shown in FIG. 7). And "presentation contents"), and presentation conditions are associated with each other.

  The presentation timing control unit 22 receives the presentation information generated by the presentation information generation unit 21 and the presentation number (“No.” of the presentation information shown in FIG. 7) associated with the presentation information. The presentation timing control unit 22 compares the received presentation number with the presentation number in the presentation timing information stored in the presentation timing storage unit 23, and when the compared values are the same, the corresponding presentation timing information (presentation Condition) and outputs the obtained information to the output device 60.

Next, an example of an operation and a display screen for technical support by the technical support device 10 will be described.
(Example 1)
First, the presentation of the greeting accompanying the person authentication will be described.
FIG. 12 is a flowchart illustrating an example of a procedure for presenting a greeting accompanying personal authentication by the technical support device according to the embodiment of the present invention.
FIG. 13 is a diagram illustrating an example of a display screen in presenting a greeting accompanying the person authentication by the technical support device according to the embodiment of the present invention.

  For personal authentication, the difference extracting unit 13 extracts the face feature point information of the user from the image captured by the camera 30 and received from the feature point detection unit 12, and extracts the face feature point information from the user. It can be stored in the feature point information storage unit 14 as reference feature point information in association with a identifiable name and profile (eg, age, place of birth, magazine that you read everyday, hobbies, etc.). The name and profile information that can specify the individual user may be stored in the external content 70 via the presentation information storage unit 20, the operation terminal 40, and the external content DB 50. The reference feature point information is not deleted unless the user performs a predetermined operation for deletion on the operation terminal 40.

  In the case where the presentation information has not yet been displayed on the monitor screen of the output device 60 since the operation of the technical support device 10, the difference extraction unit 13 outputs the user's face from the image captured by the camera 30 from the feature point detection unit 12. The feature point information is received, that is, the user's face is detected (S11). Then, the difference extracting unit 13 acquires the face feature point information and the above-mentioned reference feature point information, compares them, and matches or is similar to the detected face feature point information on the reference feature point information. The person is recognized by acquiring the name associated with the information (S12), and the profile associated with this name is read (S13).

  In this example, the difference extraction unit 13 outputs the name and the profile to the presentation information storage unit 20. Further, the presentation information storage unit 20 stores the acquired name and profile in the corresponding presentation content.

  In the case where the presentation information has not been displayed as described above, when the presentation information acquisition unit 19 receives the work start signal from the difference extraction unit 13, the presentation information acquisition unit 19 converts it into “Situation ID = 0 ′”, and converts the “Situation ID = 0”. When “0 ′” is acquired, the captured image of the user is displayed on the monitor screen of the output device 60 and the content of the row “NO. 1” of the presentation information shown in FIG. is defined in detail "○○ (name of user) 's, Hello!" greetings and, to be output from the speaker (not shown) of the output device 60 by voice. Alternatively, the information may be displayed on a monitor screen of the output device 60 as shown in FIG.

Next, a description will be given of the case where the user presents makeup information as advice according to the part on which makeup is applied.
FIG. 14 is a flowchart illustrating an example of a procedure for presenting makeup information by the technical support device according to the embodiment of the present invention.
FIG. 15 is a diagram illustrating an example of a display screen in the part confirmation by the technical support device according to the embodiment of the present invention.
FIG. 16 is a diagram illustrating an example of a display screen in presenting makeup information by the technical support device according to the embodiment of the present invention.

  After the above greeting is presented, the presentation information generating unit 21 notifies the theme selecting unit 15 that the greeting has been presented. Then, the theme selecting unit 15 determines the sleep time, the schedule, the amount of exercise (for example, the number of steps), etc., of the user, which are managed by the operation terminal 40, the external content DB 50, and the like, in the presentation of the above greeting. The derived theme suitable for the user, here, the makeup theme is acquired. When the user selects a makeup theme from the makeup themes using the operation terminal 40, theme information is output to the situation recognition unit 16, the target area recognition unit 17, and the operation recognition unit 18. Then, in a state where the captured image in which the user is captured is displayed, the situation recognition unit 16 outputs the situation information including the situation ID “0” to the presentation information acquisition unit 19, and the target area determination unit 17c of the target area recognition unit 17 However, it is assumed that the part ID corresponding to the part (the user's eyebrows) at which the user starts to make is output to the presentation information acquisition unit 19, that is, the part at which the user starts making is detected (S21).

  Then, the presentation information acquisition unit 19 sends a confirmation message of “make eyebrows!” Defined in the row of “NO. 11” in the presentation information shown in FIG. 15 is displayed on the screen of the output device 60 as shown in FIG.

  Then, after the display of the above confirmation message, the presentation information acquisition unit 19 replaces this confirmation message with the presentation information shown in FIG. As advice information on makeup points and methods determined by the line, a message on the points of the makeup method associated with the detected makeup theme and a moving image of the makeup method are displayed on the screen of the output device 60 as shown in FIG. It is displayed (S22).

  With the advice information on the point and method of the makeup displayed, the presentation information acquisition unit 19 places this information in the area corresponding to the part of the user's captured image on the screen that is detected in S21 and that corresponds to the part where the user starts to make. The image of the area where makeup is applied to the part is superimposed and displayed (S23).

  Thereby, according to the part on which the user applies makeup, information on makeup related to the part can be presented to the user. If the advice information displayed in S24 based on the makeup theme does not conform to the user's preference, the theme selection unit 15 is caused to again detect a theme suitable for the user by performing a predetermined operation on the operation terminal by the user. Alternatively, regardless of this detection, an arbitrary make-up theme is set by a predetermined operation of the operation terminal by the user, and a message of a make-up point associated with this make-up theme and a video of the make-up method are displayed. The information may be displayed on the screen of the output device 60 as described above. Similarly, information on the makeup tool used can be presented.

  Next, presentation of life information will be described. FIG. 17 is a flowchart illustrating a first example of a procedure for presenting life information by the technical support device according to the embodiment of the present invention. FIG. 18 is a flowchart illustrating a second example of a procedure for presenting life information by the technical support device according to the embodiment of the present invention. FIG. 19 is a diagram illustrating an example of a display screen in presenting life information by the technical support device according to the embodiment of the present invention.

The weather forecast information of the weather forecast application is updated before the start of the makeup by the user and in a state in which the photographed image of the user is displayed, and the latest weather forecast information is stored in the presentation information storage unit 20 (S31). . When the situation recognition unit 16 outputs situation information including the situation ID “0” to the presentation information acquisition unit 19, that is, when it is detected that the user is about to make (S32), the presentation information acquisition is performed. The unit 19 stores, in the presentation information storage unit 20, the presentation information illustrated in FIG. 7, the weather forecast information defined by the row “NO. 6”, and a message of makeup advice based on this weather, for example, A message "Today's weather is rainy. Base makeup should be thin because of high humidity!" Is displayed on the screen of the output device 60 as shown in FIG. 19 (S33). The order of the update of S31 and the detection of S32 may be reversed.
When the user activates this application and the train operation information of the external content DB 50 is updated, if the latest train operation information is stored in the presentation information storage unit 20 (S31A), the presentation information acquisition is performed. The unit 19 stores a message indicating the operation status of the train determined by the row “NO. 5” in the presentation information shown in FIG. Is displayed on the screen of the output device 60 as shown in FIG. 19 (S32A). Since the message of the operation status of the train determined in the row of “NO. 5” is “Status ID = 3”, it is not dependent on the work status of the makeup, and is displayed at any time while the application is running. Good.

  Thereby, the living information can be presented to this user at an appropriate time according to the content to be presented.

Next, the saving of the makeup image after the completion of makeup by the user and the selection of a theme at the start of the next makeup will be described.
FIG. 20 is a flowchart illustrating an example of a procedure of storing information and presenting previous information by the technical support device according to the embodiment of the present invention.
FIG. 21 is a diagram illustrating an example of a display screen in information storage by the technical support device according to the embodiment of the present invention.
FIG. 22 is a diagram illustrating an example of a display screen in presenting previous information by the technical support device according to the embodiment of the present invention.

  Some applications registered in the external content DB 50 store, for example, a photographed image of the user when the user recognizes a facial expression or action such as a smile or a piece of the user, and manages the image. Conceivable. From this, it is conceivable that the external content DB 50 stores difference information based on feature point information of an image indicating a facial expression or an action such as making a smile or making a piece after makeup. In this case, the difference information Is acquired in advance by the motion recognition unit 18. After the makeup by the user is completed, and in a state in which an image (for example, a smile or a piece) photographed after the completion of the makeup is displayed, the action recognition unit 18 sends the difference information (the feature point information shown in FIG. ) And face difference information (difference information obtained from the face feature point information shown in FIG. 3), and performs the difference information and the face difference information together with the smile or piece obtained from the external content DB 50. The face difference information (the face of the user) and the difference information (the hand other than the user's face, etc.) of the image indicating the facial expression and motion are compared, and the difference information and the face difference information are compared with the face difference information or the difference information. When it is detected that the user has completed the makeup (S41), the user can determine the captured image, date, and makeup theme (for example, “for going out”). , URL sources of makeup items, URL sources makeup method, if wishing to save makeup preference degrees) by the user, performs a predetermined operation or the like on the operation terminal 40.

Then, the theme selecting unit 15 adds the theme information, and links the captured image and the makeup information such as the date, the makeup method, and the points with the theme information to the external content via the external content DB 50, for example. 70 and the operation terminal 40 (S42). Further, in the presentation information storage unit 20, for example, “No. = 3, situation ID = 0, theme ID = 2, site ID = −, tool ID = −, presentation content = make-theme (past favorite make-up) details , Content details = there are the following two! (Past makeup contents) ", and the presentation information storage unit 20 stores the makeup information corresponding to" content details "from the external content DB 50 after completion of the makeup information. Get and save the shooting image, date, makeup theme, makeup information. The above-mentioned make-up favorite degree can be set by the user operating the operation terminal 40 or the like after the make-up is completed.
If the user does not want to save the captured image, date, and makeup theme after makeup is completed, the user can perform a predetermined operation on the operation terminal 40 or the like so that the saving is not performed.

  At the same time as this storage, the presentation information acquisition unit 19 sends a message indicating that the date, makeup theme, and the like have been saved (for example, the job is well done! Save it as shown in FIG. 21). Is displayed on the screen of the output device 60.

  Then, the use of the technical support device 10 is terminated, and when the user performs the next make-up, the technical support device 10 is operated again, and after the above-described person authentication is performed again, a captured image in which the user is displayed is displayed. In this state, the user instructs the operation terminal 40 to present the last time a make-up method according to the same make-up theme (for example, “Tell me the make-up method for going out before!” Character input or voice input).

  Then, the presentation information acquisition unit 19 saves the presentation information corresponding to the above “No. 3” in the presentation information storage unit 20, that is, as described above, at the time of completion of the previous and previous make-up in the past two times, The photographed image after makeup is completed, the date, and the makeup theme (for example, “going out”, the URL of the information source of the makeup item, the URL of the information source of the makeup method, and the degree of favorite of the makeup by the user) are acquired. The presentation information acquisition unit 19 acquires the various types of information acquired in the presentation information storage unit 20 shown in FIG. .3 "is obtained and displayed on the screen of the output device 60 as shown in FIG. 22 (S43).

  As described above, in the present embodiment, the technical support device 10 can provide appropriate technical support when a user performs a makeup operation.

(About processing division and arrangement)
Note that, in the above-described embodiment, an example in which the processing is executed as one technical support apparatus 10 is described. However, this processing may be executed separately by the server and the client (user terminal).

  For example, processing with a high processing cost (for example, feature point detection unit 12, difference extraction unit 13, situation recognition unit 16, target region recognition unit 17, motion recognition unit 18, presentation information acquisition unit 19, presentation information generation unit 21, presentation The processing performed by the timing control unit 22) is executed by the server, and a database (for example, the feature point information storage unit 14, the presentation information storage unit 20, the presentation timing storage unit 23, and the external content DB 50) that requires a large amount of data is stored in the server. The processing may be implemented in a storage device and execute other functions (for example, the image acquisition unit 11 and the theme selection unit 15) on the client.

  Alternatively, processing until the presentation information is generated (the image acquisition unit 11, the feature point detection unit 12, the difference extraction unit 13, the feature point information storage unit 14, the theme selection unit 15, the situation recognition unit 16, the target region recognition unit 17, The client executes the action recognition unit 18, the presentation information acquisition unit 19, the presentation information generation unit 21, and the external content DB 50), and controls the presentation timing of the presentation information (the presentation timing control unit 22, the presentation timing storage unit 23). The processing may be executed by a server.

  When processing is executed by distributing processing between the server and the client as described above, it is necessary to exchange images and coordinate value information between the server and the client. Instead of exchanging, it is desirable to perform a thinning process of the exchanged image, a process of changing the image format, and the like. The thinning process may be performed, for example, for every several frames of the image, or only for the difference between the previous frame of the image and the current frame. In the change processing, for example, only the coordinate information of the feature point specified by the feature point detection unit 12 may be extracted instead of the image data of one frame. In the change process, the image acquisition unit 11 may extract image data that has been subjected to a compression process or the like. Furthermore, it is conceivable that coordinate information of the difference information extracted by the difference extracting unit 13 is extracted.

(Application examples other than makeup)
Although the above description has been made on the assumption that the user performs makeup as an application example of the technical support apparatus 10, the present invention is not limited to this, and is applicable to, for example, coaching of surgery for doctors and medical college students.

Specifically, the image acquisition unit 11 may be configured to acquire an image from an in-camera of a terminal such as a tablet or a camera attached to the periphery. As the output device 60, a tablet, a monitor, an HMD (Head Mount Display) (registered trademark), or the like may be used.
The feature point detection unit 12 may extract a body feature point, such as a body part or a bone muscle, in an image obtained by photographing a region around a surgical operation on a human body (a human body model).

  The difference extraction unit 13 stores feature point information on the operation state, such as body feature points such as body parts and bone muscles, and feature point information on the entire image in the feature point information storage unit 14 and stores the feature point information. The difference information may be obtained by comparing the feature point information with the feature point information of the latest frame.

The situation determination unit 16a of the situation recognition unit 16 may derive work conditions before, during, and after the operation from the reference state storage unit 16b using the difference information.
The reference state storage unit 16b stores the feature point coordinates of images, theoretical values of colors (human body model and human body), and situation information (situation ID) for difference values for each operation content and body part. No.

  As the target area recognizing unit 17, the target area obtaining unit 17a may obtain a target area based on difference information about the body. The target area is considered to be a body part. Further, the tool recognition unit 17b can acquire information indicating the target area (part ID) and the tool from the tool information stored in the tool information storage unit 17d using the feature point information.

The tool information storage unit 17d can store in advance characteristic point information and color information theoretical values regarding surgical tools such as a scalpel and tweezers.
The target area determination unit 17c can determine the target area or tool received from the target area acquisition unit 17a and the tool recognition unit 17b, and can output the determination result to the presentation information acquisition unit 19.

  The motion recognition unit 18 recognizes a motion such as raising a hand based on the difference information acquired from the difference extraction unit 13 and outputs information indicating the motion to the presentation information acquisition unit 19.

The presentation information acquisition unit 19 can acquire the presentation information from the presentation information storage unit 20 using the received situation information and the determination result of the target area tool, and can output the presentation information to the presentation information generation unit 21.
The presentation information storage unit 20 can store information other than the surgery, such as information on the surgery, such as the method and points of the surgery, correspondence to emergency surgery, and disaster information. The presentation information generation unit 21 receives the presentation information, corrects the presentation information, and outputs the corrected presentation information to the presentation timing control unit 22.

  The presentation timing control unit 22 acquires presentation timing information from the presentation timing storage unit 23 using the received presentation information subjected to the processing. The presentation timing storage unit 23 stores timing information to be presented in correspondence with the content to be presented, such as information about the operation such as the method of operation and points, information on non-operation such as correspondence to emergency operation and disaster information. . The presentation timing control unit 22 outputs presentation information processed in accordance with the acquired presentation timing to the output device 60.

(Other application examples)
In addition to the operation, the technical support apparatus 10 can be applied to, for example, coaching of how to write a character such as a calligraphy using a brush, work support in a factory, and the like. That is, the technical support device 10 according to the present embodiment can be used in any case as long as the information on the work of the user and the information on the tools used by the user can be obtained.

  Note that the present invention is not limited to the above-described embodiment as it is, and can be embodied by modifying constituent elements in an implementation stage without departing from the scope of the invention. In addition, various inventions can be formed by appropriately combining a plurality of components disclosed in the above embodiments. For example, some components may be deleted from all the components shown in the embodiment. Further, components of different embodiments may be appropriately combined.

  In addition, the method described in each embodiment can be implemented by a computer (computer) as a program (software means) such as a magnetic disk (floppy (registered trademark) disk, hard disk, or the like), an optical disk (CD-ROM, It can be stored in a recording medium such as a DVD, MO, or a semiconductor memory (ROM, RAM, flash memory, etc.), and can also be transmitted and distributed via a communication medium. The program stored on the medium side includes a setting program for causing the computer to execute software means (including not only the execution program but also tables and data structures) to be executed in the computer. A computer that realizes the present apparatus reads a program recorded on a recording medium, and in some cases, constructs software means by using a setting program, and executes the above-described processing by controlling the operation of the software means. The recording medium referred to in the present specification is not limited to a medium for distribution, but includes a storage medium such as a magnetic disk or a semiconductor memory provided in a computer or a device connected via a network.

  DESCRIPTION OF SYMBOLS 10 ... Technical support apparatus, 11 ... Image acquisition part, 12 ... Feature point detection part, 13 ... Difference extraction part, 14 ... Feature point information storage part, 15 ... Theme selection part, 16 ... Situation recognition part, 17 ... Target area recognition Unit, 18 operation recognition unit, 19 presentation information acquisition unit, 20 presentation information storage unit, 21 presentation information generation unit, 22 presentation timing control unit, 23 presentation timing storage unit, 30 camera, 40 operation Terminal, 50: external content DB, 60: output device, 70: external content.

Claims (8)

Image acquisition means for acquiring a target image which is an image relating to a processing target including a user ,
Feature point detection means for detecting feature point information from the acquired target image,
Feature point information storage means for storing the detected feature point information;
Difference extracting means for extracting difference information between frames of the stored feature point information,
And theme selection means for selecting a theme information indicating the work theme of the user with respect to the processing target,
Presentation information storage means for storing presentation information for the user;
Based on the difference information and the selected theme information, a situation recognition unit that recognizes a current situation related to the work , among situations related to each of a plurality of stages including before to after work by the user. ,
Based on the difference information and the selected theme information, a target area recognition unit that recognizes a target area that is a part where a user works on the processing target,
An operation recognizing unit that recognizes the operation of the user based on the difference information;
When the difference information is extracted by the difference extraction unit, based on a result of recognition by the situation recognition unit, information on the target region recognized by the target region recognition unit, and a result of recognition by the operation recognition unit. Presentation information acquisition means for acquiring presentation information to the user from presentation information stored in the presentation information storage means;
Presentation information generating means for generating information obtained by performing predetermined processing on the obtained presentation information,
Presentation timing storage means for storing information about the timing of presenting the presentation information for each presentation content;
Based on the information generated by the presentation information generation unit, information on the timing of presenting the presentation information is obtained from the presentation timing storage unit, and based on this information, the presentation information subjected to a predetermined process is obtained. A technical support device comprising: a presentation timing control unit that outputs to an output device. The difference extracting means,
Extracting the difference information based on the feature point information,
Further comprising a reference state storage means for storing status information indicating the status of the work to be referred to by the user,
The situation recognition means,
2. The technical support device according to claim 1, further comprising a status determination unit configured to determine a status of the user's work based on the difference information and the status information. The processing target includes a tool used by the user for work,
The difference extracting means,
Extracting the difference information based on the feature point information,
A tool used by the user for work, and tool information storage means for storing tool information indicating a relationship between target areas which are targets of work using the tool,
The target area recognition means,
A target area acquisition unit that acquires a target area that is a target of work using the tool based on the difference information based on the feature point information,
Tool recognition means for acquiring information on the target area and the tool, which is a target of work using the tool , based on the difference information and the tool information based on the feature point information,
A target area determination for determining information of the target area and the tool obtained by the target area and the tool recognition means obtained by the target area obtaining means, and outputting a determination result as a result of recognition by the target area recognition means; 2. The technical support device according to claim 1, further comprising: means. The reference state storage means,
Storing the status information for each type of the theme;
The situation determination means includes:
Determining, based on the theme information selected by the theme selecting means, the difference information based on the feature point information, and the situation information, a situation of the user's work related to the theme information selected by the theme selecting means; The technical support device according to claim 2, wherein: The tool information storage means,
Storing the tool information for each type of the theme;
The target area acquisition means,
Acquiring the target area based on the difference information based on the theme information and the feature point information selected by the theme selection means,
The tool recognition means,
4. The technical support device according to claim 3, wherein information on the target area and the tool is acquired based on the theme information selected by the theme selection unit and the difference information based on the feature point information. A method applied to a technical support apparatus having presentation information storage means for storing presentation information for a user,
Obtaining a target image that is an image related to a processing target including the user ,
Detecting feature point information from the acquired target image,
The detected feature point information is stored in feature point information storage means,
Extracting difference information between frames of the stored feature point information,
Select a theme information indicating the work theme of the user with respect to the processing target,
Based on the difference information and the selected theme information, among the situations relating to each of a plurality of stages including from before the work to after the work by the user , the current situation related to the work is recognized,
Based on the difference information and the selected theme information, recognizing a target area that is a part where a user works on the processing target,
Based on the difference information, recognize the operation of the user,
When the difference information is extracted, the presentation information stored in the presentation information storage means based on the recognition result of the situation, the recognized information of the target area, and the recognition result of the user's operation. Obtain presentation information to the user from
Generate information obtained by performing predetermined processing on the obtained presentation information,
On the basis of the generated information, for each of the presentation contents, from the presentation timing storage means for storing information on the timing of presenting the presentation information, obtain information on the timing of presenting the presentation information, and And outputting presentation information subjected to predetermined processing to an output device.   A program for causing a computer to execute as each unit of the technical support device according to any one of claims 1 to 5. A technical support system including a client and a server,
The server is
Feature point detection means for detecting feature point information from a target image that is an image relating to a processing target including a user ;
Difference extracting means for extracting difference information between frames of the feature point information,
Feature point information storage means for storing the detected feature point information;
Based on the difference information and the processing theme information indicating the work theme of the user for the subject, among the conditions according to each of a plurality of steps up to and including post-work before the work by the user, the current according to the working Situation recognition means for recognizing the situation;
A target area recognition unit that recognizes a target area that is a part where the user works on the processing target based on the presentation information and the theme information for the user;
An operation recognizing unit that recognizes the operation of the user based on the difference information;
When the difference extracting means extracts the difference information, the presentation information is presented to the user from the presentation information based on the result of the recognition by the situation recognizing means and the information of the target area recognized by the target area recognizing means. Presentation information acquisition means for acquiring information;
Presentation information storage means for storing the presentation information;
Presentation information generating means for generating information obtained by performing predetermined processing on the obtained presentation information,
A step of obtaining information relating to the timing of presenting the presentation information based on the information output by the presentation information generating means, and outputting the presentation information subjected to a predetermined process to the output device based on the information; Timing control means;
A presentation timing storage unit for storing information on a timing of presenting the presentation information for each presentation content,
The client,
Image acquisition means for acquiring the target image,
A technical selecting system for selecting the theme information.