JP7433126B2 - Image display system, image display device, server, image display method and program - Google Patents

Description

The present disclosure relates to an image display system, an image display device, a server, an image display method, and a program.

There is a technology that provides a user with an image in which information about an object in the image is superimposed on a captured image. For example, in Patent Document 1, an image obtained by adding an effect image obtained based on an AR (Augmented Reality) trigger attached to an accessory of the user to an image of the user obtained by the camera function of a mobile terminal is displayed to the user. An image sharing system is disclosed.

Japanese Patent Application Publication No. 2014-229090

In the image sharing system described in Patent Document 1, an AR trigger attached to a user's accessory specifies an effect image to be superimposed on a photographed image of the user. For this reason, it has not been possible to provide a user with an image in which information corresponding to the movement of the object, such as a person or a machine, is superimposed on an image of the movement of the object, such as a person or a machine.

An object of the present disclosure is to provide a user with an image in which information corresponding to the movement of a photographed object is superimposed on an image of the movement of the object.

In order to achieve the above object, an image display system according to the present disclosure includes an image display device that displays captured images, and a server connected to the image display device. The image display device includes a photographing section, a feature point extraction section, and a feature point physical quantity information transmission section. The photographing unit photographs images. If the target is included in the image taken by the imaging unit, the feature point extraction unit extracts the feature points of the target from the image taken by the imaging unit, and generates time-series feature point physical quantity information indicating the physical quantities of the feature points. do. The feature point physical quantity information transmitting unit transmits the feature point physical quantity information to the server. The server includes a feature point physical quantity information receiving section, a motion correspondence information storage section, a motion information collation section, a motion card correspondence information storage section, a motion card information collation section, and a motion card information transmission section. The feature point physical quantity information receiving unit receives feature point physical quantity information from the image display device. The motion correspondence information storage unit stores motion correspondence information in which information indicating a series of motions is associated with motion information indicating the motion. The motion information matching unit refers to the motion correspondence information and identifies motion information corresponding to the motion of the target based on the feature point physical quantity information. The action card correspondence information storage section stores action card correspondence information in which action information is associated with action card information that describes the action indicated by the action information. The operational card information collation section refers to the operational card correspondence information and identifies operational card information corresponding to the operational information specified by the operational information collation section. The operational card information transmitting section transmits the operational card information specified by the operational card information matching section to the image display device. The image display device includes an operation card information receiving section, an image composition section, and a display section. The operating card information receiving unit receives operating card information from the server. The image composition section generates a composite image by superimposing the operation card information on the image photographed by the photographing section. The display section displays the image photographed by the photographing section and the composite image generated by the image composition section. The feature point physical quantity information includes time-series information indicating distances between feature points. The motion information matching unit calculates the relative ratio of the distance between the feature points and the difference between the relative ratio of the distance between the feature points and the previous point in time based on the time series information indicating the distance between the feature points. relative ratio difference information is generated, and motion information corresponding to the movement of the target indicated by the relative ratio difference information is identified.

According to the present disclosure, an action corresponding to the movement of the object is identified based on a temporal change in the physical quantity of the feature point of the object extracted from an image taken by an image display device, and action card information explaining the identified action is provided. To provide a user with an image in which information corresponding to the movement of the photographed object is superimposed on the image of the movement of the object by superimposing it on the photographed image and displaying it on an image display device. I can do it.

A diagram illustrating a configuration example of an image display system according to an embodiment. A diagram showing an example of a functional configuration of an image display device according to an embodiment. A diagram showing an example of inter-joint distance information according to an embodiment A diagram showing an example of a functional configuration of a server according to an embodiment A diagram showing an example of relative ratio difference information according to an embodiment A diagram showing an example of work correspondence information according to an embodiment A diagram illustrating an example of work information matching results according to an embodiment A diagram showing an example of work card correspondence information according to an embodiment A diagram showing an example of a work card information matching result according to an embodiment A diagram showing an example of a work card information transmission history according to an embodiment A diagram showing an example of a composite image according to an embodiment Flowchart showing screen display processing according to the embodiment Flowchart showing work card information transmission processing according to the embodiment A diagram illustrating an example of a hardware configuration of an image display device according to an embodiment. A diagram illustrating an example of the hardware configuration of a server according to an embodiment. A diagram showing an example of work correspondence information according to a modified example A diagram showing an example of work card correspondence information according to a modification

Below, an image display system, an image display device, a server, an image display method, and a program according to the present embodiment will be described in detail with reference to the drawings. In addition, the same reference numerals are given to the same or corresponding parts in the figures. In this embodiment, the content of the work is identified from an image taken of a worker working in a work place, and an image in which information explaining the work content is superimposed on the image of the worker is provided to the user. An example will be explained below. In this embodiment, the movement of the worker is defined by a change over time in the relative ratio of distances between joint points.

The configuration of an image display system 100 according to this embodiment will be explained using FIG. 1. As shown in FIG. 1, the image display system 100 includes an image display device 1 used by a user in a work place W, and a server 2 connected to the image display device 1 via a network 3. In the figure, one representative workplace W and one image display device 1 are shown, but there may be multiple workplaces W, and there may be multiple image display devices 1 used by users within the workplace W. There may be. The image display device 1 may be a head mounted display (HMD) as shown in FIG. 1, or may be a terminal such as a tablet or a smartphone. Alternatively, the image display device 1 may be a device that separately includes a display panel, a computer, and a camera, and a user holds the display panel and wears the computer and camera on his or her body.

The functional configuration of the image display device 1 will be explained using FIG. 2. The image display device 1 includes a photographing unit 11 that photographs images, a joint point extracting unit 12 that extracts joint points of a worker from the photographed images, and a time series of inter-joint distances indicating distances between the extracted joint points. and an inter-joint distance information transmitter 13 that transmits information. A worker is an example of a target. The joint point extraction unit 12 is an example of a feature point extraction unit. Joint points are an example of feature points. The distance between joint points is an example of a physical quantity of a feature point. Inter-joint distance information is an example of feature point physical quantity information. The inter-joint distance information transmitter 13 is an example of a feature point physical quantity information transmitter.

The image display device 1 also includes a work card information receiving unit 14 that receives work card information explaining the work content from the server 2, and generates a composite image in which the work card information is superimposed on the image photographed by the photographing unit 11. It includes an image composition section 15 and a display section 16 that displays the image photographed by the photographing section 11 and the composite image generated by the image composition section 15. A task is an example of an action. Work card information is an example of action card information. The work card information receiving section 14 is an example of an operation card information receiving section.

First, when the user starts the image display device 1, the photographing unit 11 of the image display device 1 starts photographing. The photographing section 11 sends the photographed image to the joint point extraction section 12 and the image composition section 15. When the user turns the photographing direction of the photographing section 11 of the image display device 1 toward the worker who is working, an image of the worker who is working is sent to the joint point extraction section 12 and the image composition section 15. It will be done. For example, if the image display device 1 is a head-mounted device and the photographing unit 11 is placed at a position to photograph the front while the image display device 1 is worn, the user wearing the image display device 1 may be working. If the camera is facing the worker, an image of the worker working is sent to the joint point extraction section 12 and the image composition section 15. For example, if the image display device 1 is a terminal such as a tablet or a smartphone and a camera with a built-in photographing unit 11, if the user points the camera at the worker who is working, the user can An image of the worker is sent to the joint point extraction section 12 and the image composition section 15.

The image composition section 15 causes the display section 16 to display the image received from the photographing section 11. The joint point extraction unit 12 extracts joint points indicating the positions of the worker's joints from the image received from the imaging unit 11 at predetermined time intervals, and extracts the date and time when the joint points were extracted and the joint points on the image. Inter-joint distance information indicating the distance between the joints is generated. If the image received from the imaging unit 11 does not include a person, the joint point extraction unit 12 cannot extract joint points, and therefore does not generate inter-joint distance information. The inter-joint distance information transmitter 13 sequentially transmits the inter-joint distance information generated by the joint point extractor 12 to the server 2 .

Inter-joint distance information will be explained using FIG. 3. FIG. 3 shows an example of inter-joint distance information for 10 seconds when joint points are extracted at 1-second intervals. For example, the distance between joint point A and joint point B on the image at 0:00:00 on 2019/12/5 is 10 (cm). The distance between joint point C and joint point D on the image at the same time is 30 (cm). The distance between joint point A and joint point B on the image at 0:00:01 on 2019/12/5 is 20 (cm). The distance between joint point C and joint point D on the image at the same time is 40 (cm).

Here, the functional configuration of the server 2 will be explained using FIG. 4. The server 2 includes an inter-joint distance information receiving unit 21 that receives inter-joint distance information from the image display device 1, and work information that specifies work information indicating work corresponding to the movement of the worker based on the inter-joint distance information. It includes a collation unit 22 and a work correspondence information storage unit 23 that stores work correspondence information in which information indicating a series of work movements and work information are associated with each other. The inter-joint distance information receiving section 21 is an example of a feature point physical quantity information receiving section. Work information is an example of operation information. The work information matching section 22 is an example of an operation information matching section. Work correspondence information is an example of action correspondence information. The work correspondence information storage section 23 is an example of an operation correspondence information storage section.

The server 2 also includes a work card information matching unit 24 that identifies work card information corresponding to the identified work information, and work card correspondence information that stores work card correspondence information that associates work information and work card information. It includes a storage unit 25 and a work card information transmitting unit 26 that transmits specified work card information to the image display device 1. Work card information is an example of action card information. The work card information verification section 24 is an example of an operation card information verification section. The work card correspondence information is an example of action card correspondence information. The work card correspondence information storage section 25 is an example of an operation card correspondence information storage section. The work card information transmitter 26 is an example of an operation card information transmitter.

When the inter-joint distance information receiving section 21 receives the inter-joint distance information from the image display device 1, it sends it to the work information matching section 22. Each time the work information matching unit 22 receives inter-joint distance information from the inter-joint distance information receiving unit 21, it calculates the relative ratio of the distance between each joint point and the distance between each joint point based on the inter-joint distance information. Relative ratio difference information indicating the difference between the relative ratio of distance and the previous point in time is generated. FIG. 5 shows an example of relative ratio difference information generated based on the inter-joint distance information shown in FIG. 3. In the example of FIG. 5, the work information collation unit 22 assigns an ID in the order of generation to the relative ratio difference information, but when determining the time series of the relative ratio difference information based on the date and time when the joint points were extracted, , it is not necessary to assign an ID.

In the example of FIG. 5, the work information collation unit 22 sets the distance 10 (cm) between joint point A and joint point B on the image indicated by the relative ratio difference information with ID 1 as "1", and sets each joint point to "1". The relative ratio of the distance between is calculated. For example, the relative ratio of the distance between joint point A and joint point B indicated by the relative ratio difference information with ID 2 is "2", and the relative ratio of the distance between joint point A and joint point B indicated by the relative ratio difference information with ID 1 at the previous point in time is "2". Since the relative ratio of the distance to point B is "1", the difference is "-1". In this way, the relative ratio difference information indicates the movement of the worker, which is the change over time in the relative ratio of the distance between each joint point. In the example of FIG. 5, the difference in relative ratio is the value obtained by subtracting the relative ratio of the distances between the joint points from the relative ratio of the distances between the joint points at the previous time, but the difference is not limited to this. It may be a value obtained by subtracting the relative ratio of the distance between the joint points at the previous point in time from the relative ratio of the distance between the joint points.

The work information collation unit 22 refers to the work correspondence information stored in the work correspondence information storage unit 23 and identifies the work information of the work corresponding to the movement of the worker indicated by the generated relative ratio difference information. Specifically, the work information matching unit 22 determines whether the movement of the worker indicated by the relative ratio difference information matches a series of movements of the work indicated by the work information, and determines whether the movement of the worker indicated by the relative ratio difference information matches Identify work information in which the movements of the person and the sequence of movements of the work match.

The work correspondence information will be explained using FIG. 6. FIG. 6 shows an example of work correspondence information in which information indicating a series of movements of a work in units of 1 second is associated with work information. For example, the relative ratio of the distance between joint point A and joint point B is "1", the relative ratio of the distance between joint point C and joint point D is "3", and the relative ratio of the distance between joint point E and joint point F is "1". A posture in which the ratio is "2" and the relative ratio of the distances between the joint points G and the joint points H is "6" is defined as a posture α. The relative ratio of the distance between joint point A and joint point B is "2", the relative ratio of the distance between joint point C and joint point D is "4", and the relative ratio of the distance between joint point E and joint point F. is "2" and the relative ratio of the distance between the joint point G and the joint point H is "5" as the posture β. The relative ratio of the distances between joint points A and B is "3", the relative ratio of the distances between joint points C and D is "5", and the relative ratio of the distances between joint points E and F is "3". The posture in which the relative ratio of the distance between the joint point G and the joint point H is "2" and "4" is defined as the posture γ. A series of movements from posture α to posture β and from posture β to posture γ correspond to the work information of work (1).

The work information collation result in which the work information collation unit 22 identifies the work information of the work corresponding to the movement of the worker indicated by the relative ratio difference information will be explained using FIG. FIG. 7 shows work information collation in which work information corresponding to the movement of the worker indicated by the relative ratio difference information shown in FIG. 5 is identified by referring to the work correspondence information stored in the work correspondence information storage unit 23 shown in FIG. Here is an example of the results. In the example of FIG. 7, the work information for work (1) corresponds to the movement of the worker indicated by the relative ratio difference information with IDs 1 to 3. The work information of work (2) corresponds to the movement of the worker indicated by the relative ratio difference information of IDs 4 and 5. The work information for work (3) corresponds to the movement of the worker indicated by the relative ratio difference information with IDs 8 to 11. The relative ratio difference information with IDs 6 and 7 indicates movements of the worker other than work.

Returning to FIG. 4, the work information matching unit 22 sends the identified work information to the work card information matching unit 24. The work card information collation unit 24 refers to the work card correspondence information stored in the work card correspondence information storage unit 25 and identifies work card information corresponding to the received work information.

The work card correspondence information will be explained using FIG. 8. In the example of the work card correspondence information shown in FIG. 8, the work card information of work cards 001 to 010 corresponds to the work information of work (1) to work (10), respectively. The work card information of work cards 001 to 010 is information that explains the contents of work (1) to work (10), respectively. The work card information may include not only information explaining the content of the work, but also information related to the work, such as the situation at the work site and the equipment to be used.

The work card information collation result in which the work card information collation unit 24 identifies the work card information corresponding to the work information will be explained using FIG. 9. FIG. 9 shows an example of a work card information collation result in which work card information corresponding to the work information shown in FIG. 7 is identified by referring to the work card correspondence information stored in the work card correspondence information storage unit 25. In the example of FIG. 9, the work card information of work card 001 corresponds to the work information of work (1) corresponding to the movement of the worker indicated by the relative ratio difference information of IDs 1 to 3. The work card information of work card 002 corresponds to the work information of work (2) corresponding to the movement of the worker indicated by the relative ratio difference information of IDs 4 and 5. The work card information of the work card 003 corresponds to the work information of the work (3) corresponding to the movement of the worker indicated by the relative ratio difference information of IDs 8 to 11.

Returning to FIG. 4, the work card information matching unit 24 sends the identified work card information to the work card information transmitting unit 26. The work card information transmitting unit 26 transmits the received work card information to the image display device 1.

Here, the work card information transmission history in which the work card information transmitter 26 has transmitted work card information will be described using FIG. 10. At the time when the relative ratio difference information with IDs 1 to 3 is generated, the work information matching unit 22 determines that the movement of the worker indicated by the relative ratio difference information matches the series of movements of work (1) indicated by the work information. Then, the work information of work (1) is specified. When the work information collation unit 22 specifies the work information of work (1), the work card information collation unit 24 specifies the work card information of the work card 001. When the work card information matching unit 24 identifies the work card information of the work card 001, the work card information transmitting unit 26 transmits the work card information of the work card 001 to the image display device 1. That is, when the joint distance information receiving unit 21 receives the joint distance information at 0:00:02 on 2019/12/5, the work card information transmitting unit 26 transmits the work card information of the work card 001 to the image display device 1. Send. It is assumed that the processing time from when the server 2 receives the inter-joint distance information until it transmits the work card information is sufficiently shorter than the time interval at which the inter-joint distance information is received.

Similarly, regarding the work card information of work card 002, when the inter-joint distance information receiving unit 21 receives the inter-joint distance information at 0:00:04 on 2019/12/5, the work card information transmitting unit 26 transmits the work card information of work card 002. The work card information of is transmitted to the image display device 1. Similarly, regarding the work card information of work card 003, when the inter-joint distance information receiving unit 21 receives the inter-joint distance information at 0:00:10 on 2019/12/5, the work card information transmitting unit 26 transmits the work card information of work card 003. The work card information of is transmitted to the image display device 1. In this way, work card information is transmitted to the image display device 1 every time a series of movements for each work indicated by the work information is completed.

Returning to FIG. 2, when the work card information receiving unit 14 of the image display device 1 receives the work card information from the server 2, it sends it to the image composition unit 15. The image composition section 15 causes the display section 16 to display a composite image in which the work card information received from the work card information reception section 14 is superimposed on the image received from the photographing section 11. The time for displaying the work card information may be, for example, the shortest time required for a series of work movements, or it may be until the next work card information is received. The position where the work card information is displayed may be, for example, a position that does not overlap with the extracted joint point, or may be a fixed position. The time and position for displaying the work card information may be set in advance or may be set by the user.

Next, the image displayed on the display unit 16 will be explained using FIG. 11. While the work card information receiving section 14 of the image display device 1 is not receiving the work card information, the image photographed by the photographing section 11 is displayed on the display section 16. Every time the work card information receiving unit 14 receives work card information, the image combining unit 15 adds the work card information received from the work card information receiving unit 14 to the image 41 received from the photographing unit 11, as shown in FIG. A composite image 4 obtained by superimposing 42 is displayed on the display section 16. That is, the image 41 photographed by the photographing section 11 is displayed on the display section 16 in real time, and work card information 42 explaining the work content is displayed each time the worker in the photographed image completes the work. It is displayed superimposed on the image 41. This allows the user to understand what kind of work the worker was doing.

Here, the flow of processing executed by the image display device 1 will be explained using FIG. 12. The image display process shown in FIG. 12 starts when the image display device 1 is started. The photographing unit 11 of the image display device 1 photographs an image (step S11). Step S11 is an example of a photographing step. The photographing section 11 sends the photographed image to the joint point extraction section 12 and the image composition section 15. The image composition section 15 causes the display section 16 to display the image received from the photographing section 11 (step S12).

If the worker is not included in the image received from the photographing unit 11 (step S13; NO), the process moves to step S18. If the worker is included in the image received from the imaging unit 11 (step S13; YES), the joint point extraction unit 12 extracts joint points indicating the positions of the worker's joints from the image received from the imaging unit 11. Then, inter-joint distance information indicating the distance between joint points on the image is generated (step S14). Step S14 is an example of a feature point extraction step. The inter-joint distance information transmitter 13 transmits the inter-joint distance information generated by the joint point extractor 12 to the server 2 (step S15). Step S15 is an example of a step of transmitting feature point physical quantity information.

In the example of the inter-joint distance information in FIG. 3, the distance between joint point A and joint point B on the image at 0:00:00 on 2019/12/5 is 10 (cm). The distance between joint point C and joint point D on the image at the same time is 30 (cm). The distance between joint point A and joint point B on the image at 0:00:01 on 2019/12/5 is 20 (cm). The distance between joint point C and joint point D on the image at the same time is 40 (cm).

Returning to FIG. 12, if the work card information receiving unit 14 does not receive the work card information from the server 2 (step S16; NO), the process moves to step S18. When the work card information is received from the server 2 (step S16; YES), the work card information receiving section 14 sends the received work card information to the image composition section 15. Step S16 is an example of an operation card information receiving step. The image composition section 15 causes the display section 16 to display a composite image in which the work card information received from the work card information reception section 14 is superimposed on the image received from the photographing section 11 (step S17). Step S17 is an example of an image composition step and a display step.

In the example of FIG. 11, a composite image 4 in which the work card information 42 of the work card 001 received from the work card information receiving unit 14 is superimposed on the image 41 received from the photographing unit 11 is displayed on the image display device 1.

Returning to FIG. 12, if the image display device 1 is not stopped (step S18; NO), the process returns to step S11 and repeats steps S11 to S18. When the image display device 1 stops (step S18; YES), the process ends.

Next, the flow of processing executed by the server 2 will be explained using FIG. 13. The work card information transmission process shown in FIG. 13 starts when the server 2 is powered on. If the inter-joint distance information receiving unit 21 of the server 2 does not receive the inter-joint distance information from the image display device 1 (step S21; NO), the process moves to step S27. When the inter-joint distance information receiving unit 21 receives the inter-joint distance information from the image display device 1 (step S21; YES), the work information matching unit 22 calculates the distance between each joint point based on the inter-joint distance information. and the difference between the relative ratio of the distance between each joint point and the previous point in time (step S22). Step S21 is an example of a feature point physical quantity information receiving step.

In the example of the relative ratio difference information in FIG. 5, the work information matching unit 22 calculates the relative ratio of the distance between each joint point, setting the distance 10 (cm) between the joint points on the image as 1. There is. For example, the relative ratio of the distance between joint point A and joint point B indicated by relative ratio difference information with ID 2 is 2, and joint point A and joint point B indicated by relative ratio difference information with ID 1 at the previous time point. Since the relative ratio of the distance between the two is 1, the difference is -1. In this way, the relative ratio difference information indicates the movement of the worker, which is the change over time in the relative ratio of the distance between each joint point.

Returning to FIG. 13, the work information matching unit 22 refers to the work correspondence information stored in the work correspondence information storage unit 23, and matches the movement of the worker indicated by the relative ratio difference information with the series of movements of the work indicated by the work information. It is determined whether they match (step S23). If the sequence of movements of the work indicated by the work information does not match (step S23; NO), the process moves to step S27. If the work information matches the series of movements of the work indicated (step S23; YES), the work information collation unit 22 identifies the work information in which the movement indicated by the relative ratio difference information matches the series of movements of the work (step S23; YES). S24). Step S23 and step S24 are examples of operation information matching steps.

FIG. 7 is an example of a work information collation result in which work information of a work corresponding to the movement of the worker indicated by the relative ratio difference information shown in FIG. 5 is identified. In the example of FIG. 7, the work information for work (1) corresponds to the movement of the worker indicated by the relative ratio difference information with IDs 1 to 3. The work information of work (2) corresponds to the movement of the worker indicated by the relative ratio difference information of IDs 4 and 5. The work information for work (3) corresponds to the movement of the worker indicated by the relative ratio difference information with IDs 8 to 11. The relative ratio difference information with IDs 6 and 7 indicates movements of the worker other than work.

Returning to FIG. 13, the work information matching unit 22 sends the identified work information to the work card information matching unit 24. The work card information matching section 24 refers to the work card correspondence information stored in the work card correspondence information storage section 25 and identifies the work card information corresponding to the received work information (step S25). Step S25 is an example of an operation card information verification step.

FIG. 9 is an example of a work card information collation result in which work card information corresponding to the work information shown in FIG. 7 is identified by referring to the work card correspondence information stored in the work card correspondence information storage unit 25. In the example of FIG. 9, the work card information of work card 001 corresponds to the work information of work (1) corresponding to the movement of the worker indicated by the relative ratio difference information of IDs 1 to 3. The work card information of work card 002 corresponds to the work information of work (2) corresponding to the movement of the worker indicated by the relative ratio difference information of IDs 4 and 5. The work card information of the work card 003 corresponds to the work information of the work (3) corresponding to the movement of the worker indicated by the relative ratio difference information of IDs 8 to 11. The work card information of work card 001 is information that describes the content of work (1). The work card information of work card 002 is information that describes the content of work (2). The work card information of the work card 003 is information explaining the content of work (3). The work card information may include not only information explaining the content of the work, but also information related to the work, such as the situation at the work site and the equipment to be used.

Returning to FIG. 13, the work card information matching unit 24 sends the identified work card information to the work card information transmitting unit 26. The work card information transmitting unit 26 transmits the received work card information to the image display device 1 (step S26). Step S26 is an example of an operation card information transmission step.

FIG. 10 is an example of a work card information transmission history in which the work card information transmitter 26 has transmitted work card information. In the example of FIG. 10, when the inter-joint distance information receiving unit 21 receives the inter-joint distance information at 0:00:02 on 2019/12/5, the work card information transmitting unit 26 sends the work card information of the work card 001 as an image. It is being sent to display device 1. When the joint distance information receiving unit 21 receives the joint distance information at 0:00:04 on 2019/12/5, the work card information transmitting unit 26 transmits the work card information of the work card 002 to the image display device 1. ing. When the inter-joint distance information receiving unit 21 receives the inter-joint distance information at 0:00:10 on 2019/12/5, the work card information transmitting unit 26 transmits the work card information of the work card 003 to the image display device 1. ing. In this way, work card information is transmitted to the image display device 1 every time a series of movements for each work indicated by the work information is completed.

Returning to FIG. 13, if the server 2 is not powered off (step S27; NO), the process returns to step S21 and repeats steps S21 to S27. When the power is turned off (step S27; YES), the process ends.

According to the image display system 100 according to the present embodiment, a task corresponding to the movement of the worker is performed based on the temporal change in the distance between the joint points of the worker extracted from the image taken by the image display device 1. By superimposing work card information explaining the identified work on the photographed image and displaying it on the image display device 1, the photographed worker's movement can be displayed on the photographed image of the worker's movement. An image on which information corresponding to movement is superimposed can be provided to the user.

As a result, for example, when explaining the site where workers are working at a work site to new users, visitors, and other users, there is no need for a veteran worker or other person who knows the work and can explain the work to be present. Users can know the details of the worker's work, reducing personnel costs. Furthermore, since the contents of work can be grasped simultaneously at the site where workers are working, it becomes much easier to investigate the current situation in order to improve work efficiency.

The hardware configuration of the image display device 1 will be explained using FIG. 14. As shown in FIG. 14, the image display device 1 includes a temporary storage section 101, a storage section 102, a calculation section 103, an input section 104, a transmission/reception section 105, a display section 106, and a sensor section 107. Temporary storage section 101, storage section 102, input section 104, transmission/reception section 105, display section 106, and sensor section 107 are all connected to calculation section 103 via BUS.

The calculation unit 103 is, for example, a CPU (Central Processing Unit). The calculation unit 103 executes each process of the joint point extraction unit 12 and the image composition unit 15 of the image display device 1 according to the control program stored in the storage unit 102.

The temporary storage unit 101 is, for example, a RAM (Random-Access Memory). The temporary storage unit 101 loads the control program stored in the storage unit 102 and is used as a work area for the calculation unit 103.

The storage unit 102 is a nonvolatile memory such as a flash memory, a hard disk, a DVD-RAM (Digital Versatile Disc-Random Access Memory), a DVD-RW (Digital Versatile Disc-ReWritable), or the like. The storage unit 102 stores in advance a program for causing the calculation unit 103 to perform the processing of the server 2, and also supplies data stored by this program to the calculation unit 103 according to instructions from the calculation unit 103. Stores data supplied by.

The input unit 104 is an input device such as a keyboard, operation buttons, or voice input device. Information input by the user is supplied to the calculation unit 103 via the input unit 104 . For example, in a configuration where the user sets the time and position for displaying the work card information on the image display device 1, the user inputs the time and position for displaying the work card information into the input unit 104.

The transmitting/receiving unit 105 is a network terminal device or a wireless communication device connected to a network, and a serial interface or a LAN (Local Area Network) interface connected thereto. The transmitter/receiver 105 functions as the inter-joint distance information transmitter 13 and the work card information receiver 14 .

The display unit 106 is a display device such as an LCD (Liquid Crystal Display) or an organic EL (electroluminescence) display. The display section 106 functions as the display section 16.

The sensor unit 107 is a photographing device such as a digital camera or a depth sensor. The sensor unit 107 functions as the imaging unit 11.

The processing of the photographing section 11, joint point extraction section 12, inter-joint distance information transmission section 13, work card information reception section 14, image composition section 15, and display section 16 of the image display device 1 shown in FIG. The processing is executed using the temporary storage section 101, the calculation section 103, the storage section 102, the input section 104, the transmission/reception section 105, the display section 106, the sensor section 107, etc. as resources.

In addition, the above hardware configuration and flowchart are merely examples, and can be changed and modified as desired.

The main parts that perform processing of the image display device 1, such as the calculation section 103, temporary storage section 101, storage section 102, input section 104, transmission/reception section 105, display section 106, and sensor section 107, do not rely on a dedicated system. , can be realized using an ordinary computer system. For example, a computer program for executing the above operations may be stored on a computer-readable recording medium such as a flexible disk, a CD-ROM (Compact Disc - Read Only Memory), or a DVD-ROM (Digital Versatile Disc - Read Only Memory). The image display device 1 that executes the above processing may be configured by storing and distributing the computer program in a computer and installing the computer program in the computer. Alternatively, the image display device 1 may be constructed by storing the computer program in a storage device included in a server device on a communication network such as the Internet, and downloading the computer program by a normal computer system.

In addition, when the functions of the image display device 1 are realized by sharing the OS (Operating System) and application programs, or by cooperating with the OS and application programs, only the application program portion is stored in a recording medium or storage device. You may.

It is also possible to superimpose a computer program on a carrier wave and provide it via a communication network. For example, the computer program may be posted on a bulletin board (BBS, Bulletin Board System) on a communication network, and the computer program may be provided via the communication network. Then, by starting this computer program and executing it in the same way as other application programs under the control of the OS, the above-mentioned processing may be executed.

The hardware configuration of the server 2 will be explained using FIG. 15. As shown in FIG. 15, the server 2 includes a temporary storage section 201, a storage section 202, a calculation section 203, an input section 204, a transmission/reception section 205, and a display section 206. The temporary storage section 201, the storage section 202, the input section 204, the transmission/reception section 205, and the display section 206 are all connected to the calculation section 203 via the BUS.

The calculation unit 203 is, for example, a CPU (Central Processing Unit). The calculation unit 203 executes each process of the work information verification unit 22 and the work card information verification unit 24 of the server 2 according to the control program stored in the storage unit 202.

The temporary storage unit 201 is, for example, a RAM (Random-Access Memory). The temporary storage unit 201 loads the control program stored in the storage unit 202 and is used as a work area for the calculation unit 203.

The storage unit 202 is a nonvolatile memory such as a flash memory, a hard disk, a DVD-RAM (Digital Versatile Disc-Random Access Memory), a DVD-RW (Digital Versatile Disc-ReWritable), or the like. The storage unit 202 stores in advance a program for causing the calculation unit 203 to perform the processing of the server 2, and also supplies data stored by this program to the calculation unit 203 according to instructions from the calculation unit 203. Stores data supplied by. The work correspondence information storage section 23 and the work card correspondence information storage section 25 are configured in the storage section 202 .

The input unit 204 is an input device such as a keyboard, a pointing device, and a voice input device, and an interface device that connects the input device to the BUS. For example, when a user directly inputs information to the server 2, the information input by the user is supplied to the calculation unit 203 via the input unit 204.

The transmitting/receiving unit 205 is a network terminal device or a wireless communication device connected to a network, and a serial interface or a LAN (Local Area Network) interface connected thereto. The transmitting/receiving section 205 functions as the inter-joint distance information receiving section 21 and the work card information transmitting section 26.

The display unit 206 is a display device such as an LCD (Liquid Crystal Display) or an organic EL (electroluminescence) display. For example, when a user directly inputs information to the server 2, an input screen is displayed on the display unit 206.

The inter-joint distance information receiving section 21, the work information collation section 22, the work correspondence information storage section 23, the work card information verification section 24, the work card correspondence information storage section 25, and the work card information transmission section of the server 2 shown in FIG. The process No. 26 is executed by the control program using the temporary storage section 201, calculation section 203, storage section 202, input section 204, transmission/reception section 205, display section 206, etc. as resources.

In addition, the above hardware configuration and flowchart are merely examples, and can be changed and modified as desired.

The main parts that perform the processing of the server 2, such as the calculation section 203, temporary storage section 201, storage section 202, input section 204, transmission/reception section 205, and display section 206, are not based on a dedicated system but on a normal computer system. It can be realized using For example, a computer program for executing the above operations may be stored on a computer-readable recording medium such as a flexible disk, a CD-ROM (Compact Disc - Read Only Memory), or a DVD-ROM (Digital Versatile Disc - Read Only Memory). The server 2 that executes the above processing may be configured by storing and distributing the computer program in a computer and installing the computer program on a computer. Alternatively, the computer program may be stored in a storage device of a server device on a communication network such as the Internet, and the server 2 may be configured by downloading the program from a normal computer system.

In addition, when the functions of the server 2 are realized by sharing the OS (Operating System) and application programs, or by cooperating with the OS and application programs, only the application program portion is stored in a recording medium or storage device. Good too.

It is also possible to superimpose a computer program on a carrier wave and provide it via a communication network. For example, the computer program may be posted on a bulletin board (BBS, Bulletin Board System) on a communication network, and the computer program may be provided via the communication network. Then, by starting this computer program and executing it in the same way as other application programs under the control of the OS, the above-mentioned processing may be executed.

In the above embodiment, the server 2 previously stored the work correspondence information and the work card correspondence information, but the work correspondence information and the work card information may be made editable.

When the work correspondence information is made editable, for example, as shown in FIG. 16, the work correspondence information includes a deletion flag. The user inputs "1" into the deletion flag of information indicating a series of operations to be deleted. The user can also return the deletion flag from "1" to "0". When identifying work information indicating a task corresponding to a worker's movement, the work information matching unit 22 excludes information indicating a series of movements of a task whose deletion flag is "1". Alternatively, the configuration may be such that the user can completely delete part of the information indicating a series of work movements or the work correspondence information. Further, the user may add work correspondence information, or may add or change the value of the relative ratio of distances between joints in information indicating a series of movements in a work.

When the work card correspondence information is made editable, for example, as shown in FIG. 17, the work card correspondence information includes a deletion flag. The user inputs "1" into the deletion flag of the work card information to be deleted. The user can also return the deletion flag from "1" to "0". When identifying work card information corresponding to work information, the work card information matching unit 24 excludes work card information whose deletion flag is "1". Alternatively, the configuration may be such that the user can completely delete the work card correspondence information. Further, the user may add work card correspondence information or change work card information. Note that the work card information collation unit 24 may output error information if there is no work card information corresponding to the work information. The error information may be outputted, for example, by displaying it on a screen, outputting it audibly, or transmitting it to a system administrator's terminal.

In the embodiment described above, an image in which the work card information is superimposed is displayed on the image display device 1 each time a series of movements for each work indicated by the work information is completed, but the present invention is not limited to this. For example, if a worker's work is divided into a (large) task and (small) tasks that make up the (large) task, while multiple tasks (small) included in the (large) task are being performed, A configuration may be adopted in which one piece of work card information explaining (large) is displayed on the image display device 1. In this case, for example, the work card correspondence information is information in which one piece of work card information describing a (large) work is associated with work information indicating a plurality of works (small). Alternatively, a configuration may be adopted in which an image on which the work card information is superimposed is displayed on the image display device 1 during the work indicated by the work information. In this case, for example, the work correspondence information is information that associates work information with information indicating the movement in the first few seconds in which each work can be specified. The work information matching unit 22 refers to the work correspondence information, determines whether the movement of the worker indicated by the relative ratio difference information matches the movement of the worker during the first few seconds of the work indicated by the work information, and Work information in which the movement of the worker indicated by the ratio difference information matches the movement in the first few seconds of the work is identified.

In the above embodiment, the work information matching unit 22 of the server 2 identifies the work corresponding to the movement of the worker based on the time change in the relative ratio of the distance between the joint points of the worker. Not limited to. For example, in addition to temporal changes in the relative ratio of distances between joint points of the worker, the work information collation unit 22 can detect at least one of the velocity direction, acceleration, or angular velocity of the joint points in a coordinate system based on a particular joint point. The work corresponding to the movement of the worker may be specified based on this change over time. As a method for determining the coordinate system, for example, among the joint points of the worker in the image, the coordinate system is determined based on three joint points that are not on a straight line (for example, the center point of the head and two points on the left and right of the waist). There is a way. Alternatively, if the posture of the imaging unit 11 can be detected using a gyro sensor or the like, it is possible to determine the horizontal and vertical directions in the image, so one of the worker's joint points in the image can be set to the origin (for example, the waist). There is a method of determining the coordinate system based on the horizontal line in the direction in which the worker's body (for example, the waist) is facing. The coordinate system determined in this way is hereinafter referred to as the reference coordinate system.

In this case, the joint point extraction unit 12 of the image display device 1 calculates at least one of the velocity direction, acceleration, and angular velocity of the joint point in the reference coordinate system on the image at predetermined time intervals. The joint point extraction unit 12 extracts joint points that indicate the date and time when at least one of the velocity direction, acceleration, and angular velocity of the joint point was calculated, and the velocity direction, acceleration, and/or angular velocity of the joint point in the reference coordinate system on the image. Generate speed information. The inter-joint distance information transmitter 13 sequentially transmits the joint point velocity information generated by the joint point extractor 12 to the server 2 . When the inter-joint distance information receiving unit 21 of the server 2 receives the joint point velocity information from the image display device 1, it sends it to the work information matching unit 22. The work information collation unit 22 refers to the work correspondence information stored in the work correspondence information storage unit 23 and determines the worker indicated by the generated relative ratio difference information and the joint point velocity information received from the inter-joint distance information reception unit 21. Identify the work information of the work corresponding to the movement of.

In the above embodiment, the joint point extracting unit 12 extracts the joint points of the worker from the image photographed by the photographing unit 11, but the invention is not limited thereto. For example, if the target is a moving object such as a machine that does not have joints, feature points of the target may be extracted from an image photographed by the photographing unit 11.

In the embodiment described above, the image display device 1 transmits the inter-joint distance information indicating the date and time when the joint points were extracted and the distance between the joint points on the image to the server 2, but the present invention is not limited thereto. For example, the image display device 1 may transmit to the server 2 joint point coordinate information indicating the date and time when the joint point was extracted and the coordinates of the joint point on the image. In this case, the server 2 calculates the distance between the joint points on the image based on the joint point coordinate information received from the image display device 1, and generates inter-joint distance information. In addition, in the configuration that generates joint point velocity information as described above, the server 2 determines the velocity direction, acceleration, or angular velocity of the joint point in the reference coordinate system on the image based on the joint point coordinate information received from the image display device 1. The joint point velocity information may be generated by calculating at least one of the following.

In the embodiment described above, while the work card information receiving section 14 of the image display device 1 is not receiving the work card information, the display section 16 displays the image photographed by the photographing section 11. That is, if the work information matching unit 22 of the server 2 cannot identify the work information of the work corresponding to the movement of the worker, the work card information is not displayed on the display unit 16. For example, even when a worker is working, if the photographing direction of the photographing unit 11 is to photograph from directly behind the worker, the work information matching unit 22 identifies the work information of the work corresponding to the movement of the worker. It may not be possible. Therefore, if the work information matching unit 22 cannot identify the work information of the work corresponding to the movement of the worker, the work card information transmitting unit 26 sends message information to the image display device indicating a message prompting the user to change the shooting direction. It may be sent to 1. When the work card information receiving section 14 of the image display device 1 receives the message information, the image composition section 15 causes the display section 16 to display a composite image 4 in which the message information is superimposed on the image 41 received from the photographing section 11. Alternatively, the photographing direction in which the user photographs the worker in the work place W may be determined and displayed on the floor, wall, etc. of the work place W. Alternatively, the message indicated by the message information sent to the image display device 1 when the server 2 cannot identify the work information of the work corresponding to the movement of the worker prompts the user to take a picture of the worker in a predetermined shooting direction. It may also be something that encourages you to do so.

In the embodiment described above, the content of the work is identified from an image taken of a worker working in a work place, and an image in which information explaining the content of the work is superimposed on the image of the worker is displayed to the user. Although the provided examples have been described, they are not limited thereto. As other examples, the following second to sixth examples are given.

As a second example, in a gymnasium, sports gym, etc., the training content being demonstrated by the instructor is identified from images taken of the instructor's movements, and information that explains the meaning of the training movements that the instructor is unable to explain is overlaid. The combined image may be provided to the students. This eliminates, for example, differences in the ease of understanding explanations given by instructors.

As a third example, we can identify the flag signal that the standard bearer is making from images taken of the flag bearer's movements at ship reviews, launching ceremonies, etc., and display an image overlaid with information explaining the meaning of the flag signal to viewers. may be provided to By understanding the meaning of flag bearer, viewers will gain a deeper understanding of ship reviews, launching ceremonies, and other events.

As a fourth example, we can identify the sign language used by a hearing-impaired person from an image taken of the person's movements during a conversation with the hearing-impaired person, and create an image overlaid with information explaining the meaning of the sign language. You can also offer it to your conversation partner. This allows hearing-impaired people to communicate with people who do not know sign language.

As a fifth example, a supervisor can identify injuries, poor conditions, etc. of players from images taken of player movements during sports practices, matches, etc., and superimpose images with information explaining the identified injuries, poor conditions, etc. , coaches, staff, etc. This makes it possible for managers, coaches, staff, etc. to detect injuries or poor health of players at an early stage, even if they lack medical knowledge.

As a sixth example, visitors to zoos, aquariums, etc. can identify biologically meaningful movements from images taken of animal movements, and then overlay images with information that explains the meaning of the identified movements. may be provided to the person. This allows visitors to enjoy zoos, aquariums, etc. more by understanding the meaning of animal movements even without specialized knowledge.

In the above embodiment, the image display device 1 and the server 2 are provided separately, but the image display device 1 may also have the functions of the server 2. In this case, the same functions as the image display system 100 can be achieved using only the image display device 1, so even in an environment without a network, the movement of the worker can be displayed on top of the image of the movement of the worker. It is possible to provide the user with an image overlaid with information on the information.

1 image display device, 2 server, 3 network, 4 composite image, 11 imaging unit, 12 joint point extraction unit, 13 inter-joint distance information transmission unit, 14 work card information reception unit, 15 image synthesis unit, 16 display unit, 21 Inter-joint distance information receiving section, 22 Work information collation section, 23 Work correspondence information storage section, 24 Work card information verification section, 25 Work card correspondence information storage section, 26 Work card information transmission section, 41 Image, 42 Work card information, Reference Signs List 100 image display system, 101, 201 temporary storage unit, 102, 202 storage unit, 103, 203 calculation unit, 104, 204 input unit, 105, 205 transmission/reception unit, 106, 206 display unit, 107 sensor unit, W workshop.

Claims (7)

An image display system comprising an image display device that displays captured images, and a server connected to the image display device,
The image display device includes:
A photography department that takes images;
When a target is included in the image photographed by the photographing unit, feature points of the target are extracted from the image photographed by the photographing unit, and time-series feature point physical quantity information indicating physical quantities of the feature points is generated. a point extraction section;
a feature point physical quantity information transmitter that transmits the feature point physical quantity information to the server;
has
The server is
a feature point physical quantity information receiving unit that receives the feature point physical quantity information from the image display device;
a motion correspondence information storage unit that stores motion correspondence information in which information indicating a series of motions and motion information indicating the motion are associated;
a motion information matching unit that refers to the motion correspondence information and identifies the motion information corresponding to the motion of the target based on the feature point physical quantity information;
an operation card correspondence information storage unit that stores operation card correspondence information in which the operation information is associated with operation card information that describes the operation indicated by the operation information;
an operational card information collation unit that refers to the operational card correspondence information and identifies the operational card information that corresponds to the operational information specified by the operational information collation unit;
an operational card information transmitting unit that transmits the operational card information specified by the operational card information matching unit to the image display device;
has
The image display device includes:
an operational card information receiving unit that receives the operational card information from the server;
an image compositing unit that generates a composite image by superimposing the action card information on the image photographed by the photographing unit;
a display section that displays an image photographed by the photographing section and a composite image generated by the image composition section;
has
The feature point physical quantity information includes time-series information indicating distances between the feature points,
The motion information matching unit calculates a relative ratio of distances between the feature points and a previous time point of the relative ratio of distances between the feature points, based on time series information indicating distances between the feature points. An image display system that generates relative ratio difference information indicating a difference between the two and identifies the motion information corresponding to the movement of the object indicated by the relative ratio difference information . The feature point physical quantity information further includes time-series information indicating at least one of a velocity direction, acceleration, or angular velocity of the feature point in a coordinate system based on the specific feature point,
The motion information collation unit is configured to calculate time-series information indicating at least one of a velocity direction, acceleration, or angular velocity of the feature point in a coordinate system based on the specific feature point, and information about the target indicated by the relative ratio difference information. identifying the motion information corresponding to the motion;
The image display system according to claim 1 . An image display device that displays a photographed image,
A photography department that takes images;
When a target is included in the image photographed by the photographing unit, feature points of the target are extracted from the image photographed by the photographing unit, and time-series feature point physical quantity information indicating physical quantities of the feature points is generated. a point extraction section;
a motion correspondence information storage unit that stores motion correspondence information in which information indicating a series of motions and motion information indicating the motion are associated;
a motion information matching unit that refers to the motion correspondence information and identifies the motion information corresponding to the motion of the target based on the feature point physical quantity information;
an operation card correspondence information storage unit that stores operation card correspondence information in which the operation information is associated with operation card information that describes the operation indicated by the operation information;
an operational card information collation unit that refers to the operational card correspondence information and identifies the operational card information that corresponds to the operational information specified by the operational information collation unit;
an image combining unit that generates a composite image by superimposing the operational card information specified by the operational card information matching unit on the image photographed by the photographing unit;
a display section that displays an image photographed by the photographing section and a composite image generated by the image composition section;
Equipped with
The feature point physical quantity information includes time-series information indicating distances between the feature points,
The motion information matching unit calculates a relative ratio of distances between the feature points and a previous time point of the relative ratio of distances between the feature points, based on time series information indicating distances between the feature points. An image display device that generates relative ratio difference information indicating a difference between the two and identifies the motion information corresponding to the movement of the object indicated by the relative ratio difference information . A server connected to an image display device that displays captured images,
a feature point physical quantity information receiving unit that receives time-series feature point physical quantity information indicating a physical quantity of a target feature point extracted from an image captured by the image display device;
a motion correspondence information storage unit that stores motion correspondence information in which information indicating a series of motions and motion information indicating the motion are associated;
a motion information matching unit that refers to the motion correspondence information and identifies the motion information corresponding to the motion of the target based on the feature point physical quantity information;
an operation card correspondence information storage unit that stores operation card correspondence information in which the operation information is associated with operation card information that describes the operation indicated by the operation information;
an operational card information collation unit that refers to the operational card correspondence information and identifies the operational card information that corresponds to the operational information specified by the operational information collation unit;
an operational card information transmitting unit that transmits the operational card information specified by the operational card information matching unit to the image display device;
Equipped with
The feature point physical quantity information includes time-series information indicating distances between the feature points,
The motion information matching unit calculates a relative ratio of distances between the feature points and a previous time point of the relative ratio of distances between the feature points, based on time series information indicating distances between the feature points. A server that generates relative ratio difference information indicating a difference between the two and identifies the motion information corresponding to the movement of the target indicated by the relative ratio difference information . The image display device executes
a photographing step for photographing an image;
When a target is included in the image photographed in the photographing step, feature points of the target are extracted from the image photographed in the photographing step, and time-series feature point physical quantity information indicating physical quantities of the feature points is generated. a point extraction step;
a feature point physical quantity information transmission step of transmitting the feature point physical quantity information to a server;
The server executes
a feature point physical quantity information receiving step of receiving the feature point physical quantity information from the image display device;
motion information that identifies the motion information corresponding to the movement of the target based on the feature point physical quantity information by referring to motion correspondence information that associates information indicating a series of motions with motion information indicating the motion; a matching step;
The operation card information corresponding to the operation information identified in the operation information matching step is identified by referring to the operation card correspondence information in which the operation information is associated with the operation card information explaining the operation indicated by the operation information. an operation card information verification step;
an operation card information transmitting step of transmitting the operation card information specified in the operation card information matching step to the image display device;
The image display device executes,
an operation card information receiving step of receiving the operation card information from the server;
an image compositing step of generating a composite image by superimposing the action card information on the image photographed in the photographing step;
a display step of displaying the composite image generated in the image composition step;
Equipped with
The feature point physical quantity information includes time-series information indicating distances between the feature points,
In the motion information matching step, the relative ratio of the distance between the feature points and the previous time point and the relative ratio of the distance between the feature points are determined based on time series information indicating the distance between the feature points. An image display method that generates relative ratio difference information indicating a difference between the target and the target, and identifies the motion information corresponding to the movement of the object indicated by the relative ratio difference information . The computer that displays the captured images,
Photography department that takes images;
When a target is included in the image photographed by the photographing unit, feature points of the target are extracted from the image photographed by the photographing unit, and time-series feature point physical quantity information indicating physical quantities of the feature points is generated. point extraction part,
motion information that identifies the motion information corresponding to the movement of the target based on the feature point physical quantity information by referring to motion correspondence information that associates information indicating a series of motions with motion information indicating the motion; collation section,
The operation card information corresponding to the operation information identified by the operation information collation unit is identified by referring to operation card correspondence information that associates the operation information with operation card information that describes the operation indicated by the operation information. Operation card information verification unit,
an image combining unit that generates a composite image by superimposing the operational card information specified by the operational card information matching unit on the image photographed by the photographing unit, and
a display section that displays an image photographed by the photographing section and a composite image generated by the image composition section;
function as
The feature point physical quantity information includes time-series information indicating distances between the feature points,
The motion information matching unit calculates a relative ratio of distances between the feature points and a previous time point of the relative ratio of distances between the feature points, based on time series information indicating distances between the feature points. A program that generates relative ratio difference information indicating a difference between the two and identifies the motion information corresponding to the movement of the target indicated by the relative ratio difference information . A computer connected to an image display device that displays captured images.
a feature point physical quantity information receiving unit that receives time-series feature point physical quantity information indicating a physical quantity of a target feature point extracted from an image captured by the image display device;
motion information that identifies the motion information corresponding to the movement of the target based on the feature point physical quantity information by referring to motion correspondence information that associates information indicating a series of motions with motion information indicating the motion; collation section,
The operation card information corresponding to the operation information identified by the operation information collation unit is identified by referring to operation card correspondence information that associates the operation information with operation card information that describes the operation indicated by the operation information. an operation card information verification unit, and
an operational card information transmitter that transmits the operational card information specified by the operational card information collation unit to the image display device;
function as
The feature point physical quantity information includes time-series information indicating distances between the feature points,
The motion information matching unit calculates a relative ratio of distances between the feature points and a previous time point of the relative ratio of distances between the feature points, based on time series information indicating distances between the feature points. A program that generates relative ratio difference information indicating a difference between the two and identifies the motion information corresponding to the movement of the target indicated by the relative ratio difference information .

Images