JP7485454B2 - Sign language translation processing device, sign language translation processing system, sign language translation processing method, program, and recording medium - Google Patents

Description

The present invention relates to a sign language translation processing device, a sign language translation processing system, a sign language translation processing method, a program, and a recording medium.

For example, Patent Document 1 describes a method for deaf and hearing people to communicate by converting sign language information into voice or text.

JP 2005-197889 A

The invention described in Patent Document 1 converts sign language information into speech or text by referring to a sign language pattern library. When referring to a sign language pattern library, it is necessary to obtain sign language patterns that assume a wide variety of patterns, such as gender, height, and clothing color. In this case, for example, if sign language information that differs from the patterns stored in the library is input, there is a problem that the translation accuracy decreases.

The present invention aims to provide a sign language translation processing device, a sign language translation processing system, a sign language translation processing method, a program, and a recording medium for performing sign language translation with high accuracy.

In order to achieve the above object, the sign language translation processing device of the present invention comprises:
The present invention includes an image acquisition unit, a hand skeletal information acquisition unit, a body skeletal information acquisition unit, an information integration unit, and a motion recognition unit,
The image acquisition unit acquires an image of a body including a hand,
The images are a plurality of time-series images captured over time,
the hand skeletal information acquisition unit acquires hand skeletal information from the acquired image;
The hand skeletal information includes hand skeletal coordinates;
the body skeletal information acquisition unit acquires body skeletal information from the acquired image;
The body skeletal information includes body skeletal coordinates;
the information integration unit integrates the hand skeletal information and the body skeletal information to calculate a position of the hand skeletal position on the body and a position of the hand;
The movement recognition unit organizes the integrated and calculated hand skeletal position and hand position, as well as the body skeletal information, in chronological order, and recognizes hand movement information from the organized hand skeletal position and hand position, as well as the body skeletal information, to infer a sign language word.

The sign language translation processing system of the present invention comprises:
A sign language translation processing device and a user terminal,
The sign language translation processing device is the sign language translation processing device of the present invention,
The user terminal is capable of acquiring an image of a body including a hand;
The images are a plurality of time-series images captured over time.

The sign language translation processing method of the present invention comprises the steps of:
The method includes an image acquisition step, a hand skeleton information acquisition step, a body skeleton information acquisition step, an information integration step, and a motion recognition step,
The image capturing step captures an image of a body including a hand;
The images are a plurality of time-series images captured over time,
The hand skeletal information acquiring step acquires hand skeletal information from the acquired image,
The hand skeletal information includes hand skeletal coordinates;
The body skeleton information acquiring step acquires body skeleton information from the acquired image,
The body skeletal information includes body skeletal coordinates;
the information integration step includes integrating the hand skeletal information and the body skeletal information to calculate a position of the hand skeletal position on the body and a position of the hand;
The movement recognition process organizes the integrated and calculated hand skeletal position and hand position, as well as the body skeletal information, in chronological order, and recognizes hand movement information from the organized hand skeletal position and hand position, as well as the body skeletal information, to infer a sign language word.

The program of the present invention comprises:
The method includes an image acquisition step, a hand skeleton information acquisition step, a body skeleton information acquisition step, an information integration step, and a motion recognition step.
The image acquisition step includes acquiring an image of a body including a hand;
The images are a plurality of time-series images captured over time,
The hand skeletal information acquisition step acquires hand skeletal information from the acquired image,
The hand skeletal information includes hand skeletal coordinates;
The body skeleton information acquisition step acquires body skeleton information from the acquired image,
The body skeletal information includes body skeletal coordinates;
the information integration step includes integrating the hand skeletal information and the body skeletal information to calculate a position of the hand skeletal position on the body and a position of the hand;
the motion recognition step includes chronologically organizing the integrated and calculated skeletal position of the hand and the position of the hand, and the skeletal information of the body, and recognizing hand motion information from the organized skeletal position of the hand and the position of the hand, and the skeletal information of the body, to estimate a sign language word;
The program is for causing a computer to execute each of the above procedures.

The recording medium of the present invention is a computer-readable recording medium on which the program of the present invention is recorded.

The present invention enables highly accurate sign language translation.

FIG. 1 is a block diagram showing an example of a configuration of a sign language translation processing apparatus according to the first embodiment. FIG. 2 is a block diagram showing an example of a hardware configuration of the sign language translation processing device according to the first embodiment. FIG. 3 is a flowchart showing an example of processing in the sign language translation processing apparatus according to the first embodiment. FIG. 4 is a block diagram showing an example of the configuration of a sign language translation processing apparatus according to the second embodiment. FIG. 5 is a block diagram showing an example of a hardware configuration of a sign language translation processing apparatus according to the second embodiment. FIG. 6 is a flowchart showing an example of processing in the sign language translation processing apparatus of the second embodiment. FIG. 7 is an explanatory diagram illustrating an example of a sign language translation processing system according to the fourth embodiment.

Next, an embodiment of the present invention will be described with reference to the drawings. The present invention is not limited to the following embodiment. In each of the drawings, the same parts are given the same reference numerals. Furthermore, the explanations of each embodiment can be mutually incorporated unless otherwise specified, and the configurations of each embodiment can be combined unless otherwise specified.

[Embodiment 1]
1 is a block diagram showing an example of the configuration of a sign language translation processing device 10 according to the present embodiment. As shown in FIG. 1, the device 10 includes an image acquisition unit 11, a hand skeletal information acquisition unit 12, a body skeletal information acquisition unit 13, an information integration unit 14, and a motion recognition unit 15.

The device 10 may be, for example, a single device including each of the above-mentioned parts, or a device to which each of the above-mentioned parts can be connected via a communication line network. The device 10 may also be connected to an external device described later via the communication line network. The communication line network is not particularly limited, and any known network can be used, and may be, for example, wired or wireless. Examples of the communication line network include the Internet line, WWW (World Wide Web), telephone line, LAN (Local Area Network), SAN (Storage Area Network), DTN (Delay Tolerant Networking), LPWA (Low Power Wide Area), L5G (Local 5G), etc. Examples of wireless communication include Wi-Fi (registered trademark), Bluetooth (registered trademark), local 5G, LPWA, etc. The wireless communication may be a form in which each device communicates directly (Ad Hoc communication), infrastructure communication, indirect communication via an access point, etc. The device 10 may be incorporated into a server as a system, for example. The device 10 may be a personal computer (PC, for example, desktop type or notebook type) in which the program of the present invention is installed, a smartphone, a tablet terminal, etc. The device 10 may be in a form such as cloud computing or edge computing in which at least one of the parts is on a server and the other parts are on a terminal.

Figure 2 shows an example block diagram of the hardware configuration of the device 10. The device 10 includes, for example, a central processing unit (CPU, GPU, etc.) 101, a memory 102, a bus 103, a storage device 104, an input device 105, an output device 106, a communication device 107, etc. Each part of the device 10 is connected to each other via the bus 103 by its respective interface (I/F).

The central processing unit 101 cooperates with other components through a controller (system controller, I/O controller, etc.) and is responsible for overall control of the device 10. In the device 10, the central processing unit 101 executes, for example, the program of the present invention and other programs, and also reads and writes various information. Specifically, for example, the central processing unit 101 functions as an image acquisition unit 11, a hand skeletal information acquisition unit 12, a body skeletal information acquisition unit 13, an information integration unit 14, and a motion recognition unit 15. The central processing unit 101 may include, as a computing device, a CPU, a GPU (Graphics Processing Unit), an APU (Accelerated Processing Unit), or a combination of these.

The bus 103 can also be connected to, for example, an external device. Examples of the external device include an external storage device (external database, etc.), a printer, an external input device, an external display device, an external imaging device, etc. The present device 10 can be connected to an external network (the above-mentioned communication line network) by, for example, a communication device 107 connected to the bus 103, and can also be connected to other devices via the external network.

The memory 102 may be, for example, a main memory (primary storage device). When the central processing unit 101 performs processing, the memory 102 reads various operating programs, such as the program of the present invention, stored in the storage device 104 described below, and the central processing unit 101 receives data from the memory 102 and executes the programs. The main memory may be, for example, a RAM (random access memory). The memory 102 may also be, for example, a ROM (read only memory).

The storage device 104 is also referred to as an auxiliary storage device, for example, in contrast to the main memory. As described above, the storage device 104 stores operating programs including the program of the present invention. The storage device 104 may be, for example, a combination of a recording medium and a drive that reads and writes from the recording medium. The recording medium is not particularly limited, and may be, for example, an internal or external type, such as a HD (hard disk), CD-ROM, CD-R, CD-RW, MO, DVD, flash memory, memory card, etc. The storage device 104 may be, for example, a hard disk drive (HDD) in which the recording medium and the drive are integrated, or a solid state drive (SSD).

In the present device 10, the memory 102 and the storage device 104 can also store various information such as log information, information acquired from an external database (not shown) or an external device, information generated by the present device 10, and information used by the present device 10 when executing processing. In this case, the memory 102 and the storage device 104 may store hand movement information and sign language conversation in association with each other, for example. Note that at least a portion of the information may be stored, for example, in an external server other than the memory 102 and the storage device 104, or may be stored in a distributed manner across multiple terminals using blockchain technology or the like.

The device 10 further includes, for example, an input device 105 and an output device 106. Examples of the input device 105 include pointing devices such as a touch panel, a track pad, and a mouse; a keyboard; an imaging means such as a camera and a scanner; a card reader such as an IC card reader and a magnetic card reader; and an audio input means such as a microphone. Examples of the output device 106 include a display device such as an LED (light-emitting diode) display and a liquid crystal display; an audio output device such as a speaker; a printer; and the like. In the first embodiment, the input device 105 and the output device 106 are configured separately, but the input device 105 and the output device 106 may be configured as an integrated device, such as a touch panel display.

Next, an example of the sign language translation processing method of this embodiment will be described based on the flowchart S10 of FIG. 3. The sign language translation processing method of this embodiment is implemented as follows, for example, using the device 10 of FIG. 1 or FIG. 2. Note that the sign language translation processing method of this embodiment is not limited to the use of the device 10 of FIG. 1 or FIG. 2.

First, the image acquisition unit 11 acquires images of the body including the hands (S11). Here, the images are a plurality of time-series images captured over time. The body may be, for example, the entire body including the upper and lower body, or only the upper body. Here, the images may be acquired, for example, by a camera provided in the device 10, or images acquired by a camera other than the device 10 may be acquired via the communication device 107. The images are acquired, for example, frame by frame. In the present invention, the term "hands" may refer to both hands or one hand, unless otherwise specified.

Next, the hand skeletal information acquisition unit 12 acquires hand skeletal information from the acquired image (S12). The hand skeletal information includes the skeletal coordinates of the hand. The hand skeletal information can be detected and acquired, for example, by a conventionally known method. The hand skeleton includes, for example, the joints of the hand. The hand skeletal coordinates may be acquired, for example, by using a hand skeleton detection model.

The body skeletal information acquisition unit 13 acquires body skeletal information from the acquired image (S13). The body skeletal information includes the body skeletal coordinates. The body skeletal information can be detected and acquired, for example, by a conventionally known method. The body skeletal coordinates can be acquired, for example, by using a body skeletal detection model. Although not shown in FIG. 3, the body skeletal coordinates acquired in S13 can be saved, for example. If the body skeletal coordinates acquired in S13 are saved, for example, when acquiring body skeletal information in the next frame, the saved body skeletal coordinates and the body image of the next frame can be integrated to acquire the body skeletal coordinates of the next frame. In this way, the accuracy of acquiring the body skeletal coordinates from the next frame onwards is improved.

In FIG. 3, after acquiring an image of the body including the hand (S11), skeletal information of the hand (S12) and skeletal information of the body (S13) are acquired simultaneously in parallel. However, this is merely an example. For example, skeletal information of the hand may be acquired first, and then skeletal information of the body may be acquired, or skeletal information of the hand may be acquired first, and then skeletal information of the body may be acquired.

Next, the information integration unit 14 integrates the hand skeletal information and the body skeletal information (S14). Based on the integrated information, the position of the hand skeleton on the body and the position of the hand are calculated (S15).

Next, the action recognition unit 15 organizes the integrated and calculated hand skeletal position and hand position, and the body skeletal information in chronological order, and recognizes hand action information from the organized hand skeletal position and hand position, and the body skeletal information to estimate a sign language word (S16). The sign language is not particularly limited, but examples include Japanese Sign Language, Japanese-compatible Sign Language, and intermediate sign language. It may also be a sign language used in countries around the world. The sign language word can be estimated, for example, by a conventionally known method. The hand action information may be recognized, for example, by using an action recognition model.

Furthermore, if the device 10 includes, for example, an output unit (not shown), the output unit may output the acquired spoken conversation to a user terminal device as text or audio. If the device 10 includes the output unit, for example, the central processing unit 101 may function as the output unit. The user terminal device may be, for example, a personal computer (PC, for example, desktop type or notebook type), a smartphone, a tablet terminal, etc. If the output is text, it may be output by a display device such as an LED display or liquid crystal display of the output device 106. If the output is audio, it may be output by an audio output device such as a speaker, for example.

According to this embodiment, as described above, the sign language is estimated based on the skeletal coordinates of the hand and the skeletal coordinates of the body. Therefore, unlike conventional sign language translation techniques that use a sign language pattern library, for example, highly accurate sign language translation is possible regardless of differences in a person's gender, height, clothing color, etc. Another advantage is that there is no need to prepare a huge sign language pattern library that takes into account differences in a person's gender, height, clothing color, etc.

[Embodiment 2]
The present embodiment is another example of the sign language translation processing device and the sign language translation processing method of the present invention. As shown in Fig. 4, the device 10 may further include, for example, a correction unit 16. Also, as shown in Fig. 5, for example, the central processing unit 101 may function as the correction unit 16.

Next, an example of the sign language translation processing method of this embodiment will be described based on the flowchart S20 of FIG. 6. The sign language translation processing method of this embodiment is implemented as follows, for example, using the device 10 of FIG. 4 or FIG. 5. Note that the sign language translation processing method of this embodiment is not limited to the use of the device 10 of FIG. 4 or FIG. 5.

First, as in the flow (S11) of the first embodiment, an image of the body including the hand is acquired (S21). Next, hand skeletal information is acquired. Here, the hand skeletal information further includes hand detection area coordinates, and the hand detection area coordinates are acquired (S220). Thereafter, the hand skeletal coordinates are acquired (S221). Next, the correction unit 16 corrects the hand detection area coordinates from the hand skeletal coordinates and stores them (S222). The stored corrected coordinates can be used, for example, when acquiring the hand skeletal coordinates of the next frame. The hand skeletal information can be acquired, for example, by a conventionally known method, as described above. The hand detection area coordinates may be acquired, for example, by using a hand detection model.

The rest of the flow is the same as steps S13 to S16 in embodiment 1 (S22 to S26).

As in this embodiment, by acquiring the hand detection area coordinates and then the hand skeletal coordinates, hand skeletal information can be acquired more quickly than, for example, acquiring the hand skeletal coordinates without acquiring the hand detection area coordinates. Also, as described above, correcting and saving the hand detection area coordinates improves the accuracy of detecting the hand skeletal coordinates in the next frame, for example.

[Embodiment 3]
In the first and second embodiments, when the device 10 further includes a storage unit, for example, the storage device 104 functions as the storage unit. The storage unit can store, for example, hand movement information and sign language words in association with each other. In this case, the movement recognition unit can estimate the sign language word from the hand movement information and the associated and stored hand movement information and sign language word.

[Embodiment 4]
Next, an example of a sign language translation processing system including the device 10 according to any one of the first to third embodiments and a user terminal is shown in Fig. 7. The user terminal is capable of acquiring an image of the body including the hands.

As shown in FIG. 7, a deaf person (sign language speaker) inputs sign language into the device 10. In FIG. 7, the device 10 may be, for example, a device such as a smartphone or a tablet terminal, and the user may sign into a camera of the device and input the sign language into the device 10. The device 10 performs sign language translation by, for example, performing any of the processes of embodiments 1 to 3, and outputs the translation result to a user terminal. Any of the processes of embodiments 1 to 3 may be performed by the device itself that includes the device 10, or may be performed by a server that includes each part of the device 10. The output translation result may be displayed as characters on a display screen of the user terminal, or may be output as audio by a speaker. A hearing person (non-sign language speaker) can check the sign language translation result output to the user terminal.

[Embodiment 5]
The program of the present embodiment is a program for causing a computer to execute each of the above-mentioned steps. Specifically, the program of the present embodiment is a program for causing a computer to execute an image acquisition procedure, a hand skeletal information acquisition procedure, a body skeletal information acquisition procedure, an information integration procedure, and a motion recognition procedure.

The image acquisition step includes acquiring an image of a body including a hand;
The images are a plurality of time-series images captured over time,
The hand skeletal information acquisition step acquires hand skeletal information from the acquired image,
The hand skeletal information includes hand skeletal coordinates;
The body skeleton information acquisition step acquires body skeleton information from the acquired image,
The body skeletal information includes body skeletal coordinates;
the information integration step includes integrating the hand skeletal information and the body skeletal information to calculate a position of the hand skeletal position on the body and a position of the hand;
The movement recognition procedure organizes the integrated and calculated hand skeletal position and hand position, as well as the body skeletal information, in chronological order, and recognizes hand movement information from the organized hand skeletal position and hand position, as well as the body skeletal information, to infer a sign language word.

The program of this embodiment can also be said to cause a computer to function as an image acquisition procedure, a hand skeletal information acquisition procedure, a body skeletal information acquisition procedure, an information integration procedure, and a motion recognition procedure.

The program of this embodiment can use the description of the sign language translation processing device and the sign language translation processing method of the present invention. For example, the "procedure" in each of the steps can be read as "processing". The program of this embodiment can be recorded in a computer-readable recording medium. The recording medium is, for example, a non-transitory computer-readable storage medium. The recording medium is not particularly limited, and examples include random access memory (RAM), read-only memory (ROM), hard disk (HD), optical disk, and floppy (registered trademark) disk (FD).

The present invention has been described above with reference to the embodiments, but the present invention is not limited to the above embodiments. Various modifications that can be understood by a person skilled in the art can be made to the configuration and details of the present invention within the scope of the present invention.

<Additional Notes>
Some or all of the above embodiments may be described as follows, but are not limited to the following:
(Appendix 1)
The present invention includes an image acquisition unit, a hand skeleton information acquisition unit, a body skeleton information acquisition unit, an information integration unit, and a motion recognition unit,
The image acquisition unit acquires an image of a body including a hand,
the images are a plurality of time-series images captured over time,
the hand skeletal information acquisition unit acquires hand skeletal information from the acquired image;
The hand skeletal information includes hand skeletal coordinates;
the body skeletal information acquisition unit acquires body skeletal information from the acquired image;
The body skeletal information includes body skeletal coordinates;
the information integration unit integrates the hand skeletal information and the body skeletal information to calculate a position of the hand skeletal position on the body and a position of the hand;
the movement recognition unit organizes the integrated and calculated hand skeletal position and hand position, and the body skeletal information in chronological order, and recognizes hand movement information from the organized hand skeletal position and hand position, and the body skeletal information, to estimate a sign language word.
Sign language translation processing device.
(Appendix 2)
Further, an output unit is included,
2. The sign language translation processing device according to claim 1, wherein the output unit outputs the sign language words to a user terminal device by characters or voice.
(Appendix 3)
the hand skeleton coordinates are detected using a hand skeleton detection model;
The body skeletal coordinates are detected using a body skeletal detection model;
The hand movement information is recognized using a movement recognition model.
3. A sign language translation processing device according to claim 1 or 2.
(Appendix 4)
Further, a correction unit is included,
The hand skeleton information further includes hand detection area coordinates,
The correction unit corrects the detection area coordinates of the hand from the skeletal coordinates of the hand.
4. A sign language translation processing device according to any one of claims 1 to 3.
(Appendix 5)
The hand detection region coordinates are detected using a hand detection model.
5. A sign language translation processing device as claimed in claim 4.
(Appendix 6)
Further, a storage unit is included which stores hand movement information and sign language words in association with each other,
The action recognition unit estimates a sign language word from the hand action information and the associated and stored hand action information and sign language word.
6. A sign language translation processing device according to any one of appendix 1 to 5.
(Appendix 7)
A sign language translation processing device and a user terminal,
The sign language translation processing device is the sign language translation processing device according to any one of Supplementary Note 1 to 6,
The user terminal is capable of acquiring an image of a body including a hand;
The images are a plurality of time-series images captured over time.
Sign language translation processing system.
(Appendix 8)
The method includes an image acquisition step, a hand skeleton information acquisition step, a body skeleton information acquisition step, an information integration step, and a motion recognition step,
The image capturing step captures an image of a body including a hand;
the images are a plurality of time-series images captured over time,
The hand skeletal information acquiring step acquires hand skeletal information from the acquired image,
The hand skeletal information includes hand skeletal coordinates;
The body skeleton information acquiring step acquires body skeleton information from the acquired image,
The body skeletal information includes body skeletal coordinates;
the information integration step includes integrating the hand skeletal information and the body skeletal information to calculate a position of the hand skeletal position on the body and a position of the hand;
the action recognition step includes chronologically arranging the integrated and calculated hand skeletal position and hand position, and the body skeletal information, and recognizing hand action information from the organized hand skeletal position and hand position, and the body skeletal information, to estimate a sign language word;
A sign language translation processing method.
(Appendix 9)
Further, an output step is included,
9. The sign language translation processing method according to claim 8, wherein the output step outputs the sign language words to a user terminal device by characters or voice.
(Appendix 10)
the hand skeleton coordinates are detected using a hand skeleton detection model;
The body skeletal coordinates are detected using a body skeletal detection model;
The hand movement information is recognized using a movement recognition model.
10. A sign language translation processing method according to claim 8 or 9.
(Appendix 11)
Further, a correction step is included,
The hand skeleton information further includes hand detection area coordinates,
The correction step includes correcting coordinates of the detection area of the hand based on skeletal coordinates of the hand.
A sign language translation processing method according to any one of appendices 8 to 10.
(Appendix 12)
The hand detection region coordinates are detected using a hand detection model.
12. A sign language translation processing method as described in appendix 11.
(Appendix 13)
A sign language translation processing method according to any one of appendices 8 to 12, wherein the action recognition step estimates a sign language word from the hand action information and the hand action information and sign language word stored in association with each other.
(Appendix 14)
The method includes an image acquisition step, a hand skeleton information acquisition step, a body skeleton information acquisition step, an information integration step, and a motion recognition step.
The image acquisition step includes acquiring an image of a body including a hand;
The images are a plurality of time-series images captured over time,
The hand skeletal information acquisition step acquires hand skeletal information from the acquired image,
The hand skeletal information includes hand skeletal coordinates;
The body skeleton information acquisition step acquires body skeleton information from the acquired image,
The body skeletal information includes body skeletal coordinates;
the information integration step includes integrating the hand skeletal information and the body skeletal information to calculate a position of the hand skeletal position on the body and a position of the hand;
the motion recognition step includes chronologically organizing the integrated and calculated hand skeletal position and hand position, and the body skeletal information, and recognizing hand motion information from the organized hand skeletal position and hand position, and the body skeletal information, to estimate a sign language word;
A program for causing a computer to execute each of the above procedures.
(Appendix 15)
Further, an output procedure is included,
The program according to claim 14, wherein the output step outputs the sign language word to a user terminal device by text or voice.
(Appendix 16)
the hand skeleton coordinates are detected using a hand skeleton detection model;
The body skeletal coordinates are detected using a body skeletal detection model;
The hand movement information is recognized using a movement recognition model.
16. The program according to claim 14 or 15.
(Appendix 17)
Further, a correction procedure is included,
The hand skeleton information further includes hand detection area coordinates,
the correction step corrects the detection area coordinates of the hand from the skeleton coordinates of the hand;
17. The program according to any one of appendixes 14 to 16.
(Appendix 18)
The hand detection region coordinates are detected using a hand detection model.
18. The program according to claim 17.
(Appendix 19)
19. The program according to any one of appendices 14 to 18, wherein the action recognition step estimates a sign language word from the hand action information and the hand action information and the sign language word stored in association with each other.
(Appendix 20)
The method includes an image acquisition step, a hand skeleton information acquisition step, a body skeleton information acquisition step, an information integration step, and a motion recognition step.
The image acquisition step includes acquiring an image of a body including a hand;
the images are a plurality of time-series images captured over time,
The hand skeletal information acquisition step acquires hand skeletal information from the acquired image,
The hand skeletal information includes hand skeletal coordinates;
The body skeleton information acquisition step acquires body skeleton information from the acquired image,
The body skeletal information includes body skeletal coordinates;
the information integration step includes integrating the hand skeletal information and the body skeletal information to calculate a position of the hand skeletal position on the body and a position of the hand;
the motion recognition step includes chronologically organizing the integrated and calculated skeletal position of the hand and the position of the hand, and the skeletal information of the body, and recognizing hand motion information from the organized skeletal position of the hand and the position of the hand, and the skeletal information of the body, to estimate a sign language word;
A computer-readable recording medium having recorded thereon a program for causing a computer to execute each of the above procedures.
(Appendix 21)
Further, an output procedure is included,
21. The recording medium according to claim 20, wherein the output step outputs the sign language word to a user terminal device by text or voice.
(Appendix 22)
the hand skeleton coordinates are detected using a hand skeleton detection model;
The body skeletal coordinates are detected using a body skeletal detection model;
The hand movement information is recognized using a movement recognition model.
22. The recording medium according to claim 20 or 21.
(Appendix 23)
Further, a correction procedure is included,
The hand skeleton information further includes hand detection area coordinates,
the correction step corrects coordinates of the detection area of the hand from skeletal coordinates of the hand;
23. A recording medium according to any one of appendices 20 to 22.
(Appendix 24)
The hand detection region coordinates are detected using a hand detection model.
24. The recording medium according to claim 23.
(Appendix 25)
25. The recording medium according to any one of appendices 20 to 24, wherein the action recognition step estimates a sign language word from the hand action information and the hand action information and the sign language word stored in association with each other.

The present invention enables highly accurate sign language translation. For example, the present invention can be applied to a sign language translation processing device aimed at smooth communication between deaf and hearing people, but the fields of application are not limited, and the present invention can be applied to a wide range of fields using sign language translation processing devices.

10 Sign language translation processing device 11 Image acquisition unit 12 Hand skeleton information acquisition unit 13 Body skeleton information acquisition unit 14 Information integration unit 15 Action recognition unit 16 Correction unit 101 CPU
102 memory 103 bus 104 storage device 105 input device 106 output device 107 communication device

Claims (9)

The present invention includes an image acquisition unit, a hand skeletal information acquisition unit, a body skeletal information acquisition unit, an information integration unit, a motion recognition unit, and a correction unit,
The image acquisition unit acquires an image of a body including a hand in a sign language of a sign language speaker ,
The sign language is a sign language for communication between the sign language user and a non-sign language user,
the images are a plurality of time-series images captured over time,
the hand skeletal information acquisition unit acquires hand skeletal information from the acquired image;
The hand skeleton information includes hand skeleton coordinates and hand detection area coordinates;
the body skeletal information acquisition unit acquires body skeletal information from the acquired image;
The body skeletal information includes body skeletal coordinates;
the information integration unit integrates the hand skeletal information and the body skeletal information to calculate a position of the hand skeletal position on the body and a position of the hand;
the movement recognition unit organizes the integrated and calculated hand skeletal position and hand position, and the body skeletal information in chronological order, recognizes hand movement information from the organized hand skeletal position and hand position, and the body skeletal information, and estimates a sign language word;
The correction unit corrects the detection area coordinates of the hand from the skeletal coordinates of the hand.
Sign language translation processing device. Further, an output unit is included,
The sign language translation processing device according to claim 1 , wherein the output unit outputs the sign language words to a user terminal device by characters or voice. the hand skeleton coordinates are detected using a hand skeleton detection model;
The body skeletal coordinates are detected using a body skeletal detection model;
The hand movement information is recognized using a movement recognition model.
3. The sign language translation processing device according to claim 1 or 2. The hand detection region coordinates are detected using a hand detection model.
3. The sign language translation processing device according to claim 1 or 2. Further, a storage unit is included which stores hand movement information and sign language words in association with each other,
The action recognition unit estimates a sign language word from the hand action information and the associated and stored hand action information and sign language word.
3. The sign language translation processing device according to claim 1 or 2. A sign language translation processing device and a user terminal,
The sign language translation processing device is a sign language translation processing device according to claim 1 or 2,
The user terminal is capable of acquiring an image of a body including a hand;
The images are a plurality of time-series images captured over time.
Sign language translation processing system. The method includes an image acquisition step, a hand skeleton information acquisition step, a body skeleton information acquisition step, an information integration step, a motion recognition step, and a correction step,
The image acquisition step acquires an image of a body including a hand of a sign language user ,
The sign language is a sign language for communication between the sign language user and a non-sign language user,
The images are a plurality of time-series images captured over time,
The hand skeletal information acquiring step acquires hand skeletal information from the acquired image,
The hand skeleton information includes hand skeleton coordinates and hand detection area coordinates;
The body skeleton information acquiring step acquires body skeleton information from the acquired image,
The body skeletal information includes body skeletal coordinates;
the information integration step includes integrating the hand skeletal information and the body skeletal information to calculate a position of the hand skeletal position on the body and a position of the hand;
the action recognition step includes chronologically arranging the integrated and calculated hand skeletal position and hand position, and the body skeletal information, and recognizing hand action information from the organized hand skeletal position and hand position, and the body skeletal information, to estimate a sign language word;
The correction step includes correcting coordinates of the detection area of the hand based on skeletal coordinates of the hand.
A sign language translation processing method. The method includes an image acquisition step, a hand skeleton information acquisition step, a body skeleton information acquisition step, an information integration step, a motion recognition step, and a correction step.
The image acquisition step includes acquiring an image of a body including a hand of a sign language user ;
The sign language is a sign language for communication between the sign language user and a non-sign language user,
The images are a plurality of time-series images captured over time,
The hand skeletal information acquisition step acquires hand skeletal information from the acquired image,
The hand skeleton information includes hand skeleton coordinates and hand detection area coordinates;
The body skeleton information acquisition step acquires body skeleton information from the acquired image,
The body skeletal information includes body skeletal coordinates;
the information integration step includes integrating the hand skeletal information and the body skeletal information to calculate a position of the hand skeletal position on the body and a position of the hand;
the motion recognition step includes chronologically organizing the integrated and calculated skeletal position of the hand and the position of the hand, and the skeletal information of the body, and recognizing hand motion information from the organized skeletal position of the hand and the position of the hand, and the skeletal information of the body, to estimate a sign language word;
the correction step corrects coordinates of the detection area of the hand from skeletal coordinates of the hand;
A program for causing a computer to execute each of the above procedures. The method includes an image acquisition step, a hand skeleton information acquisition step, a body skeleton information acquisition step, an information integration step, a motion recognition step, and a correction step.
The image acquisition step includes acquiring an image of a body including a hand of a sign language user ;
The sign language is a sign language for communication between the sign language user and a non-sign language user,
the images are a plurality of time-series images captured over time,
The hand skeletal information acquisition step acquires hand skeletal information from the acquired image,
The hand skeleton information includes hand skeleton coordinates and hand detection area coordinates;
The body skeleton information acquisition step acquires body skeleton information from the acquired image,
The body skeletal information includes body skeletal coordinates;
the information integration step includes integrating the hand skeletal information and the body skeletal information to calculate a position of the hand skeletal position on the body and a position of the hand;
the motion recognition step includes chronologically organizing the integrated and calculated skeletal position of the hand and the position of the hand, and the skeletal information of the body, and recognizing hand motion information from the organized skeletal position of the hand and the position of the hand, and the skeletal information of the body, to estimate a sign language word;
the correction step corrects the detection area coordinates of the hand from the skeleton coordinates of the hand;
A computer-readable recording medium having recorded thereon a program for causing a computer to execute each of the above procedures.

Images