JP7430088B2 - speech control device - Google Patents

Description

The present invention relates to a speech control device that can be applied to, for example, an interpersonal robot having a speech function.

2. Description of the Related Art Conventionally, there is a known technology that detects a person and causes a robot or the like to speak (for example, see Patent Document 1). In this example of prior art, when an interactive robot is used in an environment where multiple people come and go, the robot determines the person's level of interest and determines whether the person has an intention to interact or is interested in the conversation. We are narrowing down the list of people. For this reason, the prior art detects a person from image information from an imaging device built into the robot, tracks the detected person in multiple images of the imaging device, and calculates the degree of interest of the tracked person in the multiple images. This is calculated based on changes in the direction of a person's face and body, and processing is performed to select dialogue candidates based on the calculated interest level.

JP2019-217558A

The above-mentioned prior art extracts a rectangular region in which a person exists from image information, and then detects the head region within the region, detects the face, and detects the head region using information within the head region. It performs complex processing such as estimating the direction of a person's body, and further estimating the direction of a person's torso. Such processing can be suitably executed by an artificial intelligence model having a neural network using a convolution filter.

However, if you try to accurately estimate not only the area of the person, but also the area of the head, the position of the face, the direction of the head, and even the direction of the person's torso, and then make the final call decision, Since the applied artificial intelligence must employ a highly accurate and highly functional model, there is a problem in that the processing time is correspondingly long and there is a delay from the input of image information to the output of the determination result. However, simply applying an artificial intelligence model that processes faster than a high-precision model will result in a problem in that the required level cannot be met because the accuracy of the judgment will be sacrificed.

Therefore, the present invention provides a technique that can speed up the determination of a person and appropriately control speech.

The present invention provides a speech control device. This speech control device performs determination (detection) of a person in an image using an artificial intelligence model that has a determination ability capable of high-speed processing. Speech control using such a high-speed artificial intelligence model has little delay in the response time from identifying the person to outputting the spoken voice, so it is especially useful for controlling speech to randomly moving people (calling out). ) has the great advantage that there is no noticeable delay in the timing of the utterance, and the person can be sure to notice the content of the utterance. However, the trade-off with speeding up the decision-making ability is that accuracy is sacrificed, so it is necessary to consider a method to compensate for this.

In other words, even if an image is taken of an imaging area where a person actually exists, the results of determining a person from that image will include a certain percentage of successes (person identification) and failures (no person identification). , and the number of occurrences and order of occurrence are irregular. In this case, if all the determination results are assumed to be correct and the output of the spoken voice is controlled, the same person may repeat the same utterance (sequential calling), or the person may not speak even though the person is present.

Therefore, the speech control device of the present invention employs a filtering method for the person determination results. That is, instead of using the series of determination results as input for the output of the uttered voice, a hypothetical person detection result is secondarily generated from the obtained determination results. In contrast to a series of synthetically generated human detection results, where a series of judgment results oscillates sensitively between success and failure (changes between extremes), it is possible to fictitiously say "detection result found" with a certain degree of certainty. It is smoothed to either "no detection result (undetected)".

Then, regarding the person indicated by such a hypothetically generated detection result, a speech sound is outputted at the timing when it is determined that the person has entered the predetermined detection area. At this time, since the detection results used to output the uttered audio are filtered (smoothed), it is possible to reliably prevent problems such as the same person repeating the same utterance or not being uttered due to an unsuccessful judgment. can do.

Further, the detection area can be defined based on, for example, a distance at which it is thought that the content of the utterance can easily reach the person and be easily heard in terms of the positional relationship between the utterance source and the person. As a result, even in an environment where an unspecified person moves in an arbitrary location in a random direction (for example, a construction site), it is possible to take advantage of the quick response that can be achieved by using a high-speed model to identify the person. By outputting the spoken voice at a timing when the positional relationship between the two is at an optimal distance, it is possible to make it easier for the person to notice that the person has spoken, and to make it easier for the person to hear the content of the utterance.

The filtering method by the speech control device includes the following preferred aspects.
(1) Based on the ratio of successful cases (with person identification) and unsuccessful cases (no person identification) included in the series of judgment results of the high-speed model, the detection result that hypothetically detects or does not detect a person is calculated. generate. For example, when focusing on a group of consecutive determination results a certain number of times, if a predetermined percentage or more of successes (person detection) is found, a detection result that falsely indicates "person detection" is generated. On the other hand, if a predetermined percentage of successes (person determination) has not been reached in a group of consecutive determination results a certain number of times, a false detection result of "no person detected (undetected)" is generated. Therefore, even if the determination result by the high-speed model fluctuates temporarily (instantaneous), the generated detection result will not fluctuate greatly and will be smoothed.

(2) If a success (person detection) is obtained from the high-speed model a predetermined number of times in a row, a detection result is generated that artificially sets a person detection state, and then a success (person detection) is obtained a predetermined number of times in a row. If a determination result indicating that a person has been detected is not obtained, a detection result that virtually indicates that no person has been detected is generated. In this case, on the condition that the high-speed model successfully determines a person a predetermined number of times in a row, the state becomes "person detected" from then on. In this state, even if an unsuccessful determination result (no person detected) is obtained during the process, the state of "person detected" is maintained as the detection result after filtering. Therefore, the detection results are smoothed without being fluctuated due to failures occurring less than a predetermined number of times.

In any case, in the above filtering modes (1) and (2), even after generating a false detection result of "person detected", a small number of judgment results by the high-speed model are unsuccessful (no person detected). In some cases, In this case, as it is, it is not possible to temporarily (instantaneously) generate a person detection result based on the determination result of the high-speed model. Therefore, when an unsuccessful determination result is obtained after a successful determination result is obtained, the speech control device determines the hypothetical person detection result based on the last (immediately) successful determination result. generate. Thereby, it is possible to prevent omissions (missing) after generating a detection result that simulates "person detected", and to stably perform output control of the uttered sound.

According to the present invention, speech can be appropriately controlled.

FIG. 2 is a diagram illustrating an example of an application scene of the speech control device. FIG. 3 is a diagram illustrating a scene in which a mobile robot RB outputs a speech voice within a construction site CS. FIG. 1 is a block diagram showing a configuration example of a speech control device 100 according to an embodiment. 2 is a diagram showing an overview of processing by the calling system 110. FIG. 3 is a diagram illustrating an overview of processing (1) by the filtering unit 142. FIG. 3 is a diagram showing an overview of processing (2) by the filtering unit 142. FIG. 3 is a diagram showing an overview of processing by a detection area determination unit 140. FIG. It is a flowchart which shows the example of a procedure of filtering processing (1). It is a flowchart which shows the example of a procedure of filtering process (2). It is a flowchart which shows the example of a procedure of voice voice output processing.

Embodiments of the present invention will be described below with reference to the drawings. In the embodiments below, an example is given in which the speech control device is applied to voice output by a mobile robot (self-propelled robot), but the present invention is not limited to this example.

FIG. 1 is a diagram showing an example of an application scene of the speech control device. In this embodiment, use can be assumed, for example, at a construction site CS of a building such as a large building, an apartment building, a medical facility, or a welfare facility. At this construction site CS, the building structure (concrete beams BM, walls WL, floors FL, columns CL, etc.) has been completed to some extent, and people (workers, etc.) can walk inside. . Although not shown in FIG. 1, in addition to open spaces, the construction site CS also includes passages, rooms, elevator shafts, staircases, and the like.

For example, a self-propelled mobile robot RB is placed at the construction site CS. The mobile robot RB can move within the construction site CS using, for example, four wheels WH. Furthermore, the mobile robot RB can take images of its surroundings using the built-in IP camera 112, and can collect and produce sounds (output speech sounds) using the microphone/speaker 128.

Information obtained by the mobile robot RB as it moves within the construction site CS is uploaded onto, for example, a cloud computer via wireless communication. Furthermore, the mobile robot RB can update the system by downloading update information from the cloud computer at a timely manner. Such a mobile robot RB is equipped with a well-known autonomous movement control system and an environment detection system, which have already been widely provided, and a detailed explanation thereof will be omitted. Note that the mobile robot RB may be of a walking type.

The speech control device of this embodiment suitably realizes control of speech output by the mobile robot RB mentioned in this application example. Hereinafter, the speech output by the mobile robot RB will also be referred to as a "call".

FIG. 2 is a diagram illustrating a scene in which the mobile robot RB outputs a speech voice within the construction site CS. The mobile robot RB uses various sensors and AI (artificial intelligence) to recognize the date and time, surrounding environment, and people, and uses voices tailored to each person's situation and the surrounding climatic conditions and environment related to construction work. Make a bet.

(A) in FIG. 2: The mobile robot RB recognizes a worker at a construction site CS, for example, and selects the content of the utterance in accordance with the date and time, the environment, and the situation of the person to be addressed. In this example, we make a comprehensive judgment based on the circumstances such as the person is standing still, the current time is daytime, and the ambient temperature exceeds some threshold. Please do so.'' It is also possible to identify an individual person through facial recognition and address the person by name, such as "Mr. ○○."

(B) in FIG. 2: In addition, the mobile robot RB recognizes a worker at a construction site CS and recognizes construction-related information, for example. In this example, the system determines the construction-related information that a person is climbing a scaffolding SC and is working in a high place, and issues a message such as ``This is dangerous! Please work carefully.'' .

This system of calling out has a great advantage in improving safety compared to the case where the mobile robot RB calls out with a fixed voice. In other words, if the mobile robot RB moves around the construction site CS and uses a pattern in which it automatically calls out to people with a fixed utterance when it recognizes a person, the utterances will not reach the person working, and caution is required. It does not lead to arousal. On the other hand, if there is a pattern of voice guidance to workers with specific health information, danger information, or construction-related information tailored to the situation on the spot, it will alert the target person and ensure safety. The benefits for sexual improvement are greater.

[Balance between processing speed and accuracy]
Here, the subject matter handled by the speech control device of this embodiment is the balance between processing speed and accuracy required for detecting a person with the mobile robot RB. That is, when the mobile robot RB autonomously moves within the construction site CS and recognizes people (persons involved in the work) at various locations, it is necessary to output speech sounds at appropriate timing each time. At this time, the timing at which the spoken voice is output depends on the positional relationship with the person when the mobile robot RB is used as the utterance source, and specifically depends on the distance to the person. However, the person does not always stay in one place, but moves to perform necessary tasks, and the mobile robot RB also moves autonomously. For this reason, when the mobile robot RB identifies (detects) a person and calls out to them based on their positional relationship, if it takes too long a processing time to recognize the person, the person may move first during that time. This results in a delay in the timing of the call.

Therefore, it is possible to speed up the human detection process. To recognize a person by the mobile robot RB, an artificial intelligence model that determines the person from an image captured by the IP camera 112 is used. At this time, by applying an artificial intelligence model with faster processing speed, it is possible to instantly determine (detect) the person in the image, but the faster the processing speed of the model, the faster the It is also true that the accuracy of is low. For this reason, when using an artificial intelligence model that specializes in high-speed processing, there is uncertainty in the judgment of people (intuitively speaking, "flickering" or "shaking"), which may result in repeated calls. On the other hand, sometimes they don't call out to you. On the other hand, person determination using a high-speed model has a characteristic that the delay is small due to the low detection rate, and the number of times of person determination per unit time is several times greater than that of a high-precision model.

Therefore, in this embodiment, in consideration of the above-mentioned characteristics, a mechanism is constructed that compensates for the inaccuracies caused by the artificial intelligence model specialized in high-speed processing, and allows the mobile robot RB to optimally call out to the user. Hereinafter, the calling mechanism used in this embodiment will be explained.

[Configuration of speech control device]
FIG. 3 is a block diagram showing a configuration example of the speech control device 100 of one embodiment. Note that in FIG. 3, some of the components of the mobile robot RB are also shown.

The speech control device 100 is configured mainly with a calling system 110. The calling system 110 inputs signals from an IP camera 112 and a microphone/speaker 128, performs internal processing using AI (high-speed model) and various calculations, and then outputs speech from the microphone/speaker 128. Achieve control.

The microphone/speaker 128 is used, for example, to measure the surrounding noise level or to output speech from the mobile robot RB. Note that the microphone/speaker 128 may have a separate configuration (the microphone and speaker are separate).

The IP camera 112 is used to capture images of the surrounding environment including people. For example, a known commercially available product can be applied to the IP camera 112. The IP camera 112 is a network camera equipped with a so-called pan, tilt, zoom (PTZ) function, but in this embodiment, the PTZ function is not particularly used (although it may be used). The IP camera 112 is built into the main body (for example, the head) of the mobile robot RB (see FIG. 1). Here, the direction of the IP camera 112 is set to be in front of the mobile robot RB in the direction of movement.

Furthermore, an AI processing acceleration device 114 is added to the calling system 110. For example, a known commercially available product can be used as the AI processing acceleration device 114, and the AI processing acceleration device 114 contributes to speeding up the AI processing executed inside the calling system 110.

The calling system 110 cooperates with the control unit 130 of the mobile robot RB. The control unit 130 cooperates with the calling system 110 to control the moving device 132 of the mobile robot RB. For example, when the greeting system 110 executes a greeting, the control unit 130 stops the movement of the mobile robot RB or adjusts the positional relationship with the target person. Alternatively, the calling system 110 may perform calling while the control unit 130 moves the mobile robot RB.

The calling system 110 can be realized using, for example, computer equipment including a CPU (central processing unit) and its peripheral equipment (not shown). The calling system 110 may be separate hardware that is additionally installed in the system of the mobile robot RB, or may be software that is installed on the hardware that the mobile robot RB already has.

The calling system 110 includes various functional blocks such as a person determination section 136, a detection area determination section 140, a filtering section 142, and a calculation section 122, for example. These functional blocks can be realized by, for example, AI processing or software processing performed using a computer program. In this embodiment, a high-speed AI model is adopted for the processing of the person determination unit 136. Each functional block executes processing while interoperating with each other through an internal bus (virtual bus) of the calling system 110.

Further, the calling system 110 includes a storage unit 124 and an output device 126. The storage unit 124 is, for example, a semiconductor memory or a magnetic recording device. The storage unit 124 stores, for example, audio data of utterances that the calling system 110 causes the mobile robot RB to output. The output device 126 is a driver amplifier or the like that drives the microphone/speaker 128. Note that the audio data can be updated as appropriate.

FIG. 4 is a diagram showing an overview of processing by the calling system 110. Note that details of specific processing will be described later using a separate flowchart.

For example, as shown in (A) to (H) in FIG. 4, an imaging signal from an IP camera 112 (omitted in FIG. 4) built into the mobile robot RB is input to the calling system 110. Imaging by the IP camera 112 is performed continuously (for example, at 30 to 60 frames per second (fps)), and these frame images are continuously input to the calling system 110. Note that for the sake of simplification, the number of frames is thinned out as appropriate (the same applies hereafter).

[Imaging area]
As shown in the central region of FIG. 4, the imaging area is defined by the angle of view of the IP camera 112 (for example, about 64 degrees left and right in the horizontal direction, about 28 degrees upward and about 10 degrees downward in the vertical direction). The frame image is an image of the surrounding environment that falls within this angle of view (field of view). Note that the range (angle) of the imaging area is not limited to this example.

[Detection area]
The calling system 110 predefines a detection area DA (range shown in gray in FIG. 4) within the imaging area. The detection area DA is, for example, a certain range with the center of the mobile robot RB (the imaging point by the IP camera 112) as a reference point, and here, it is a nearly fan-shaped belt-shaped range with a radius of R1 to R3 (for example, 2 m to 5 m). It is. The detection area DA includes a speaking point that is considered to be at an optimal distance (for example, 4 m) for a call from the mobile robot RB. Note that the distance to the speaking point and the range of the detection area DA are not limited to this example.

[Person identification department]
The person determination unit 136 executes a person determination process using a high-speed AI model from consecutive frame images. Determination of a person is performed by image recognition processing using, for example, a convolutional neural network. Here, by using the support of the AI processing acceleration device 114, it is possible to quickly determine a person at a frequency of several times (3 to 4 times) or more per second, for example.

[Judgment accuracy]
However, as described above, the person determination results obtained by the high-speed AI model include a certain amount of successful samples and unsuccessful samples. For example, in the frame images of (A) and (B) in FIG. 4, the image area where a person has been determined is indicated by a rectangular frame (bounding box) with a dashed dotted line, and these are the areas where the person determination unit 136 determines the person. It means that it is successful (detected). However, in the next frame image shown in FIG. 4 (C), the bounding box has disappeared, which means that the person determination unit 136 has failed to determine the person (undetected). .

Similarly, in the frame image (D) in Figure 4, the person was successfully determined (detected), but in the following frame images (E) and (F), it was unsuccessful (undetected). It has become. In the frame images (G) and (H), the person was successfully identified (detected), but the person was unsuccessfully identified twice (undetected) from (D). ).

In such a case, if you try to use the series of judgment results obtained by the person judgment unit 136 as they are to control the calling, the mobile robot RB will be able to detect each position marked with "detection" in the central area in FIG. The person indicated by the solid line has been recognized (detected), but the person indicated by the two-dot chain line at each position marked with "undetected" has not been recognized (detected). In other words, from the mobile robot RB, after the person recognized (detected) at the position of the frame image (A) moves to the position of the frame image (B), the person disappears halfway and is recognized (detected) at the position of the frame image (D). It will be recognized as if the image has moved significantly to the position of the frame image at the next moment (G).

If the mobile robot RB directly calls out based on the result of such person determination (detection), for example, after calling out at the position of the frame image (D), it will also repeat the same content at the position of the frame image (G). Problems arise with repeated calls. In this case, the effort made to call out to the person becomes annoying and makes the person feel uncomfortable.

Or, even though the person has approached the position of the frame images in (E) and (F), the mobile robot RB has not been able to determine (detect) the person, so there is no response such as not calling out to them. problem occurs. In this case, even though we were able to determine (detect) the presence of a person from a faraway position as shown in the frame image (A), we missed the opportunity to call out to them when the positional relationship was appropriate. become. In particular, since the position of the frame image (F) is within the detection area DA, it is not preferable that the call is not made here.

[Filtering section]
For this reason, in this embodiment, processing by the filtering section 142 is used. 5 and 6 are diagrams showing an overview of processing by the filtering unit 142. The filtering unit 142 of this embodiment can perform filtering processing in two different ways, for example. For this reason, an outline of the filtering process (1) is shown in FIG. 5, and an outline of the filtering process (2) is shown in FIG. The processing of the filtering section 142 will be explained below.

[Filtering process (1)]
The filtering unit 142 continuously observes the person determination result by the person determining unit 136. In this example, (detection data A), (detection data B), (detection data C), (detection data D), (detection data E), (detection data F), (detection data (No data), (Detection data G), (Detection data H), (Detection data I), (Detection data J), (No detection data), (Detection data K), (No detection data), (No detection data ) and (no detection data), determination results are obtained for each series of frame images.

Here, (detection data A), (detection data B), ... (detection data K) represent that a person has been determined (detected) in each frame image. Furthermore, the symbols A, B, . . . K identify the determination results for each frame image. For example, between (detection data A) to (detection data F) and (detection data G) to (detection data K), the size of the bounding box used to determine the person is different, which means that the position of the person is different. are doing. Therefore, the distances from the mobile robot RB to the person are different between (detection data A) to (detection data F) and (detection data G) to (detection data K). Furthermore, the distance to the person decreases from (detection data G) to (detection data K).

The processing by the filtering unit 142 can be explained using the processing table shown in the lower area of FIG. This processing table is, for example, a convenient visualization of a data array developed in a memory space. At this time, in the processing table, the data areas of "detection result", "internal state", and "output" are defined in the vertical direction, and the data corresponding to each data area is arranged in chronological order in the horizontal direction. ing.

[Detection result data array]
As shown in the upper part of the processing table, the "detection results" data area contains a series of determination results (detection results) by the person determination unit 136 from the left (oldest in chronological order) to the right (latest). ) are arranged sequentially. Here, the third frames from the left all have no data, and the fourth to ninth frames have detected data "A" to "F" arranged in order. Further, the 10th frame has no data, and the 11th to 14th frames have detected data of "G" to "J" arranged in order. The 15th frame again has no data, but the 16th frame has "K" detection data arranged. The 17th and subsequent frames continue to have no data. Such a data array corresponds to the determination results for each of the series of frame images shown in the upper frame in FIG.

[Internal state data array]
The data array of "internal state" shown in the middle row of the processing table is determined based on the data array of "detection result" in the upper row. Specifically, when the detection data is included in a predetermined ratio (e.g. 60%) or more in n consecutive data (e.g. 3 pieces), the filtering unit 142 sets the internal state to the "detection state" and sets the internal state to the "detection state", and sets the internal state to the "detection state". If the condition is not met, the internal state is set to "undetected state". In this example, since there is no detection data in the three frames from the left, the internal state up to this point is "undetected state." There is one detection data A in the second to fourth frames, but since it is less than 60%, the internal state remains in the "undetected state". The third to fifth frames have detection data A and B, which account for 60% or more, so the internal state becomes the "detection state" from this point on. Thereafter, similarly, if 60% or more of the detected data is present in n consecutive pieces of data, the internal state becomes the "detected state". There is one detection data K in the 15th to 17th frames, and the internal state becomes the "undetected state" from this point on.

[Output data array]
The "output" data array shown at the bottom of the processing table indicates the detection data output by the filtering section 142. The output from the filtering section 142 is a hypothetical detection result generated based on the determination result of the person determining section 136. Specifically, when the "internal state" is the "detection state", the filtering unit 142 outputs the detection data obtained last, assuming that it is the detection result at that time. In this example, when the internal state becomes the "detection state" for the first time in time series, the last obtained detection data B is output. Thereafter, detection data C, D, E, and F are sequentially output, but when there is no detection data in the 10th frame, the detection data F that was obtained last at this point is output. From the next time, detection data G, H, I, and J will be output again, but since there is no detection data in the 15th frame, the detection data J that was obtained last at this point will be output. ing. Since the 16th piece of detection data K is the last, the detection data K is output at this point.

[Filtering process (2)]
Filtering process (2) shown in FIG. 6 processes "internal state" and "output" using a logic different from that of filtering process (1) described above. That is, although the determination results shown in the upper frame in FIG. 6 are the same, the data arrangement of the "internal state" in the middle part of the processing table and the "output" in the lower part of the processing table shown in the lower part are different from those in FIG. Note that the "detection results" at the top of the processing table are the same as in FIG. 5.

[Internal state data array]
For example, at an early stage when detection data is not yet obtained, the internal state is in the "undetected state." From here, when detection data is obtained for n consecutive frames (for example, 3 frames), the filtering unit 142 sets the internal state to the "detection state." In this example, the fourth to sixth frames indicated by thick frames include detection data A, B, and C, and since they are continuous for n frames, the internal state becomes the "detection state" from this point on. Thereafter, the same internal state continues, and if no detection data is obtained for n consecutive frames, the internal state is set to "undetected state." In this example, the 17th to 19th frames indicated by thick frames have no data, and since there are n consecutive frames, the internal state becomes "undetected state" from this point on.

[Output data array]
Similarly, in the filtering process (2), when the "internal state" is the "detection state", the filtering unit 142 outputs the detection data obtained last, assuming that it is the detection result at that point. do. This example differs from the filtering process (1) in that when the internal state becomes the "detection state" for the first time in time series, the detection data C obtained last is output. Thereafter, detection data D, E, and F are sequentially output, and when there is no detection data in the 10th frame, the detection data F that was obtained last at this point is output. From the next time on, detection data G, H, I, and J will be output, but since there is no detection data in the 15th frame, the detection data J that was obtained last at this point will be output. Since the detection data K is the last in the 17th and 18th frames, the detection data K is output for each frame.

[Speech timing]
FIG. 7 is a diagram showing an overview of processing by the detection area determination unit 140. The detection area determining unit 140 determines whether the person P has entered the detection area DA, based on the person P indicated by the detection result (detection data B, C, . . . K) by the filtering unit 142. At this time, the location (distance) of the person P is estimated from the size of the bounding box shown in each detection data. By storing the relationship between the distance to the person P and the size (height) of the bounding box in advance as correlation data, the distance to the person P is estimated from the size of the bounding box shown in each detection data. .

The detection area determination unit 140 tracks the person P from a distance outside the detection area DA based on the output from the filtering unit 142, and constantly estimates the distance. As a result, when it is determined that the person P has entered the detection area DA (within 5 m in this example), the detection area determination section 140 outputs the determination result to the calculation section 122 at that timing. In response to this, the arithmetic unit 122 drives the output device 126 to cause the microphone/speaker 128 to output the spoken voice. As a result, when the person P actually enters the detection area DA, the mobile robot RB will immediately (without delay) appropriately say, "Hello, please be careful of heatstroke." . Note that the content of the call is not limited to this.

[Example of processing program]
The above description has clarified the outline of the processing by each functional block of the calling system 110, but below, the specific processing procedure will be explained using a flowchart.

[Filtering process (1)]
FIG. 8 is a flowchart illustrating an example of a procedure for filtering processing (1) as part of a program executed by the filtering unit 142. This process corresponds to the process table shown in FIG. The procedure will be explained below using an example procedure.

Step S100: The filtering unit 142 defines the number of n frames for the first time. Here, for example, the number of n frames is defined as "3". Note that the definition is performed only when the process is executed for the first frame, and is not defined again when the process is repeatedly executed for subsequent frames. Further, the value of the number of n frames defined here can be arbitrarily rewritten in the calling system 110.

Step S102: The filtering unit 142 inputs the determination result (detection data) of the person determining unit 136 for each frame. The determination results input here are (detection data A), (detection data B), . . . (detection data K), (no detection data), etc. of each frame.

[1st frame processing]
Step S104: If there is detection data (Yes), the filtering unit 142 proceeds to Step S106, but in the example of the processing table of FIG. 5, since there is no detection data in the first frame (No), the filtering unit 142 proceeds to Step S118. move on.

Step S118: The filtering unit 142 checks whether the variable N is greater than 0. Here, since the variable N is set to the initial value 0, the variable N is not greater than 0 (No), and the process advances to step S124.

Step S124: The filtering unit 142 increments the variable N by 1. Here, the value "1" is assigned to the variable N, which had an initial value of 0.
Step S126: The filtering unit 142 sets the internal state to "undetected". Therefore, in the example of the processing table of FIG. 5, the internal state becomes "undetected" in the first frame.

Step S128: The filtering unit 142 outputs the detection data "none". That is, in the example of the processing table in FIG. 5, there is no output in the first frame.
The filtering unit 142 temporarily leaves (returns) this process. Then, this process is executed for the second frame.

[2nd frame processing]
Step S118: In the process of the second frame, even if there is no detected data (step S104=No), the variable N is greater than 0 (Yes), so the process proceeds to step S106.
Step S106: The filtering unit 142 increments the variable N by 1. In the second frame, the value "2" is assigned to the variable N.

Step S108: If the variable N is equal to the defined number of frames n (Yes), the filtering unit 142 proceeds to Step S110, but here, since the number of frames is less than n (3) (No), the process proceeds to Step S126. .

Step S126: The filtering unit 142 sets the internal state to "undetected". Therefore, in the example of the processing table of FIG. 5, the internal state becomes "undetected" in the second frame.
Step S128: Then, the filtering unit 142 outputs the detection data "none". That is, in the example of the processing table in FIG. 5, there is no output in the second frame.
The filtering unit 142 temporarily leaves (returns) this process. Then, this process is executed for the third frame.

[3rd frame processing]
Step S118: In the third frame process, even if there is no detected data (step S104=No), the variable N is greater than 0 (Yes), so the process proceeds to step S106.
Step S106: The filtering unit 142 increments the variable N by 1. In the third frame, the value "3" is assigned to the variable N.

Step S108: In this case, since the variable N is equal to the defined number of frames n (Yes), the process proceeds to step S110.
Step S110: The filtering unit 142 compares the number of detected data in n frames with a threshold value x (for example, x=2), and if the number is equal to or greater than the threshold value x (Yes), the process proceeds to step S112. However, in the example of the processing table in FIG. 5, the number of detected data is still 0 in the third frame (No), so the process advances to step S120. Note that the value of the threshold x can be arbitrarily rewritten.

Step S120: The filtering unit 142 sets the internal state to "undetected". Therefore, in the example of the processing table of FIG. 5, the internal state becomes "undetected" in the third frame.

Step S122: Then, the filtering unit 142 outputs the detection data "none". That is, in the example of the processing table in FIG. 5, there is no output in the third frame.
Step S116: Here, the filtering unit 142 decrements the variable N by 1. As a result, the value "2=3-1" is assigned to the variable N.
The filtering unit 142 temporarily leaves (returns) this process. Then, this process is executed for the fourth frame.

[4th frame processing]
Step S104: In the example of the processing table in FIG. 5, detection data A is input in the fourth frame. Therefore, it is determined that there is detection data (Yes), and the process advances to step S106.
Step S106: The filtering unit 142 increments the variable N by 1. In the fourth frame, the value "3=2+1" is assigned to the variable N again.

Step S108: In this case, since the variable N is equal to the defined number of frames n (Yes), the process proceeds to step S110.
Step S110: In the example of the processing table in FIG. 5, the number of detected data is 1 in the fourth frame (No), so the process proceeds to step S120.

Step S120: The filtering unit 142 sets the internal state to "undetected". Therefore, in the example of the processing table of FIG. 5, the internal state becomes "undetected" in the fourth frame.

Step S122: Then, the filtering unit 142 outputs the detection data "none". That is, in the example of the processing table in FIG. 5, there is no output in the fourth frame.
Step S116: Also, the filtering unit 142 decrements the variable N by 1. As a result, the value "2=3-1" is assigned to the variable N again.
The filtering unit 142 temporarily leaves (returns) this process. Then, this process is executed for the fifth frame.

[5th frame processing]
Step S104: In the example of the processing table in FIG. 5, detection data B is input in the fifth frame. Therefore, it is determined that there is detection data (Yes), and the process advances to step S106.
Step S106: The filtering unit 142 increments the variable N by 1. In the fifth frame, the value "3=2+1" is assigned to the variable N again.

Step S108: In this case, since the variable N is equal to the defined number of frames n (Yes), the process proceeds to step S110.
Step S110: In the example of the processing table of FIG. 5, the number of detected data is 2 in the 5th frame (Yes), so the process proceeds to step S112.

Step S112: Here, the filtering unit 142 sets the internal state to "detection". Therefore, in the example of the processing table in FIG. 5, the internal state becomes "detected" in the fifth frame.

Step S114: Then, the filtering unit 142 outputs the latest detection data. That is, in the example of the processing table in FIG. 5, the latest detection data B is output in the fifth frame.
Step S116: Also, the filtering unit 142 decrements the variable N by 1. As a result, the value "2=3-1" is assigned to the variable N again.
The filtering unit 142 temporarily leaves (returns) this process. Then, this process is sequentially executed for the sixth frame and subsequent frames.

[10th frame processing]
The processing for the 10th frame is as follows.
Step S118: In the example of the processing table of FIG. 5, even if there is no detected data in the processing of the 10th frame (step S104=No), the variable N is greater than 0 (Yes), and the process proceeds to step S106.
Step S106: The filtering unit 142 increments the variable N by 1. In the 10th frame, the value "3" is assigned to the variable N.

Step S108: In this case, since the variable N is equal to the defined number of frames n (Yes), the process proceeds to step S110.
Step S110: In the example of the processing table of FIG. 5, the number of detected data is 2 in the 10th frame (Yes), so the process proceeds to step S112.

Step S112: The filtering unit 142 sets the internal state to "detection". Therefore, in the example of the processing table in FIG. 5, the internal state becomes "detected" in the 10th frame.

Step S114: Then, the filtering unit 142 outputs the latest detection data. That is, in the example of the processing table in FIG. 5, the latest detection data F is output in the tenth frame.
Step S116: Also, the filtering unit 142 decrements the variable N by 1. As a result, the value "2=3-1" is assigned to the variable N again.
The filtering unit 142 temporarily leaves (returns) this process. Then, this process is executed sequentially for the 11th frame and thereafter.

[17th frame processing]
The processing for the 17th frame is as follows.
Step S118: In the example of the processing table of FIG. 5, even if there is no detected data in the processing of the 17th frame (step S104=No), the variable N is greater than 0 (Yes), and the process proceeds to step S106.
Step S106: The filtering unit 142 increments the variable N by 1. In the 10th frame, the value "3" is assigned to the variable N.

Step S108: In this case, since the variable N is equal to the defined number of frames n (Yes), the process proceeds to step S110.
Step S110: In the example of the processing table in FIG. 5, the number of detected data is 1 in the 17th frame (No), so the process proceeds to step S120.

Step S120: The filtering unit 142 sets the internal state to "undetected" here. Therefore, in the example of the processing table of FIG. 5, the internal state becomes "undetected" in the 17th frame.

Step S122: Then, the filtering unit 142 outputs the detection data "none". That is, in the example of the processing table in FIG. 5, there is no output at the 17th frame.
Step S116: Also, the filtering unit 142 decrements the variable N by 1. As a result, the value "2=3-1" is assigned to the variable N again.
The filtering unit 142 temporarily leaves (returns) this process. Then, this process is sequentially executed for the 18th frame and thereafter.

[Filtering process (2)]
FIG. 9 is a flowchart illustrating an example of the procedure of filtering process (2). This process corresponds to the process table shown in FIG. The procedure will be explained below using an example procedure.

Step S200: The filtering unit 142 defines the number of n frames for the first time. The content of the process is the same as step S100 of filtering process (1).
Step S202: The filtering unit 142 inputs the determination result (detection data) of the person determining unit 136 for each frame. The content of the process is the same as step S102 of filtering process (1).

[1st frame processing]
Step S204: If there is detection data (Yes), the filtering unit 142 proceeds to Step S206, but in the example of the processing table of FIG. 6, since there is no detection data in the first frame (No), the filtering unit 142 proceeds to Step S216. move on.

Step S216: The filtering unit 142 resets the variable _N1 to the value "0" and increments the variable _N2 by 1. Since the variable _N2 is set to an initial value of 0, the value "1" is assigned to the variable _N2 here. Note that the variable _N1 also has an initial value of 0.
Step S218: If the variable _N2 is equal to the defined number of frames n (Yes), the filtering unit 142 proceeds to Step S224, but here, since the number of frames is less than n (3) (No), the filtering unit 142 proceeds to Step S220. move on.

Step S220: If the internal state is "detected" (Yes), the filtering unit 142 proceeds to step S214. However, in the example of the processing table of FIG. 6, the internal state of the first frame is "undetected" (No), so the process advances to step S222.

Step S222: The filtering unit 142 outputs the detection data "none". That is, in the example of the processing table in FIG. 6, there is no output in the first frame.
The filtering unit 142 temporarily leaves (returns) this process. Then, this process is sequentially executed in the same manner as above for the second and subsequent frames.

[4th frame processing]
The processing for the fourth frame is as follows.
Step S204: In the example of the processing table of FIG. 6, since the detection data A is present in the fourth frame (Yes), the process proceeds to step S206.
Step S206: The filtering unit 142 resets the variable _N2 to the value "0" and increments the variable _N1 by one. Since the variable _N1 is set to the initial value 0, the value " ₁ " is assigned to the variable N1 here.

Step S208: If the variable _N1 is equal to the defined number of frames n (Yes), the filtering unit 142 proceeds to Step S210, but here, since the number of frames is less than n (3) (No), the filtering unit 142 proceeds to Step S220. move on.

Step S220: If the internal state is "detected" (Yes), the filtering unit 142 proceeds to step S214. However, in the example of the processing table in FIG. 6, the internal state of the fourth frame is "undetected" (No), so the process advances to step S222.

Step S222: The filtering unit 142 outputs the detection data "none". That is, in the example of the processing table in FIG. 6, there is no output in the fourth frame.
The filtering unit 142 temporarily leaves (returns) this process. Then, for the fifth frame as well, this process is executed in the same manner as above.

[6th frame processing]
The processing for the 6th frame is as follows.
Step S204: In the example of the processing table of FIG. 6, since the detection data C exists in the sixth frame (Yes), the process advances to step S206.
Step S206: The filtering unit 142 resets the variable _N2 to the value "0" and increments the variable _N1 by one. Since the value "2=1+1" was assigned to the variable _N1 in the previous process of the fifth frame, the value "3=2+1" is assigned to the variable _N1 here.

Step S208: Since the variable _N1 is equal to the defined number of frames n (Yes), the filtering unit 142 proceeds to step S210.

Step S210: The filtering unit 142 sets the internal state to "detection". As a result, in the example of the processing table of FIG. 6, the internal state of the sixth frame becomes "detection".
Step S212: Then, the variable _N1 is reset to the value "0".

Step S214: The filtering unit 142 outputs the latest detection data. That is, in the example of the processing table in FIG. 6, the latest detection data C is output in the sixth frame.
The filtering unit 142 temporarily leaves (returns) this process. Then, this process is executed in the same manner as above for the seventh frame as well. From the 7th frame to the 18th frame, the internal state is "detected" because the variable _N2 is less than the defined number of frames n.

[19th frame processing]
The processing for the 19th frame is as follows. Up to the previous processing of the 18th frame, the variable _N2 has a value of "2".
Step S204: In the example of the processing table of FIG. 6, since there is no detection data in the 19th frame (No), the process advances to step S216.

Step S216: The variable _N1 is reset to the value "0" and the variable _N2 is incremented by one. Since the value "2=1+1" was assigned to the variable _N2 in the previous process of the 18th frame, the value "3=2+1" is assigned to the variable _N2 here.

Step S218: Since the variable _N2 is equal to the defined number of frames n (Yes), the filtering unit 142 proceeds to step S224.

Step S224: The filtering unit 142 sets the internal state to "undetected". As a result, in the example of the processing table of FIG. 6, the internal state of the 19th frame becomes "undetected."
Step S226: Then, the variable _N2 is reset to the value "0".

Step S222: The filtering unit 142 outputs the detection data "none". That is, in the example of the processing table of FIG. 6, there is no output at the 19th frame.
The filtering unit 142 temporarily leaves (returns) this process. Then, this process is sequentially executed in the same manner as above for the 20th frame and thereafter.

[Voice voice output processing]
FIG. 10 is a flowchart illustrating an example of a procedure for outputting a greeting voice as part of a program executed by the calculation unit 122. The procedure will be explained below using an example procedure.

Step S300: The calculation unit 122 inputs the detection data from the filtering unit 142. The detection data input here is virtually generated, as shown in the processing table example of FIG. 5 or 6.
Step S302: If there is detection data (Yes), proceed to step S304. If there is no detection data (No), this process is temporarily exited (return).

Step S304: The calculation unit 122 obtains the determination result of the detection area determination unit 140, and if it is determined that the person has entered the detection area DA (Yes), then executes step S306. Otherwise (No), the process is temporarily exited (return).

Step S306: The calculation unit 122 instructs the output device 126 to output the greeting voice. Thereby, the utterance sound is output from the microphone/speaker 128 at the utterance timing when the person enters the detection area DA.

After executing the above procedure, the calculation unit 122 exits (returns) this process. Then, the same procedure as above is repeated.

In this way, each part of the greeting system 110 executes the processing in conjunction or in cooperation, so that the mobile robot RB can appropriately execute the greeting.

Note that in the above process, for convenience, status information such as "no detected data" is output in the undetected state, but the detected information itself may not be output in the undetected state.

According to the speech control device 100 of the embodiment as described above, by determining (detecting) a person at high speed, speech can be controlled without missing an appropriate timing and without repeating calls. As a result, even if an unspecified person is moving randomly, such as at a construction site CS, the mobile robot RB can autonomously move around the construction site CS during the day and call out to workers in a timely manner. At that time, it is possible to ensure that the person hears the content of the call. Moreover, the uncertainty (low detection rate) due to the installation of a high-speed AI model is appropriately compensated, and it is possible to realize a practical and natural calling system 110.

In addition, at a construction site CS, for example, there may be frames in which the IP camera 112 cannot capture a clear image of a person due to insufficient brightness in the surrounding environment, or frames in which the person's movement is faster than expected and the image of the person is unclear. Sometimes there is. In these cases, it often happens that detection data cannot be obtained for n consecutive frames, so the detection rate becomes even lower with the high-speed model, but if you use the logic of filtering processing (1), it is possible to detect data in n frames. By assuming that there is detected data if the ratio of undetected frames is equal to or higher than a predetermined ratio (the detected data is a predetermined ratio or more), the absolute number of undetected frames can be kept low.

The present invention is not limited to the embodiments described above, and can be implemented with various modifications.
As already mentioned, the object to which the speech control device 100 is applied is not limited to the mobile robot RB, but may also be a fixed robot, a vehicle or other machine that is not a robot, or a stationary device. It's okay.

The number, positions, shapes, orientations, etc. of the IP cameras 112 and microphones/speakers 128 can be selected or changed as appropriate. Furthermore, the AI processing acceleration device 114 is not essential, and it is not necessary to use it.

Furthermore, the procedure examples listed in the various processes (FIGS. 8 to 10) can be changed as appropriate, and the processing does not necessarily have to be performed according to the procedure examples. Further, the trigger (interrupt event processing or trigger event processing) at which each type of processing is executed may be determined as appropriate.

In addition, the structures mentioned with illustrations in the embodiments, etc. are just preferred examples, and it is possible to suitably implement the present invention even if various elements are added to the basic structure or some parts are replaced. Not even.

100 Speech control device 110 Calling system 112 IP camera 118 Interpersonal distance determination unit 122 Arithmetic unit (audio output unit)
126 Output device (audio output unit)
128 Microphone/speaker (audio output section)
136 Person determination unit 140 Detection area determination unit 142 Filtering unit DA detection area

Claims (3)

When human identification is performed continuously from images obtained by continuously capturing images of an imaging area where a person exists, the series of judgment results may contain irregular cases in which the identification of a person is successful and cases in which it is unsuccessful. a person determination unit having the determination ability included in;
The continuity of a detected state or an undetected state of a person is determined fictitiously from the ratio of successful cases and unsuccessful cases included in a series of judgment results obtained by the person judgment unit, and the person is based on the judgment. a filtering unit that generates a detected state or an undetected state as a virtual person detection result;
a detection area determination unit that defines a predetermined detection area within the imaging area and determines whether a person indicated by a hypothetical detection result by the filtering unit has entered the detection area;
A speech control device comprising: an audio output section that outputs a speech sound at a timing when the detection area determination section determines that a person has entered the detection area. When human identification is performed continuously from images obtained by continuously capturing images of an imaging area where a person exists, the series of judgment results may contain irregular cases in which the identification of a person is successful and cases in which it is unsuccessful. a person determination unit having the determination ability included in;
When the person determination section obtains a successful determination result a predetermined number of times in a row, a detection result is generated that virtually puts the person in a detection state based on the continuity of the determination results, and after that, a success determination result is generated a predetermined number of times in a row. a filtering unit that generates a detection result that virtually indicates that the person is not detected based on the continuity of the fact that no determination result is obtained when no determination result is obtained ;
a detection area determination unit that defines a predetermined detection area within the imaging area and determines whether a person indicated by a hypothetical detection result by the filtering unit has entered the detection area;
A speech control device comprising: an audio output section that outputs a speech sound at a timing when the detection area determination section determines that a person has entered the detection area. The speech control device according to claim 1 or 2 ,
The filtering section includes:
When an unsuccessful determination result is obtained after a successful determination result is obtained by the person determination section, a hypothetical person is continuously brought into the detection state based on the last successful determination result obtained. A speech control device characterized by generating a detection result.

Images