JP7418189B2 - Display image generation device and display image generation method - Google Patents

Description

The present disclosure relates to a display image generation device and a display image generation method.

2. Description of the Related Art Techniques for generating information regarding the position of a recognized object outside the vehicle are known. For example, Patent Document 1 discloses a technology that identifies an object outside the vehicle that a vehicle occupant is paying attention to using line of sight detection and voice recognition, and generates a display image that shows the relative position of the identified object with respect to the vehicle. .

Japanese Patent Application Publication No. 2006-90790

However, the above-mentioned conventional technology assumes that the user is looking in the direction in which the object exists, and the position of the object is determined regardless of whether the object is within the user's field of vision. It is not possible to generate information about Furthermore, the above-mentioned conventional technology attempts to generate information regarding the position of an object recognized by the user himself/herself, and the user generates information regarding the position of the object recognized by an entity other than the user. There is no consideration given to generation for this purpose.

The present disclosure has been made in view of such circumstances, and regardless of whether or not the extraction target recognized by an entity other than the user is included in the user's visual field, the extraction target An object of the present invention is to provide a display image generation device and a display image generation method that appropriately generate information regarding the position of a person.

A display image generation device according to the present disclosure is a display image generation device that identifies a target object included in a statement uttered by a speaker as an extraction target object, and generates a display image related to the extraction target object. A display image generation device according to the present disclosure includes a statement data acquisition section, a target object extraction section, a visual field image acquisition section, a target object determination section, and a display image generation section. The utterance data acquisition unit acquires utterance data of a utterance uttered to a user by a utterer. The target object extraction unit stores a plurality of target object data in advance, compares the plurality of target object data with the statement data acquired by the statement data acquisition unit, and extracts data that matches the target object data from among the statement data. Extract as an extraction target. The visual field image acquisition unit acquires an image including at least a visual field image corresponding to the user's visual field. The target object determining unit determines whether the extraction target extracted by the target object extracting unit is included in the visual field image. The display image generation unit acquires object information that is information regarding the position of the extraction object, and generates a display image that includes extracted object information different from the visual field image. The display image generation unit determines the display mode of the display image regarding the extraction target based on the determination result of the target object determination unit as to whether or not the extraction target is included in the visual field image , and determines whether the extraction target is included in the visual field image. Decide on different display modes depending on whether it is included or not.

According to the present disclosure, it is possible to appropriately generate information regarding the position of an object recognized by a subject other than the user, regardless of whether the object is included in the user's visual field. .

FIG. 1 is a block diagram showing a display image generation device according to a first embodiment. FIG. 2 is a diagram showing a user and a fellow passenger wearing a terminal and riding together in a vehicle. FIG. 2 is a plan view for explaining a user's visual field when viewed from above the vehicle. FIG. 3 is a diagram illustrating a surrounding situation corresponding to the visual field of user X in which display images are displayed in a superimposed manner in a first display mode. FIG. 3 is a diagram showing a display image display device on which a first display image is displayed. FIG. 6 is a diagram illustrating a surrounding situation corresponding to the visual field of user X in which display images are displayed in a superimposed manner in a second display mode. It is a figure which shows the display image display apparatus on which the 2nd display image was displayed. FIG. 6 is a diagram showing a surrounding situation corresponding to the visual field of user X in which display images are displayed in a superimposed manner in a third display mode. FIG. 6 is a diagram showing a display image display device on which a third display image is displayed. 7 is a flowchart showing display image generation processing according to the first embodiment. FIG. 2 is a block diagram showing a display image generation device according to a second embodiment. 7 is a flowchart showing display image generation processing according to the second embodiment. FIG. 3 is a block diagram showing a display image generation device according to a third embodiment. It is a flowchart which shows display image generation processing concerning a 3rd embodiment.

Hereinafter, exemplary embodiments of the present disclosure will be described with reference to the drawings. In the following description, the same or equivalent parts are given the same reference numerals, and redundant description will be omitted.
[First embodiment]

FIG. 1 is a block diagram showing a display image generation device 1A according to the first embodiment. FIG. 2 is a diagram showing a user X and a user Y who are wearing terminals and riding together in a vehicle 2A. FIG. 3 is a plan view for explaining the visual field Ex of the user X when viewed from above the vehicle 2A. FIG. 4A, FIG. 5A, and FIG. 6A are diagrams showing the surrounding situation corresponding to the visual field Ex of the user X in which display images are displayed in a superimposed manner in each display mode. FIG. 4B, FIG. 5B, and FIG. 6B are diagrams showing a display image display device on which each display image is displayed. As shown in FIGS. 1 to 6, the display image generation device 1A extracts an object T included in the utterance uttered by the utterer (that is, mentioned in the utterance uttered by the utterer). This is a device that identifies an object Te and generates a display image P regarding the extraction target object Te.

More specifically, the display image generation device 1A is a device that generates a display image P that is displayed superimposed on the surrounding situation corresponding to the user's X visual field Ex. User X is riding in a vehicle 2A with user Y (the speaker), who is a person, and is visually observing, for example, the scenery outside the vehicle. User X is wearing the user terminal 3A. User Y is wearing the speaker terminal 4 (see FIG. 2). In this embodiment, the display image generation device 1A will be described by exemplifying a situation in which user Y talks to user X.

Here, "user X's visual field Ex" means a visible area that is visible to user X. The "visible area" is an effective field of view that is a range in which information can be effectively obtained from the outside world when a human uses the eyes to set a fixation point (fixation point) near the center of the physiological visual field. For example, the visual field Ex of the user X may be set to all visible areas above, below, left and right about the central axis of the visual field Ex of the user X. FIG. 3 shows the visible area of the user X in the horizontal direction when viewed from above the vehicle 2A. The field of view Ex of the user X changes as the vehicle 2A moves. For example, in FIG. 3, the current location of the user X is indicated by the current location X1, and the location of the user X after moving from the current location is indicated by the moved location X2. In the following explanation, the field of view Ex of the user X is that the user X can see through the transparent display of the user terminal 3A when the user X wearing the user terminal 3A, which will be described later, is facing a predetermined direction. It is assumed that all areas above, below, left and right are set. Note that the visual field Ey of the user Y means a visible area that can be visually recognized by the user Y, similarly to the visual field Ex of the user X (see FIG. 2).

The "surrounding situation" means the actual scenery outside the vehicle (outside scenery) that is a region around the user X and is visible to the user X. The surrounding situation is, for example, a region spanning 360 degrees in the horizontal direction centering on the current position of the user X, and is the scenery outside the vehicle (external scenery) of the region including above and below the user X. The "surrounding situation corresponding to user X's visual field Ex" means the scenery outside the vehicle (external scenery) included in user X's visual field Ex. In other words, the surrounding situation is the scenery outside the vehicle within the user's X's visual field Ex. As shown in FIG. 3, as the vehicle 2A moves, the surrounding situation corresponding to the user's X's visual field Ex changes.

"Generating a display image" means generating image information to be displayed on a display or the like. When the image information generated by the display image generation device 1A is transmitted to a display or the like by wired communication or wireless communication, the display image P related to the transmitted image information can be displayed on the display. Note that the "display image P" is an image that displays information regarding the extraction target Te, and more specifically, an image that displays information regarding the position of the target object T. Here, the display image P is displayed superimposed on the surrounding situation corresponding to the user's X visual field Ex. The display image P may be, for example, an image containing characters indicating whether or not the extraction target Te is included in the visual field image, and a specific extraction target Te included in the visual field image appears surrounded by a frame. It may be an image of a rectangular frame displayed as shown in FIG. Note that details will be described later. Here, the "visual field image" is an image corresponding to the user's X's visual field Ex. In other words, the visual field image is an image of the surrounding situation corresponding to the user's X visual field Ex. In this embodiment, the surrounding situation corresponding to the visual field Ex of the user X is the actual scenery outside the vehicle, and the visual field image is the surrounding situation corresponding to the visual field Ex of the user This is an image taken by.

The display image generation device 1A is configured as a server, for example, and includes a processor (processing device), a memory (storage device), and the like.

The processor may be configured by, for example, a CPU (Central Processing Unit) or an MPU (Micro-Processing Unit). The memory may include at least one of a semiconductor storage device, a magnetic storage device, and an optical storage device. Further, the memory may include a register, a cache memory, a ROM (Read Only Memory) used as a main storage device, a RAM (Random Access Memory), or the like.

The display image generation device 1A, the vehicle 2A, the user terminal 3A, and the speaker terminal 4 are connected to each other so that they can communicate (transmit and receive) by wire or wirelessly. Note that the functional configuration of the display image generation device 1A will be described later.

The vehicle 2A is a passenger car or the like in which the user X and the user Y ride. The vehicle 2A may be capable of both manual operation and automatic operation by switching, or may be capable of operating only one of them. The vehicle 2A includes a navigation device 21 and a surrounding imaging device 22. The navigation device 21 sets a travel route for the vehicle 2A to a set destination based on, for example, location information of the vehicle 2A detected by GPS (Global Positioning System) or the like and map information, and This is a device that guides the vehicle 2A along a route. The navigation device 21 stores (holds) a history of the position of the vehicle 2A (for example, position coordinates detected by GPS) in chronological order. The navigation device 21 may acquire the traveling direction of the vehicle 2A based on the stored history of the position of the vehicle 2A.

The surrounding imaging device 22 is a device that images the surrounding situation of the user X and obtains a surrounding image. The "peripheral image" is an image of a region around the user X that includes the user's X's visual field Ex (that is, a visual field image) and includes the user's X's visual field Ex. The surrounding image of user X may be, for example, an image in which an area spanning 360 degrees in the horizontal direction centering on user X is imaged, or may be an image in which an area including above user X is imaged. . Alternatively, the peripheral image of user There may be. Alternatively, it may be the same area as the area corresponding to the user's X visual field Ex. The range of the "area including user X's visual field Ex" is not particularly limited as long as it is an area including user X's visual field Ex.

The surrounding imaging device 22 includes, for example, one or more cameras. The camera of the peripheral imaging device 22 may be provided outside the vehicle interior, such as on the roof of the vehicle 2A, or may be provided inside the vehicle interior, such as behind the windshield. The vehicle 2A transmits the surrounding image of the user X captured by the surrounding imaging device 22 to the display image generation device 1A. Note that "sending an image" means transmitting image data of an image.

The user terminal 3A is a device worn on the head of the user X, and includes a display image display device 31A and a visual field image acquisition device 32. The display image display device 31A has a display capable of displaying the display image P generated by the display image generation device 1A. The display of the display image display device 31A is, for example, a transmissive display such as a glasses type or a goggle type, and is located right in front of the eyes of the user X when the user terminal 3A is worn by the user X. Therefore, the user X can visually recognize the surrounding situation corresponding to the visual field Ex of the user X via the display image display device 31A. Further, when the display image P is displayed on the display image display device 31A, the display image P (see FIGS. 4B, 5B, and 6B) is superimposed on the surrounding situation corresponding to the user X's visual field Ex, as seen from the user X. will be displayed. In other words, the display image display device 31A has a function as an HMD (Head Mounted Display) used in so-called AR (Augmented Reality) technology.

The visual field image acquisition device 32 is an imaging device that images the surrounding situation corresponding to the visual field Ex of the user X and acquires a visual field image. The visual field image acquisition device 32 is provided in the user terminal 3A so as to be oriented so that it can capture an image of the line of sight of the user X when the user terminal 3A is worn by the user X. The visual field image acquisition device 32 is provided, for example, on the side of the display image display device 31A. The user terminal 3A transmits the visual field image captured by the visual field image acquisition device 32 to the display image generation device 1A. Note that "transmitting a visual field image" means transmitting image data of a visual field image. Further, the visual field image acquisition device 32 may include a sensor (not shown) that detects the visual line direction of the user X, and may transmit information on the visual line direction of the user X detected from the sensor together with the image data of the visual field image.

The speaker terminal 4 is a device worn on the head of the user Y, and includes a statement data acquisition device 41 . The utterance data acquisition device 41 is a device that acquires utterances uttered by user Y to user X as utterance data. The speech data acquisition device 41 includes, for example, a microphone. Here, the speech data acquisition device 41 is a headset, and the speech subject terminal 4 is a headset, and the speech data acquisition device 41 is a microphone provided in the headset. Note that the speech data acquisition device 41 may be an in-vehicle microphone or an earphone. Moreover, the speaker terminal 4 may further include a display image display device 31A and a visual field image acquisition device 32 similar to the user terminal 3A. "Speech data" is data having information about the content of a statement, and here, the statement data is speech signal data of a statement. "Speech signal data" means an audio signal of a speech. Note that the speech data also includes data in which user Y has not uttered anything.

The speaker terminal 4 transmits the statement acquired by the statement data acquisition device 41 to the display image generation device 1A. At this time, the speaker terminal 4 further transmits information specifying that the speaker terminal 4 is worn by user Y (information specifying user Y) to the display image generation device 1A. "Information identifying that the speaker terminal 4 is worn by user Y" is information linked to user Y, such as the ID of the speaker terminal 4 linked to user Y. (Identification) number may be used. Note that "sending a comment" means transmitting comment signal data (details will be described later) of the comment.

Next, the functional configuration of the display image generation device 1A will be explained. The display image generation device 1A includes a peripheral image acquisition unit 11, a statement data acquisition unit 12, a target object extraction unit 13, a visual field image acquisition unit 14A, a target object determination unit 15A, an existence determination unit 16A, a positional relationship acquisition unit 17A, and a display It has an image generation section 18A.

The peripheral image acquisition unit 11 acquires and stores peripheral images transmitted from the vehicle 2A. The peripheral image acquisition unit 11 acquires peripheral images of the user X and stores them in chronological order. More specifically, the peripheral image acquisition unit 11 acquires the peripheral image of the user X by receiving the peripheral image of the user X captured by the peripheral imaging device 22 of the vehicle 2A from the vehicle 2A. The surrounding image acquisition unit 11 stores the acquired surrounding images of the user X in chronological order. That is, the peripheral image acquisition unit 11 acquires the current peripheral image of the user X, and also stores (accumulates) the acquired peripheral image as a past peripheral image. The surrounding image acquisition unit 11 may delete the stored information on past surrounding images at a preset timing.

The utterance data acquisition unit 12 acquires utterance data of a utterance uttered by user Y to user X. More specifically, the utterance data acquisition unit 12 receives from the utterance terminal 4 the utterance signal data of user Y's utterance acquired by the utterance data acquisition device 41 of the utterance terminal 4, utterance data of the utterance uttered to user X is obtained. Note that the utterance data acquisition unit 12 determines whether or not the utterance data includes the utterance of user Y. That is, if user Y has not made a statement, it is determined that the statement data does not include user Y's statement.

The statement data acquisition unit 12 also obtains information that identifies the user Y who made the statement to the user X. For example, the statement data acquisition unit 12 receives information identifying user Y from the speaker terminal 4.

Based on the statement data acquired by the statement data acquisition unit 12, the target object extraction unit 13 extracts a character string representing a pre-stored target object T included in the statement related to the statement data. To be more specific, the object extraction unit 13 stores character strings (object data) representing a plurality of objects T in advance, and converts character strings representing the plurality of objects T and speech data ( A type of comment data) is compared, and a character string (data) that matches the character string representing the target object T from among the character strings obtained by converting the comment data is extracted as an extraction target Te. “Target T” is an object that actually exists. The object may be, for example, the type of object expressed as a common noun (bicycle, streetlight, building, etc.), or the name of an object expressed as a proper noun (Mt. Fuji, the National Diet Building, etc.). . Furthermore, the object may be limited in terms of its attributes, characteristics, etc. (for example, a blue bicycle, the top of Mt. Fuji, etc.). The object extraction unit 13 has a storage unit, and stores general nouns, proper nouns, attributes, or features representing the object T in advance. The object extraction unit 13 extracts a common noun, proper noun, attribute, or feature representing the object T stored in advance from the comment data acquired by the comment data acquisition unit 12.

As an example, a case will be described in which user Y says to user X, "There's a bicycle over there." In this case, based on the utterance data of the utterances uttered by the user Y by the utterance data acquisition unit 12, the object extraction unit 13 extracts the extraction target Te from the utterances uttered by the user Y. Here, it is assumed that the object extraction unit 13 has previously stored that the word "bicycle" represents the type of object T (bicycle). The object extracting unit 13 extracts an extraction object Te called "bicycle" from the utterance uttered by the user Y. Note that the extraction target Te may not be extracted from the content of user Y's statement.

The object extraction unit 13 extracts character strings representing a plurality of pre-stored objects T mentioned in the utterance related to the utterance data, for example, by utterance recognition (speech recognition). Here, a known speech recognition technique can be applied to the "utterance recognition". For example, the object extraction unit 13 uses speech recognition to recognize the speech audio signal as a character string based on the speech data, and extracts the extraction object Te from the recognized character string and character strings representing the plurality of objects T. do.

The visual field image acquisition unit 14A acquires an image that includes at least a visual field image that is an image corresponding to the user's X visual field Ex. The "image including at least the visual field image" may be an image in the same range as the visual field image, or may be an image in a wider range than the visual field image. The visual field image acquisition unit 14A receives the visual field image captured by the visual field image acquisition device 32 of the user terminal 3A from the visual field image acquisition device 32, and thereby acquires the visual field image. Further, the visual field image acquisition unit 14A may acquire information on the user's X line of sight direction from the visual field image acquisition device 32.

The target object determining unit 15A determines whether the extraction target Te is included in the visual field image of the visual field Ex of the user X acquired by the visual field image acquiring unit 14A. As described above, the "extraction object Te" is one that matches the statement data from among the plurality of objects T stored in the object extraction unit 13. Here, the object extraction unit 13 has extracted an extraction object Te called "bicycle."

The target object determination unit 15A determines whether the extraction target Te is included in the visual field image, for example, by image recognition. Here, as the "image recognition", a known image recognition technique can be applied. For example, the object determination unit 15A uses a machine learning model, a deep learning model, and an OpenCV (Open Source Computer An image processing algorithm using the Vision Library) may be applied.

For example, the target object determining unit 15A detects identification information of a plurality of objects included in the visual field image, and extracts data representing the extraction target Te acquired by the target object extracting unit 13 (object type, etc.) and the visual field image. Detected identification information of multiple objects included in the image is compared. Thereafter, the target object determination unit 15A determines whether or not the data expressing the extracted target object Te matches the plurality of objects included in the visual field image in at least one of the type and name. It is determined whether or not the extraction target Te is included. In addition, the target object determining unit 15A uses OCR (Optical Character Recognition) as image recognition to recognize the character content of the signboard included in the visual field image, and the target object extracting unit 13 recognizes the acquired extraction target Te. The name is compared with the data representing the name and the content of the recognized signboard, and based on whether the name of the extraction target Te matches a part of the content of at least one signboard included in the field of view image. Then, it may be determined whether or not the field of view image includes the extraction target Te.

The target object determining unit 15A outputs information on the determination result as to whether or not the extraction target Te is included in the visual field image to the user X's display image display device 31A. The “determination result as to whether or not it is included in the visual field image” means information on the determination result as to whether or not the extraction target Te is visible to the user X (within the visual field Ex of the user X). Here, the target object determination unit 15A outputs information on the determination result to the user terminal 3A of the user X. Note that information on the determination result is also output to the speaker terminal 4 of user Y, who is the speaker.

The presence/absence determining unit 16A determines whether the extraction target Te exists within a preset target range when the target object determining unit 15A determines that the extraction target Te is not included in the visual field image. do. Specifically, the presence/absence determination unit 16A determines whether or not the extraction target Te exists within the target range based on the current or past peripheral images acquired by the peripheral image acquisition unit 11. The "target range" is a predetermined range that is set in advance around the position of the user X or the vehicle 2A. For example, the target range may be a predetermined range that is visible to the user X and centered on the position of the user X or the vehicle 2A (the range shown by the two-dot chain line in FIG. 3). The range may be a circular range of, for example, 50 kilometers from the user X or the vehicle 2A, or may be a range of any shape other than a circle. Depending on the size of the extraction target Te, the target range may be a range in which the extraction target Te can be visually recognized by the user X. For example, if the extraction target Te is Mt. It is sufficient to set the range up to 300 kilometers from the position (center) of 2A. In this example, the target range is a radius of 300 kilometers from the center.

First, the presence/absence determination unit 16A determines whether or not the extraction target Te is included in the current and past peripheral images of the user X acquired by the peripheral image acquisition unit 11 and stored in chronological order. . More specifically, the presence/absence determination unit 16A acquires the current peripheral image and past peripheral images stored by the peripheral image acquisition unit 11, and adds the extraction target to the acquired current peripheral image and past peripheral image. It is determined whether Te is included. The presence/absence determining unit 16A may perform this determination by, for example, image recognition. The presence/absence determining unit 16A causes the object determining unit 15A to perform image recognition processing for a plurality of images included in the current peripheral image and past peripheral images acquired and stored by the peripheral image acquiring unit 11, and performs the image recognition process. The determination may be made based on the results.

In addition, the presence/absence determination unit 16A detects various objects in a plurality of images included in the current peripheral image and past peripheral images stored by the peripheral image acquisition unit 11, and determines the name, type, and shape of the object. , color, and orientation, and assigns one or more image tags to the detected identification information to generate and store a tagged image. Thereafter, the presence/absence determination unit 16A determines whether there is a surrounding image that matches the statement data expressing the extraction target Te acquired by the target object extraction unit 13 and at least one of the name and type of the object among the plurality of image tags. Based on whether or not, it is determined whether or not the extraction target Te is included in the acquired current surrounding image and past surrounding image.

In addition, when it is determined that the extraction target Te is not included in the acquired current peripheral image and past peripheral images, the presence/absence determining unit 16A determines that the extraction target Te is within a preset target range. It is judged that it does not exist.

Next, when it is determined that the extraction target Te is included in the acquired current peripheral image and past peripheral images, the presence/absence determining unit 16A determines that the position where the extraction target Te exists is within the target range. Determine whether or not. When the extraction target Te is included in the current surrounding image, the presence/absence determining unit 16A can obtain the direction and distance from the user X or the vehicle 2A to the extraction target Te using a known method. For example, the presence/absence determination unit 16A estimates the direction and distance from the user X or the vehicle 2A to the extraction target Te based on the current surrounding image of the user X acquired by the surrounding image acquisition unit 11, and It may be determined whether the position where Te exists is within the target range. Alternatively, the presence/absence determination unit 16A uses RADAR (Radio Detection and Ranging) or LIDAR (Light Detection and Ranging) provided in the vehicle 2A (not shown) to detect the distance from the user X or the vehicle 2A to the extraction target Te. The direction and distance may be measured to determine whether the position where the extraction target Te exists is within the target range.

Note that, when the extraction target Te is not included in the current peripheral image, the presence/absence determination unit 16A acquires the last peripheral image that includes the extraction target Te in time order from the peripheral image acquisition unit 11. Next, the presence/absence determining unit 16A calculates the relative direction and distance between the current user X or the vehicle 2A and the user X or the vehicle 2A at the time of image capture based on the actual vehicle position history acquired from the navigation device 21. Next, the presence/absence determining unit 16A determines the direction and distance from the user X or the vehicle 2A to the extraction target Te based on the relative direction and distance and the relative direction and distance from the user X or the vehicle 2A to the extraction target Te. Estimate distance. Subsequently, the presence/absence determination unit 16A may determine whether the distance from the user X or the vehicle 2A to the extraction target Te is within the target range.

The positional relationship acquisition unit 17A acquires the relative positional relationship between the extraction target Te and the user X. The "positional relationship" may be expressed by the direction and distance of the position of the extraction target Te with respect to the reference position (for example, the center position of the vehicle 2A) set at the position of the user X or a position near the user X. Alternatively, information may be expressed that the extraction target Te does not exist within a preset target range. The positional relationship acquisition unit 17A may acquire the direction and distance from the user X or the vehicle 2A to the extraction target Te from the existence determination unit 16A. Further, the positional relationship acquisition unit 17A may estimate the direction and distance from the user X or the vehicle 2A to the extraction target Te based on the current or past surrounding images acquired by the surrounding image acquisition unit 11. Further, the positional relationship acquisition unit 17A may estimate the direction and distance from the user X or the vehicle 2A to the extraction target Te using a radar, a lidar, or the like provided in the vehicle 2A. Further, the positional relationship acquisition unit 17A may acquire information that the extraction target Te does not exist within a preset target range from the presence/absence determination unit 16A.

The positional relationship acquisition unit 17A calculates the direction of the extraction target Te with respect to the direction of the user's X line of sight. The positional relationship acquisition unit 17A may estimate the user's gaze direction based on the visual field image of the user X acquired from the visual field image acquisition unit 14A and the peripheral image of the user X acquired from the peripheral image acquisition unit 11. Further, the positional relationship acquisition unit 17A may acquire the user X's line of sight direction from the visual field image acquisition unit 14A. The positional relationship acquisition unit 17A estimates the direction of the extraction target Te with respect to the user X's line-of-sight direction based on the calculated direction from the user X or vehicle 2A to the extraction target Te and the user's X line-of-sight direction. Moreover, the direction of the extraction target Te with respect to the line-of-sight direction of the user X may be one of two types: rear left in the line-of-sight direction and rear right in the line-of-sight direction.

The display image generation unit 18A acquires the extraction object information and generates a display image P including the extraction object information. "Extraction target object information" means information regarding the position of the extraction target Te. The extraction target object information may be information indicating the position of the extraction target Te itself, or may be information indicating the direction or distance in which the extraction target Te exists, and may be information indicating the direction or distance in which the extraction target Te is located within a predetermined area. It may be information indicating whether or not it exists.

The display image generation unit 18A determines the display mode of the display image P of the extraction target Te based on the determination result of the target object determination unit 15A. The "display mode" is a display mode of an image showing extraction target object information. The display mode may be an image that shows the position of the extraction target Te itself, or an image that shows the distance and direction of the extraction target Te as seen from the user, and may be an image that shows the extraction target Te within a predetermined area. It may also be an image indicating whether or not it exists.

When the target object determination unit 15A determines that the extraction target Te is included in the visual field image, the display image generation unit 18A acquires the visual field image of the visual field Ex from the visual field image acquisition unit 14A, and extracts it from the visual field image. The target object Te is image-recognized, and a first display image P1 is generated that shows the extraction target object information in a display mode that emphasizes the extraction target object Te itself, which is displayed superimposed on the extraction target object Te. The "display mode that emphasizes the extraction object itself" may be, for example, a display mode in which the extraction object Te is surrounded by a square or a circle, or a display mode in which the extraction object Te is directly pointed with an arrow. (See Figure 4).

In addition, when the target object determination unit 15A determines that the extraction target Te is not included in the visual field image, the display image generation unit 18A determines that the extraction target Te exists within the target range by the presence/absence determination unit 16A. Based on whether or not the determination has been made, the display mode of the extraction target object information is determined. More specifically, when the presence/absence determination unit 16A determines that the extraction target Te exists within the target range, the display image generation unit 18A calculates the direction and distance of the extraction target Te with reference to the reference position. A second display image P2 is generated that shows the extraction target object information in a display mode that displays the positional relationship including the extracted object information (see FIG. 5). The "display mode for displaying the positional relationship" is a display mode for an image showing the direction and distance of the position of the extraction target Te with reference to the reference position. The display image generation unit 18A acquires positional relationship information including the direction and distance of the position of the extraction target Te with the reference position as a reference by the positional relationship acquisition unit 17A, and calculates the position of the extraction target Te with the acquired reference position as a reference. A second display image P2 is generated that displays the positional relationship including the direction and distance of the positions. For example, if the extraction target Te is located at the rear left of the user X's visual field Ex, as shown in FIG. Display to the left of the image.

In addition, when the presence/absence determination unit 16A determines that the extraction target Te does not exist within the target range, the display image generation unit 18A indicates information that the extraction target Te does not exist within the preset target range. A third display image P3 is generated (see FIG. 6).

The display image generation unit 18A generates a display image P that includes information that identifies the person who made the statement, which was acquired by the statement data acquisition unit 12. For example, if the person who made the statement acquired by the statement data acquisition unit 12 is the user Y, the display image generation unit 18A generates the first to third display images P1 to P3 that say “Mentioned by Y.” (See Figures 4 to 6).

The display image generation unit 18A indicates whether or not the extraction target Te is visible to the user A display image P containing information is generated (see FIGS. 4 to 6). More specifically, the display image generation unit 18A indicates that the extraction target Te is visible to the user X when the target object determination unit 15A determines that the extraction target Te is included in the visual field image. Generates a first display image P1 including information, and indicates that the extraction target Te is not visible to the user X when the target object determination unit 15A determines that the extraction target Te is not included in the visual field image. A second display image P2 and a third display image P3 containing information are generated. For example, when the object determination unit 15A determines that the extraction target Te is included in the visual field image, the display image generation unit 18A generates the first display image P1 that reads “Bicycle is visible now.” (See Figure 4). On the other hand, when the object determining unit 15A determines that the extraction target Te is not included in the visual field image, the display image generating unit 18A generates a second display image P2 that reads “Bicycle is invisible now.” A display image P3 may be generated (see FIGS. 5 and 6).

Next, the image generation process executed by the display image generation device 1A will be described. FIG. 7 is a flowchart showing display image generation processing. In the flowchart of FIG. 7, for example, the display image generation process by the display image generation device 1A is started when the vehicle 2A is started.

As shown in FIG. 7, in step S101, the display image generation device 1A acquires a peripheral image of the user X using the peripheral image acquisition unit 11. The surrounding image acquisition unit 11 obtains a surrounding image captured by the surrounding imaging device 22 of the vehicle 2A. After that, the display image generation device 1A proceeds to step S102.

In step S<b>102 , the display image generation device 1</b>A uses the statement data acquisition unit 12 to obtain statement data of the statement uttered by the user (the subject of the statement) Y to the user X. The utterance data acquisition unit 12 acquires utterance data of the utterance uttered by the user Y to the user X, which is acquired from the utterance data acquisition device 41 of the utterance subject terminal 4. Note that, as described above, the speech data includes data in which user Y has not uttered anything. Further, the statement data acquisition unit 12 acquires information identifying the fellow passenger Y, and transmits it to the display image generation device 1A. After that, the process advances to step S103.

In step S103, the display image generation device 1A uses the statement data acquisition unit 12 to determine whether or not the statement data includes a statement by the user (the subject of the statement) Y. If it is determined that the statement by user Y is included, the process advances to step S104. If it is determined that the statement of user Y is not included, the process advances to the end.

In step S104, the display image generation device 1A determines whether the object extraction unit 13 can extract an extraction object Te that matches the object T from the statement data. If it is determined that the extraction target Te can be extracted, the process advances to step S105. If it is determined that the extraction target Te cannot be extracted, the process proceeds to the end.

In step S105, the display image generation device 1A acquires the visual field image of the user X using the visual field image acquisition unit 14A. The visual field image acquisition unit 14A acquires the visual field image of the user X from the visual field image acquisition device 32 of the user terminal 3A worn by the user X. After that, the process advances to step S106.

In step S106, the display image generation device 1A determines whether the extraction target Te extracted from the target object extraction unit 13 is included in the visual field image of the user X acquired from the visual field image acquisition unit 14A. Determine whether If it is determined that the extraction target Te is included in the visual field image of the user X, the process advances to step S107. If it is determined that the extraction target Te is not included in the visual field image of the user X, the process advances to step S108.

If it is determined that the extraction target Te is included in the visual field image of the user Generate image P1. The display image generation unit 18A acquires the visual field image of the visual field Ex from the visual field image acquisition unit 14A, performs image recognition of the extraction target Te from the visual field image, and emphasizes the extraction target Te itself displayed superimposed on the visual field image. A first display image P1 showing extraction target object information in a first display mode is generated (see FIG. 4). Note that the display image generation unit 18A generates information indicating that the extraction target Te is visible to the user The first display image P1 may be generated that further includes information identifying the speaker ("Mentioned by Y." in FIG. 4). The display image generation unit 18A transmits the generated first display image P1 to the display image display device 31A of the user terminal 3A.

If it is determined that the extraction target Te is not included in the visual field image of the user X, in step S108, the display image generation device 1A uses the current or Based on past surrounding images, it is determined whether the extraction target Te exists within the target range. When the presence/absence determining unit 16A determines that the extraction target Te does not exist within the target range, the process proceeds to step S111. When the presence/absence determining unit 16A determines that the extraction target Te exists within the target range, the process proceeds to step S109.

If it is determined that the position where the extraction target Te exists is within the target range, in step S109, the display image generation device 1A uses the positional relationship acquisition unit 17A to determine the position of the extraction target Te and the user X. Get relationships. The positional relationship acquisition unit 17A determines the distance from the extraction target Te to the user X or the vehicle 2A and the direction with respect to the visual field Ex of the user X, based on the current or past surrounding images of the user X acquired from the surrounding image acquisition unit 11. Estimate. Further, the positional relationship acquisition unit 17A may acquire the distance from the extraction target Te to the user X or the vehicle 2A from the presence/absence determination unit 16A. After that, the process advances to step S110.

In step S110, the display image generation device 1A causes the display image generation unit 18A to display the positional relationship including the direction and distance of the extraction target Te with reference to the reference position acquired from the positional relationship acquisition unit 17A. 2 display image P2 is generated. The display image generation unit 18A displays a symbol image (arrow in FIG. 5) indicating the direction with respect to the visual field Ex of the user X acquired from the positional relationship acquisition unit 17A and a distance (“20 m” in FIG. 5) in a second display mode. A second display image P2 showing extraction target object information is generated. Note that the display image generation unit 18A generates information indicating that the extracted object Te is not visible to the user A second display image P2 may be generated that includes information identifying the person who made the statement (“Mentioned by Y.” in FIG. 5). The display image generation unit 18A transmits the generated second display image P2 to the display image display device 31A of the user terminal 3A.

If it is determined that the position where the extraction target Te exists is not within the target range, in step S111, the display image generation device 1A uses the positional relationship acquisition unit 17A to determine the position of the extraction target Te and the user X. Get relationships. Specifically, the positional relationship acquisition unit 17A acquires positional relationship information in which the extraction target Te does not exist within a preset target range from the presence/absence determination unit 16A. After that, the process advances to step S112.

In step S112, the display image generation device 1A displays the positional relationship between the extraction target Te and the user X, which indicates that the extraction target Te acquired from the positional relationship acquisition unit 17A does not exist within the preset target range. A third display image P3 is generated. The display image generation unit 18A generates information indicating that the extracted object Te is invisible to the user X from the visual field image (“Bicycle is invisible now.” in FIG. 6) and the statement acquired by the statement data acquisition unit 12. A third display image P3 including information identifying the subject ("Mentioned by Y." in FIG. 6) is generated. Note that the positional relationship including the direction and distance of the extraction target object Te (positional relationship according to the second display mode) is not displayed. The display image generation unit 18A transmits the generated third display image P3 to the display image display device 31A of the user terminal 3A.

When the display image generation unit 18A completes the above-described processing, the display image generation device 1A ends the current processing and repeats the display image generation processing from step S101 again.

As described above, this embodiment discloses a display image generation device 1A that identifies an object T included in a statement uttered by a speaker as an extraction object Te, and generates a display image P regarding the extraction object Te. do. The display image generation device 1A includes a statement data acquisition section 12, a target object extraction section 13, a visual field image acquisition section 14A, a target object determination section 15A, and a display image generation section 18A. The utterance data acquisition unit 12 acquires utterance data of a utterance uttered to user X by user Y, who is the main speaker. The object extraction unit 13 stores a plurality of object data (character strings) in advance, compares the plurality of object data with the utterance data (character string) acquired by the utterance data acquisition unit 12, and extracts the utterance data. Among them, data that matches the target object data is extracted as the extracted target object Te. The visual field image acquisition unit 14A acquires an image that includes at least the visual field image of the user X. The target object determining unit 15A determines whether the extraction target Te extracted by the target object extracting unit 13 is included in the visual field image. The display image generation unit 18A acquires extraction target information that is information regarding the position of the extraction target Te, and generates a display image P that includes the extraction target information that is different from the visual field image. Further, the display image generation unit 18A determines the display mode of the display image P regarding the extraction target Te based on the determination result of the target object determination unit 15A as to whether the extraction target Te is included in the visual field image.

As a result, the display image generation device 1A can specify the extraction target Te recognized by a subject other than the user X (user Y) using the statement data acquisition unit 12 and the target object extraction unit 13. The display image generation device 1A can obtain a determination result as to whether the extraction target Te is included in the visual field image using the visual field image acquisition unit 14A and the target object determining unit 15A. Then, the display image generation unit 18A determines the display mode of the display image P regarding the extraction target Te based on the determination result of the target object determination unit 15A. As a result, the display image generation device 1A can determine the position of the extraction target Te, regardless of whether the extraction target Te recognized by a subject other than the user X is included in the user X's field of view Ex. Information can be appropriately generated (FIGS. 4 to 6).

Further, in the embodiment described above, the display image generation unit 18A uses a display mode that emphasizes the extraction target Te itself when the target object determination unit 15A determines that the extraction target Te is included in the visual field image. A first display image P1 showing extraction target object information is generated. As a result, if the target object determination unit 15A determines that the extraction target Te is included in the visual field image of the user X, the display image generation device 1A allows the user X to specify the extraction target Te. (S107 in FIG. 7).

Furthermore, in the embodiment described above, the display image generation device 1A includes a positional relationship acquisition unit 17A that acquires the relative positional relationship between the extraction target Te and the user X. When the object determination section 15A determines that the extraction object Te is not included in the visual field image, the display image generation section 18A generates a second display image showing the extraction object information in a display mode that displays the positional relationship. Generate P2. As a result, when the display image generation device 1A determines that the extraction target Te is not included in the visual field image by the target object determination unit 15A, the positional relationship acquisition unit 17A determines the relative relationship between the extraction target Te and the user X. Get the positional relationship. The display image generation device 1A generates a second display image P2 showing extraction target object information in a display mode that displays the acquired positional relationship. Thereby, the display image generation device 1A can appropriately generate information regarding the position of the extraction target Te even when the target T is not included in the user's X visual field Ex.

Furthermore, in the embodiment described above, the display image generation unit 18A determines whether or not the extraction target Te is included in the visual field image by the target object determining unit 15A. A display image P (first display image P1 to third display image P3) including information indicating whether or not it is visible is generated. As a result, the display image generation device 1A generates a display image P including information indicating whether or not the target object Te extracted from the visual field image is visible to the user X based on the determination result of the target object determination unit 15A. do. Thereby, the display image generation device 1A allows the user X to easily grasp information as to whether the extraction target Te is visible or not.

In the above-described embodiment, the display image generation device 1A is configured such that when the object determination unit 15A determines that the extraction object Te is not included in the visual field image, the display image generation device 1A selects a It includes an existence/absence determining section 16A that determines whether or not it exists within the range. The display image generation unit 18A determines the display mode of the extraction target information based on the determination result of whether the extraction target Te exists within the target range. As a result, the display image generation device 1A determines whether or not the extraction target Te is visible to the user A display image P (second display image P2, third display image P3) including information shown is generated. From this, the display image generation device 1A can appropriately generate information regarding the position of the extraction target Te, regardless of whether or not the extraction target Te exists in the target range.

Further, in the embodiment described above, the display image generation device 1A includes the peripheral image acquisition unit 11 that acquires peripheral images and stores the acquired peripheral images. The presence/absence determination unit 16A determines whether or not the extraction target Te exists within the target range based on the current or past peripheral images acquired by the peripheral image acquisition unit 11. As a result, the presence/absence determining unit 16A can determine in more detail whether or not the extraction target Te exists within the target range based on the acquired current or past surrounding images.

Furthermore, in the embodiment described above, the display image generation device 1A includes a positional relationship acquisition unit 17A that acquires the relative positional relationship between the extraction target Te and the user X. When the presence/absence determination unit 16A determines that the extraction target Te exists within the target range, the display image generation unit 18A displays the positional relationship including the direction and distance of the extraction target Te with the reference position as a reference. A display image P (second display image P2, third display image P3) showing extraction target object information in a display mode is generated. As a result, when the presence/absence determination unit 16A determines that the extraction target Te exists within the target range, the display image generation device 1A uses the positional relationship acquisition unit 17A to determine the position of the extraction target Te based on the reference position. Get direction and distance. Next, the display image generation device 1A can generate positional relationship information including the direction and distance of the position of the extraction target Te with reference to the reference position. Thereby, in the display image generation device 1A, when the presence/absence determining unit 16A determines that the extraction target Te exists within the target range, the user X can grasp the positional relationship of the extraction target Te.

Furthermore, in the embodiment described above, the person who makes the statement is a person (user Y), and the statement data is statement signal data of the statement. As a result, the display image generation device 1A can acquire speech signal data of a statement from a person who is a sender. As a result, the display image generation device 1A, even if the speaking subject is a person, regardless of whether or not the extraction target Te recognized by a subject other than the user X is included in the field of view of the user X. Information regarding the position of the extraction target Te can be appropriately generated.

Further, in the embodiment described above, the target object determination unit 15A outputs information on the determination result as to whether or not the extraction target object Te is included in the visual field image of the user X to the user Y who is the main speaker. As a result, the display image generation device causes the object determining unit 15 to output the determination result of whether or not the extracted object Te is included in the visual field image of the user X to the user Y who is the main speaker, so that the main speaker can Information on whether or not user X can visually recognize the target object can be acquired, and it is possible to decide how to proceed with the topic depending on whether or not user X can visually recognize the target object.

Furthermore, in the embodiment described above, the comment data acquisition unit 12 acquires information that identifies the user Y who made the comment to the user X. The display image generation unit 18A generates a display image P (first display image P1 to third display image P3) including information identifying the user Y acquired by the comment data acquisition unit 12. As a result, the display image generation device 1A can acquire information that specifies user Y using the comment data acquisition section 12, and can generate a display image P that includes information that specifies user Y using the display image generation section 18A. This allows user X to understand user Y.

Furthermore, in the embodiment described above, the display image generation device 1A includes a display image generation method in which the display image generation device 1A identifies the extraction target Te included in the utterance uttered by the speaker and generates the display image P regarding the extraction target Te. Disclose. The display image generation device 1A executes a statement data acquisition step, a target object extraction step, a visual field image acquisition step, a target object determination step, and a display image generation step. The utterance data acquisition step acquires utterance data of a utterance uttered to user X by user Y, who is the main speaker (S103 in FIG. 7). The target object extraction step compares a plurality of pre-stored target object data (character strings) with the acquired utterance data (character string), and extracts data that matches the target object data from among the utterance data. It is extracted as Te (S104 in FIG. 7). The visual field image acquisition step acquires the visual field image of the user ). The display image generation step acquires extraction target information that is information regarding the position of the extraction target Te, and generates a display image P that includes the extraction target information that is different from the visual field image (S107, S110 in FIG. 7, S112). Furthermore, in the display image generation step, the display mode of the display image P regarding the extraction target Te is determined based on the determination result of whether the extraction target Te is included in the visual field image in the target object determination step (see FIG. 7 S107, S110, S112).

As a result, the display image generation device 1A can specify the extraction target Te recognized by a subject other than the user X (user Y) through the statement data acquisition step and the target object extraction step. The display image generation device 1A can obtain a determination result as to whether or not the extraction target Te is included in the visual field image through the visual field image acquisition step and the target object determination step. Then, in the display image generation step, the display mode of the display image P regarding the extraction target Te is determined based on the determination result of the target object determination step. As a result, the display image generation device 1A can determine the position of the extraction target Te, regardless of whether the extraction target Te recognized by a subject other than the user X is included in the user X's field of view Ex. Information can be appropriately generated (FIGS. 4 to 6).
[Second embodiment]

FIG. 8 is a block diagram showing a display image generation device 1B according to the second embodiment. In this embodiment, a display image generation device 1B that can perform display image generation processing using POI (Point of Interest) information will be described. Here, "POI" means a specific place on a map, such as a store, facility, or interesting place, whose name and location information (latitude and longitude) are registered in the POI information storage unit 19. Furthermore, the statement ``There's a bicycle over there'' by user Y to user X, which is an example of the first embodiment, is uttered as an example in the second embodiment, ``There's a convenience store over there.'' ”. It is assumed that the object extracting unit 13 extracts the extraction object Te "convenience store" from the utterance uttered by the user Y. Note that in the second embodiment, descriptions similar to those in the first embodiment will be omitted or simplified.

In FIG. 8, the display image generation device 1B is different from the display image generation device 1A according to the first embodiment in that it does not include the peripheral image acquisition unit 11, and in place of the visual field image acquisition unit 14A, a visual field image acquisition unit 14B, a target object determination section 15B is provided in place of the target object determination section 15A, a presence/absence determination section 16B is provided in place of the presence/absence determination section 16A, and a positional relationship acquisition section 17A is provided in place of the positional relationship acquisition section 17A. They are different in that they include a positional relationship acquisition section 17B, a display image generation section 18B instead of the display image generation section 18A, and a POI information storage section 19. are otherwise identical.

The display image generation device 1B, the vehicle 2B, the user terminal 3B, and the speaker terminal 4 are connected to each other so that they can communicate (transmit and receive) by wire or wirelessly.

The vehicle 2B is different from the vehicle 2A according to the first embodiment in that it does not include the peripheral imaging device 22, and is otherwise the same.

The user terminal 3B is different from the user terminal 3A according to the first embodiment in that it includes a display image display device 31B instead of the display image display device 31A, and is otherwise the same. It is.

The speaker terminal 4 is the same as the speaker terminal 4 according to the first embodiment.

The POI information storage unit 19 stores POI information including at least information regarding the position of the target object T included in the map information. This "POI information" includes at least the name of the landmark that is the POI, the usage classification of the landmark, the characteristic information of the landmark, the image of the landmark, and the position information of the landmark. Note that landmarks include buildings, parks, commercial facilities, retail stores (convenience stores, etc.), and the like. The POI information storage unit 19 may acquire POI information from outside the vehicle 2B through communication, or may acquire landmark information stored in the navigation device 21 from the navigation device 21. The POI information storage unit 19 may update the POI information of the area where the vehicle 2B is located in real time according to the acquired position of the vehicle 2B. Furthermore, when the navigation device 21 performs a route search, the POI information storage unit 19 may acquire POI information on the route downloaded by the navigation device 21.

The visual field image acquisition section 14B is the same as the object determination section 15A according to the first embodiment.

The target object determining section 15B differs from the target object determining section 15A according to the first embodiment in the following points, but is otherwise the same. The target object determination unit 15B acquires POI information from the POI information storage unit 19, and determines whether the extraction target Te acquired by the target object extraction unit 13 is a POI. Then, it is determined whether the image of the extraction target Te is included in the visual field image acquired by the visual field image acquisition unit 14B. Note that when the extraction target Te is not a POI or when the extraction target Te is not included in the visual field image, the target object determination unit 15B determines that the image of the extraction target Te is not included in the visual field image. judge.

The presence/absence determination section 16B differs from the object determination section 15A according to the first embodiment in the following points (use of POI information), but is otherwise the same. The presence/absence determination unit 16B determines whether or not the extraction target Te exists within a preset target range when the target determination unit 15B determines that the extraction target Te is not included in the visual field image. do. Specifically, the presence/absence determination unit 16B determines whether or not the extraction target Te exists within the target range based on the POI information.

First, the presence/absence determination unit 16B determines whether or not the extraction target Te is included in the POI information acquired by the POI information storage unit 19. More specifically, the presence/absence determination unit 16B acquires the POI information acquired by the POI information storage unit 19, and determines whether the acquired POI information includes the extraction target Te acquired by the target object extraction unit 13. Determine whether or not.

Further, when it is determined that the extraction target Te is not included in the acquired POI information, the presence/absence determining unit 16B determines that the extraction target Te does not exist within the preset target range.

Next, when it is determined that the acquired POI information includes the extraction target Te (here, for example, a convenience store), the presence/absence determination unit 16B determines the extraction target Te (here, for example, a convenience store) based on the acquired POI information. It is determined whether the position where Te exists is within the target range. When the extraction target Te is included in the POI information, the presence/absence determining unit 16B determines whether the extraction target Te included in the position information of the vehicle 2B acquired from the navigation device 21 and the POI information stored in the POI information storage unit 19 The distance from the vehicle 2B to the extraction target Te is calculated using the position information. Furthermore, the presence/absence determination unit 16B determines whether or not the extraction target Te is within a preset target range based on the calculated distance.

The positional relationship acquisition unit 17B acquires the relative positional relationship between the extraction target Te and the user X. The positional relationship acquisition unit 17B acquires the position information of the extraction target Te from the POI information storage unit 19, acquires the position of the vehicle 2B from the navigation device 21, and combines the acquired position information of the extraction target Te with the position of the vehicle 2B. The direction and distance from the vehicle 2B to the extraction target Te may be calculated based on the position information.

The display image generation unit 18B acquires the extraction object information and generates a display image P including the extraction object information.

The display image generation unit 18B determines the display mode of the display image P of the extraction target Te based on the determination result of the target object determination unit 15B. Note that the meanings of terms such as "extraction target object information" are the same as in the first embodiment. Furthermore, "Bicycle" displayed in FIGS. 4 to 6 is referred to as "Convenience store" in the second embodiment. When the target object determination unit 15B determines that the extraction target Te is a POI and that the extraction target Te is included in the visual field image, the display image generation unit 18B generates the first display image P1. do. In this case, the display image generation unit 18B acquires the visual field image of the visual field Ex from the visual field image acquisition unit 14B, performs image recognition on the extraction target Te from the visual field image, and performs image recognition on the extraction target Te to be displayed superimposed on the extraction target Te. A first display image P1 is generated that shows the extraction target object information in a display mode that emphasizes the object Te itself (see FIG. 4). Note that if the target object determining unit 15B determines that the extraction target Te is not a POI, or if it is determined that the extraction target Te is not included in the visual field image, the first Display image P1 is not generated.

In addition, when the target object determination unit 15B determines that the extraction target Te is not included in the visual field image, the display image generation unit 18B determines that the extraction target Te is included in the POI information and that the extraction target Te is not included in the visual field image. The display mode of the extracted target object information is determined based on whether the existence determining unit 16B determines that the target object Te exists within the target range. More specifically, when the POI information includes the extraction target Te and the presence/absence determination unit 16B determines that the extraction target Te exists within the target range, the display image generation unit 18B performs the acquisition. A second display that shows the extraction object information in a display mode that displays the positional relationship of the extraction object Te including the direction and distance with respect to the reference position, based on the position information of the extraction object Te and the position information of the vehicle 2B. Generate image P2. The display image generation unit 18B acquires positional relationship information including the direction and distance of the extraction target Te to the reference position by the positional relationship acquisition unit 17B, and determines the position of the extracted target Te including the direction and distance to the reference position. A second display image P2 that displays the relationship is generated (see FIG. 5).

In addition, when the presence/absence determination unit 16B determines that the extraction target Te is not included in the POI information or that the extraction target Te does not exist within the target range, the display image generation unit 18B generates an extraction target Te. A third display image P3 is generated that indicates information that Te does not exist within a preset target range (see FIG. 6).

Further, the display image generation unit 18B may generate a display image P (first display image P1 to third display image P3) including information identifying the speaking subject acquired by the statement data acquisition unit 12 (Fig. 4 to Figure 6).

In addition, the display image generation unit 18B determines whether the extraction target Te is visible to the user A display image P (first display image P1 to third display image P3) including information indicating the information is generated (see FIGS. 4 to 6).

Next, the image generation process executed by the display image generation device 1B will be described. FIG. 9 is a flowchart showing display image generation processing. In the flowchart of FIG. 9, for example, the display image generation process by the display image generation device 1B is started when the vehicle 2B is started.

As shown in FIG. 9, in step S201, the POI information storage unit 19 acquires and stores POI information from the outside or the vehicle 2B. After that, the display image generation device 1B proceeds to step S202.

In step S202, the display image generation device 1B uses the utterance data acquisition unit 12 to acquire utterance data of the voice uttered by the user (the utterer) Y to the user X. The utterance data acquisition unit 12 acquires the utterance data of the voice uttered to the user X by the user Y, which is acquired from the utterance data acquisition device 41 of the utterance main terminal 4. Note that, as described above, the speech data includes data in which user Y has not uttered anything. Furthermore, the statement data acquisition unit 12 acquires information that identifies the user Y, and transmits it to the display image generation device 1B. After that, the process advances to step S203.

In step S203, the display image generation device 1B uses the statement data acquisition unit 12 to determine whether or not the statement data includes a statement by the user (the subject of the statement) Y. If it is determined that the statement by user Y is included, the process advances to step S204. If it is determined that the statement of user Y is not included, the process advances to the end.

In step S204, the display image generation device 1B determines whether the object extraction unit 13 can extract the extraction object Te that matches the object T from the statement data. If it is determined that the extraction target Te can be extracted, the process advances to step S205. If it is determined that the extraction target Te cannot be extracted, the process proceeds to the end.

In step S205, the display image generation device 1B acquires the visual field image of the user X using the visual field image acquisition unit 14B. The visual field image acquisition unit 14B acquires the visual field image of the user X from the visual field image acquisition device 32 of the user terminal 3B worn by the user X. After that, the process advances to step S206.

In step S206, the display image generation device 1B uses the target object determination unit 15B to determine whether the extraction target Te is a POI. Furthermore, the display image generation device 1B uses the object determination unit 15B to determine whether or not the extraction target Te extracted from the target object extraction unit 13 is included in the visual field image of the user X acquired from the visual field image acquisition unit 14B. judge. If it is determined that the extraction target Te is not a POI, or if it is determined that the extraction target Te is not included in the visual field image of the user X, the process advances to step S208. If it is determined that the extraction target Te is a POI and it is determined that the extraction target Te is included in the visual field image of the user X, the process advances to step S207. Here, for example, if the extraction target Te is a convenience store and the convenience store is stored as POI information in the POI information storage unit 19, it is determined that the extraction target Te is a POI. Further, for example, if the extraction target Te is a running bicycle, it is determined that the extraction target Te is not a POI because it is not stored in the POI information storage unit 19 as POI information.

If it is determined that the extraction target Te is a POI and it is determined that the extraction target Te is included in the visual field image of the user X, in step S207, the display image generation device 1B 18B, a first display image P1 that emphasizes the extraction target Te itself is generated. The display image generation unit 18B acquires the visual field image of the visual field Ex from the visual field image acquisition unit 14B, performs image recognition of the extraction target Te from the visual field image, and emphasizes the extraction target Te itself displayed superimposed on the visual field image. A first display image P1 showing extraction target object information in a first display mode is generated. Note that the display image generation unit 18B generates a first display that further includes information indicating that the extraction target Te from the visual field image is visible to the user X and information identifying the speaker acquired by the statement data acquisition unit 12. An image P1 may be generated. The display image generation unit 18B transmits the generated first display image P1 to the display image display device 31B of the user terminal 3B.

If it is determined that the extraction target Te is not a POI, or if it is determined that the extraction target Te is not included in the visual field image of the user X, in step S208, the display image generation device 1B first Based on the POI information stored in the POI information storage section 19, the presence/absence determination section 16B determines whether or not the extraction target Te is included in the POI information. Further, when it is determined that the extraction target Te is included in the POI information, the display image generation device 1B causes the presence/absence determination unit 16B to perform extraction based on the POI information stored in the POI information storage unit 19. It is determined whether the target object Te exists within the target range. If it is determined that the extraction target Te is not included in the POI information, or if it is determined that the extraction target Te does not exist within the target range, the process advances to step S211. If it is determined that the extraction target Te is included in the POI information and it is determined that the extraction target Te is present within the target range, the process advances to step S209. Here, for example, if the extraction target Te is a convenience store and the convenience store is stored in the POI information storage unit 19 as POI information, it is determined that the extraction target Te is included in the POI information. . Further, for example, if the extraction target Te is a running bicycle, it is not stored in the POI information storage unit 19 as POI information, and therefore it is determined that the extraction target Te is not included in the POI information.

If it is determined that the extraction target Te is included in the POI information, and if it is determined that the position where the extraction target Te exists is within the target range, in step S209, the display image generation device 1B , the positional relationship between the extraction target Te and the user X is acquired by the positional relationship acquisition unit 17B. The positional relationship acquisition unit 17B calculates and estimates the direction and distance from the extraction target Te to the user X or the vehicle 2B based on the acquired position information of the extraction target Te and the position information of the vehicle 2B. After that, the process advances to step S210.

In step S210, the display image generation device 1B uses the display image generation unit 18B to extract the extraction target from the vehicle 2B based on the position information of the extraction target Te acquired from the positional relationship acquisition unit 17B and the position information of the vehicle 2B. A second display image P2 is generated that displays the positional relationship including the direction and distance to Te. The display image generation unit 18B generates a second display image P2 that shows the extraction object information in a second display mode that displays the distance and a symbol image indicating the direction with respect to the visual field Ex of the user X acquired from the positional relationship acquisition unit 17B. generate. Note that the display image generation unit 18B generates a second display image P2 that includes information indicating that the extraction target Te is not visible to the user X and information identifying the speaker acquired by the statement data acquisition unit 12. May be generated. The display image generation unit 18B transmits the generated second display image P2 to the display image display device 31B of the user terminal 3B.

If it is determined that the extraction target Te is not included in the POI information, or if it is determined that the position where the extraction target Te exists is not within the target range, in step S211, the display image generation device 1B acquires the positional relationship between the extraction target Te and the user X by the positional relationship acquisition unit 17B. Specifically, the positional relationship acquisition unit 17B acquires positional relationship information in which the extraction target Te does not exist within a preset target range from the presence/absence determination unit 16B. After that, the process advances to step S212.

In step S212, the display image generation device 1B displays the positional relationship between the extraction target Te and the user X, which indicates that the extraction target Te acquired from the positional relationship acquisition unit 17B does not exist within the preset target range. A third display image P3 is generated. The display image generation unit 18B generates a third display image P3 that includes information indicating that the extracted object Te is not visible to the user X from the visual field image and information identifying the subject of the statement acquired by the statement data acquisition unit 12. generate. Note that the positional relationship including the direction and distance of the extraction target object Te (positional relationship according to the second display mode) is not displayed. The display image generation unit 18B transmits the generated third display image P3 to the display image display device 31B of the user terminal 3B.

When the display image generation unit 18B completes the above-described processing, the display image generation device 1B ends the current processing and repeats the display image generation processing from step S201 again.

As described above, this embodiment includes the POI information storage unit 19 that stores POI information including at least information regarding the position of the extraction target Te. The presence/absence determination unit 16B determines whether or not the extraction target Te exists within the target range based on the POI information stored by the POI information storage unit 19. As a result, the presence/absence determining unit 16B can reliably determine whether or not the extraction target Te exists within the target range based on the POI information stored in the POI information storage unit 19.
[Third embodiment]

FIG. 10 is a block diagram showing a display image generation device 1C according to the third embodiment. In this embodiment, a display image generation device 1C that can perform display image generation processing using a user terminal 3C, which is a display device installed in a vehicle 2C, will be described. Note that in the second embodiment, descriptions similar to those in the first embodiment will be omitted or simplified.

In FIG. 10, the display image generation device 1C is different from the display image generation device 1A according to the first embodiment in that it includes a visual field image acquisition unit 14C instead of the visual field image acquisition unit 14A, and a target object determination unit. 15A, a presence/absence judgment section 16C is provided instead of the presence/absence judgment section 16A, and a positional relationship acquisition section 17C is provided instead of the positional relationship acquisition section 17A. , is different in that it includes a display image generation section 18C instead of the display image generation section 18A, and that it includes a line of sight recognition section 20, and is the same in other respects.

The display image generation device 1C, the vehicle 2C, the user terminal 3C, and the speaker terminal 4 are connected to each other so that they can communicate (transmit and receive) by wire or wirelessly.

The vehicle 2C is different from the vehicle 2A according to the first embodiment in that it includes an attitude acquisition device 23, and is the same in other respects.

The user terminal 3B is different from the user terminal 3A according to the first embodiment in that it does not include the visual field image acquisition device 32 and includes a display image display device 31C in place of the display image display device 31A. They are different in some respects and are the same in other respects.

The speaker terminal 4 is the same as the speaker terminal 4 according to the first embodiment.

The posture acquisition device 23 acquires image information including the user's X face image. The posture acquisition device 23 captures an image including a face image of the user X from an in-vehicle camera installed in the vehicle 2C.

The line of sight recognition unit 20 recognizes the line of sight of the user X. The "line of sight" is a line of sight direction that passes through the center of both eyes of user X and indicates the direction of user's X's face. The line of sight recognition unit 20 acquires image information including the face image of the user X from the posture acquisition device 23, and recognizes the direction of the user's X line of sight.

The visual field image acquisition unit 14C acquires a visual field image based on the real-time peripheral image acquired by the peripheral image acquisition unit 11 and the line of sight of the user X recognized by the line of sight recognition unit 20. More specifically, the visual field image acquisition unit 14C acquires the visual line direction of the user X from the visual line recognition unit 20, and estimates the visual field Ex of the user X. The visual field image acquisition unit 14C acquires a real-time image around the vehicle from the peripheral image acquisition unit 11, cuts out an area corresponding to the visual field Ex of the user X estimated from the image around the vehicle, and acquires a visual field image. Here, the "area corresponding to the estimated visual field Ex of user area.

The target object determining unit 15C determines whether the extraction target Te is included in the visual field image of the visual field Ex of the user X acquired by the visual field image acquiring unit 14C. The target object determining section 15C may perform determination using the same method as the target object determining section 15A according to the first embodiment.

The presence/absence determining unit 16C determines whether the extraction target Te exists within a preset target range when the target object determining unit 15C determines that the extraction target Te is not included in the visual field image. do. The presence/absence determining section 16C may perform the determination using the same method as the presence/absence determining section 16A according to the first embodiment.

The positional relationship acquisition unit 17C acquires the relative positional relationship between the extraction target Te and the user X. The positional relationship acquisition unit 17C may estimate the direction and distance from the user X or the vehicle 2C to the extraction target Te using the same method as the positional relationship acquisition unit 17A according to the first embodiment. Further, the positional relationship acquisition unit 17C acquires information that the extraction target Te does not exist within a preset target range from the presence/absence determination unit 16C using the same method as the positional relationship acquisition unit 17A according to the first embodiment. You may.

The display image generation unit 18C acquires the extraction object information and generates a display image P including the extraction object information.

The display image generation unit 18C determines the display mode of the display image P of the extraction target Te, based on the determination result of the target object determination unit 15C, similarly to the first embodiment. Note that the meanings of terms such as "extraction target object information" are the same as in the first embodiment. When the target object determination unit 15C determines that the extraction target Te is included in the visual field image, the display image generation unit 18C acquires the visual field image of the visual field Ex from the visual field image acquisition unit 14C, and extracts it from the visual field image. Image recognition is performed on the target object Te, and a first display image P1 is generated that shows the extraction target information in a display mode that emphasizes the extraction target Te itself, which is displayed superimposed on the extraction target Te (see FIG. 4). .

Further, when the target object determining unit 15C determines that the extraction target Te is not included in the visual field image, the display image generating unit 18C determines that the extraction target Te exists within the target range by the presence/absence determining unit 16C. Based on whether the determination has been made or not, the display mode of the extraction target object information is determined in the same manner as in the first embodiment. More specifically, when the presence/absence determination unit 16C determines that the extraction target Te exists within the target range, the display image generation unit 18C determines the direction and distance of the extraction target Te with reference to the reference position. A second display image P2 is generated that shows the extraction target object information in a display mode that displays the positional relationship including the extracted object information (see FIG. 5). The display image generation unit 18C acquires positional relationship information including the direction and distance of the position of the extraction target Te with the reference position as a reference by the positional relationship acquisition unit 17C, and calculates the position of the extraction target Te with the acquired reference position as a reference. A second display image P2 is generated that displays the positional relationship including the direction and distance of the positions.

Furthermore, when the presence/absence determination unit 16C determines that the extraction target Te does not exist within the target range, the display image generation unit 18C indicates information that the extraction target Te does not exist within the preset target range. A third display image P3 is generated (see FIG. 6).

In addition, the display image generation unit 18C generates a display image P (first display image P1 to third display image P3) that includes information for specifying the speaking subject acquired by the statement data acquisition unit 12, as in the first embodiment. (See Figures 4 to 6).

In addition, the display image generation unit 18C determines whether the extraction target Te is visible to the user A display image P (first display image P1 to third display image P3) including information indicating the information is generated (see FIGS. 4 to 6).

Next, the image generation process executed by the display image generation device 1C will be described. FIG. 11 is a flowchart showing display image generation processing. In the flowchart of FIG. 11, for example, display image generation processing by the display image generation device 1C is started when the vehicle 2C is started.

As shown in FIG. 11, in step S301, the display image generation device 1C acquires a peripheral image of the user X using the peripheral image acquisition unit 11. The surrounding image acquisition unit 11 obtains a surrounding image captured by the surrounding imaging device 22 of the vehicle 2C. After that, the display image generation device 1C proceeds to step S302.

In step S302, the display image generation device 1C uses the utterance data acquisition unit 12 to acquire utterance data of the voice uttered by the user (the utterer) Y to the user X. The utterance data acquisition unit 12 acquires the utterance data of the voice uttered to the user X by the user Y, which is acquired from the utterance data acquisition device 41 of the utterance main terminal 4. Note that, as described above, the speech data includes data in which user Y has not uttered anything. Furthermore, the statement data acquisition unit 12 acquires information that identifies the user Y, and transmits it to the display image generation device 1C. After that, the process advances to step S303.

In step S303, the display image generation device 1C uses the statement data acquisition unit 12 to determine whether or not the statement data includes a statement by the user (the subject of the statement) Y. If it is determined that the statement by user Y is included, the process advances to step S304. If it is determined that the statement of user Y is not included, the process advances to the end.

In step S304, the display image generation device 1C determines whether the object extraction unit 13 can extract the extraction object Te that matches the object T from the statement data. If it is determined that the extraction target Te can be extracted, the process advances to step S305. If it is determined that the extraction target Te cannot be extracted, the process proceeds to the end.

In step S305, the display image generation device 1C recognizes the line of sight of the user X using the line of sight recognition unit 20. The line of sight recognition unit 20 acquires image information including the face image of the user X from the posture acquisition device 23, and recognizes the direction of the line of sight of the user X based on the acquired image information. After that, the process advances to step S306.

In step S306, the display image generation device 1C acquires the visual field image of the user X using the visual field image acquisition unit 14C. The visual field image acquisition unit 14C acquires the visual line direction of the user X from the visual line recognition unit 20, and estimates the visual field Ex of the user X. The visual field image acquisition unit 14C acquires a real-time image around the vehicle from the peripheral image acquisition unit 11, cuts out an area for the visual field Ex of the user X estimated from the image around the vehicle, and acquires a visual field image. After that, the process advances to step S307.

In step S307, the display image generation device 1C determines whether the extraction target Te extracted from the target object extraction unit 13 is included in the visual field image of the user X acquired from the visual field image acquisition unit 14C. Determine whether If it is determined that the extraction target Te is included in the visual field image of the user X, the process advances to step S308. If it is determined that the extraction target Te is not included in the visual field image of the user X, the process advances to step S309.

If it is determined that the extraction target Te is included in the visual field image of the user Generate image P1. The display image generation unit 18C acquires the visual field image of the visual field Ex from the visual field image acquisition unit 14C, performs image recognition of the extraction target Te from the visual field image, and emphasizes the extraction target Te itself displayed superimposed on the visual field image. A first display image P1 showing extraction target object information in a first display mode is generated. Note that the display image generation unit 18C generates a first display that further includes information indicating that the extracted object Te is visible by the user X from the visual field image and information identifying the subject of the statement acquired by the statement data acquisition unit 12. An image P1 may be generated. The display image generation unit 18C transmits the generated first display image P1 to the display image display device 31C of the user terminal 3C.

If it is determined that the extraction target Te is not included in the visual field image of the user X, in step S309, the display image generation device 1C uses the current or Based on past surrounding images, it is determined whether the extraction target Te exists within the target range. If the presence/absence determination unit 16C determines that the extraction target Te does not exist within the target range, the process proceeds to step S312. When the presence/absence determination unit 16C determines that the extraction target Te exists within the target range, the process proceeds to step S310.

If it is determined that the position where the extraction target Te exists is within the target range, in step S310, the display image generation device 1C uses the positional relationship acquisition unit 17C to determine the position of the extraction target Te and the user X. Get relationships. The positional relationship acquisition unit 17C obtains the distance from the extraction target Te to the user X or the vehicle 2C and the direction with respect to the visual field Ex of the user X, based on the current or past surrounding images of the user X acquired from the surrounding image acquisition unit 11. Estimate. Further, the positional relationship acquisition unit 17C may acquire the distance from the extraction target Te to the user X or the vehicle 2C from the presence/absence determination unit 16C. After that, the process advances to step S311.

In step S311, the display image generation device 1C causes the display image generation unit 18C to display a positional relationship including the direction and distance of the position of the extraction target Te using the reference position acquired from the positional relationship acquisition unit 17C as a reference. 2 display image P2 is generated. The display image generation unit 18C generates a second display image P2 that shows the extraction object information in a second display mode that displays the distance and a symbol image indicating the direction with respect to the visual field Ex of the user X acquired from the positional relationship acquisition unit 17C. generate. Note that the display image generation unit 18C generates a second display image P2 that includes information indicating that the extraction target Te is not visible to the user X and information identifying the subject of the statement acquired by the statement data acquisition unit 12. May be generated. The display image generation unit 18C transmits the generated second display image P2 to the display image display device 31C of the user terminal 3C.

If it is determined that the position where the extraction target Te exists is not within the target range, in step S312, the display image generation device 1C uses the positional relationship acquisition unit 17C to determine the position of the extraction target Te and the user X. Get relationships. Specifically, the positional relationship acquisition unit 17C acquires positional relationship information in which the extraction target Te does not exist within a preset target range from the presence/absence determination unit 16C. After that, the process advances to step S313.

In step S313, the display image generation device 1C displays the positional relationship between the extraction target Te and the user X, which indicates that the extraction target Te acquired from the positional relationship acquisition unit 17C does not exist within the preset target range. A third display image P3 is generated. The display image generation unit 18C generates a third display image P3 that includes information indicating that the extracted object Te is not visible to the user X from the visual field image and information identifying the subject of the statement acquired by the statement data acquisition unit 12. generate. Note that the positional relationship including the direction and distance of the extraction target object Te (positional relationship according to the second display mode) is not displayed. The display image generation unit 18C transmits the generated third display image P3 to the display image display device 31C of the user terminal 3C.

When the display image generation unit 18C completes the above-described processing, the display image generation device 1C ends the current processing and repeats the display image generation processing from step S301 again.

As described above, in this embodiment, the display image generation device 1C includes the peripheral image acquisition unit 11 that acquires and stores peripheral images, and the line-of-sight recognition unit 20 that recognizes the user's X line of sight. The visual field image acquisition unit 14C acquires a visual field image based on the current peripheral image acquired by the peripheral image acquisition unit 11 and the current line of sight of the user X recognized by the line of sight recognition unit 20. As a result, the display image generation device 1C uses the peripheral image acquisition unit 11 to acquire a peripheral image that is an image of the area including the visual field Ex of the user X, and the line of sight recognition unit 20 recognizes the line of sight of the user X. A visual field image corresponding to the line of sight of user X that includes peripheral images can be acquired. Thereby, even if the user terminal 3C does not have the visual field image acquisition device 32, the visual field image of the user X can be acquired by the visual line recognition unit 20.

The display image generation device and display image generation method of the present disclosure have been described above based on the above-mentioned embodiments, but the specific configuration is not limited to these embodiments, and the scope of the claims Changes and additions to the design are permitted as long as they do not depart from the gist of the invention claimed in each claim.

In each embodiment, an example has been shown in which both user X and user Y are riding in a vehicle, but the present invention is not limited to this. For example, one or both of the user X and the user Y who is the main speaker may exist (position) outside the vehicles 2A to 2C (that is, in a place separated from the vehicles 2A to 2C). In this case, the user terminal of user X or the server to which the user terminal of user Must have at least one. Note that the visual field image acquisition unit is included in, for example, a visual field image acquisition device included in the user terminal. Then, the peripheral image obtained by the peripheral imaging device may be a visual field image corresponding to the visual field Ex of the user X, or the peripheral image obtained by the peripheral imaging device may be transmitted to the user terminal if the user X has the peripheral imaging device. Further, when the user X is outside the vehicle, the posture acquisition device 23 is installed on the user terminal 3C or around the user X, and acquires the facial orientation information of the user X using the user X's facial image or sensor. The line-of-sight recognition unit 20 then recognizes the line-of-sight direction of the user X based on the face image or face orientation information of the user X acquired by the posture acquisition device 23. The visual field image acquisition unit generates a visual field image of the user X based on the peripheral image captured by the peripheral imaging device 22 and the visual line direction of the user X recognized by the visual line recognition unit 20. Note that when the user X is outside the vehicle, the line of sight recognition unit 20 is included in the user terminal of the user X or a server to which the user terminal of the user X can be connected. Even when the user X is outside the vehicle, the user terminal generates a display image P regarding the extraction target Te included in the statement uttered by the speaker. Then, the display image P is displayed on the display image display device.

In each embodiment, the object determination unit outputs information on the determination result as to whether or not the extraction target Te is included in the visual field image of the user X to the speaker terminal 4 of the user Y who is the speaker. shown, but is not limited to this. For example, as the information to be output to the user Y, a visual field image, a display image P, a peripheral image, etc. of the user X may be output. Furthermore, when user Y is particularly present outside the vehicle, an image is displayed on user Y's speaking terminal 4, VR (Virtual Reality, image display device), or the like. In this way, by displaying the image to user Y, who is the main speaker, user Y can obtain information about user It becomes easier to decide.

Further, the peripheral image captured by the peripheral imaging device 22 is not limited to those described in each of the above embodiments, and may be a visual field image corresponding to the visual field Ex of the user X, for example. Here, for example, if the user Y who is the main speaker is present outside the vehicle 2A, the terminal 4 for the main speaker displays part or all of the peripheral images captured by the peripheral imaging device 22. may be done. Thereby, it is possible to decide how the topic between user X and user Y will proceed.

Furthermore, although the display image display devices 31A to 31C of the user terminals 3A to 3C are shown as transmissive displays, they may be head-up displays installed in the vehicles 2A to 2C. For example, the head-up display is set at the lower part of the front window of the vehicles 2A to 2C, and displays an image on the windshield using a lamp. In this case, the image is a display image P generated by the display image generation units 18A to 18C that corresponds to the user's X's Ex field of view.

Furthermore, the speaking subject may be a speaking device that makes a comment to user X instead of a person. In the case of a comment device, the comment data is output sentence data. The output sentence data may be audio data in which the speaking device outputs an output sentence (character string) as voice, or may be an output sentence (character string). Therefore, the display image generation devices 1A to 1C can acquire output sentence data from the comment device that makes a comment to the user X using the comment data acquisition device. Furthermore, in this case, the "utterance uttered by the speaking subject" is the "voice uttered (outputted) by the speaking device." Further, the display image generation units 18A to 18C may obtain information specifying a speech device that emits a voice to the user X, and generate a display image P that reads, for example, "Mentioned by Speech output device." As a result, it is possible to display extraction target object information for the extraction target Te included in the utterance of the speaking device in an appropriate display manner for the user X. Specifically, the speaking device may be a so-called interactive agent device capable of voice interaction with user X.

Furthermore, in the above description, only one user Y or one speaking device was the subject of the statement, but there may be a plurality of subject subjects. For example, the subject of the speech may be two or more fellow passengers (users), or may be one fellow passenger (user) and one speech device. In this case, the utterance data acquisition unit 12 acquires information that identifies the utterer who made the utterance to user X. Next, the display image generation units 18A to 18C generate a display image P that includes information that specifies the person who made the statement, which was acquired by the statement data acquisition unit 12. As a result, the display image generation devices 1A to 1C acquire information specifying the speaker using the comment data acquisition unit 12, and generate display images P including information specifying the speaker using the display image generating units 18A to 18C. be able to. Thereby, when there are multiple targets of the speaker, the user X can clearly understand the speaker.

In the above example, the comment data acquisition section 12 and the comment data acquisition device 41 are provided. However, if the comment data acquisition section 12 has the function of the comment data acquisition device 41, the comment data acquisition device 41 can be omitted. You can. Further, although an example is shown in which the visual field image acquisition units 14A, 14B and the visual field image acquisition device 32 are provided, if the visual field image acquisition units 14A, 14B have the function of the visual field image acquisition device 32, the visual field image acquisition device 32 can be used. You don't have to prepare. Further, although an example is shown in which the line-of-sight recognition unit 20 and the posture acquisition device 23 are provided, the posture acquisition device 23 may not be provided as long as the line-of-sight recognition unit 20 has the function of the posture acquisition device 23. Furthermore, although an example is shown in which the peripheral image acquisition unit 11 and the peripheral imaging device 22 are provided, the peripheral imaging device 22 may not be provided as long as the peripheral image acquisition unit 11 has the function of the peripheral imaging device 22. .

In the second embodiment, the target object determination unit 15B determines whether the extraction target Te is a POI or not, and determines whether the image of the extraction target Te is included in the visual field image acquired by the visual field image acquisition unit 14B. Although an example of determining whether or not is shown has been shown, the present invention is not limited to this. For example, the target object determination unit may not determine whether the extraction target is a POI, but only determine whether the image of the extraction target is included in the visual field image acquired by the visual field image acquisition unit. good. In this case, even if the extraction target is not a POI, if it is determined that it is included in the visual field image, the first display image is generated.

1A, 1B, 1C Display image generation device 11 Surrounding image acquisition unit 12 Speech data acquisition unit 13 Object extraction unit 14A, 14B, 14C Visual field image acquisition unit 15A, 15B, 15C Object determination unit 16A, 16B, 16C Existence determination unit 17A, 17B, 17C Positional relationship acquisition unit 18A, 18B, 18C Display image generation unit 19 POI information storage unit 20 Line of sight recognition unit 2A, 2B, 2C Vehicle 21 Navigation device 22 Peripheral imaging device 23 Attitude acquisition device 3A, 3B, 3C User terminals 31A, 31B, 31C display image display device 32 visual field image acquisition device 4 speaker terminal 41 speech data acquisition device

Claims (14)

A display image generation device that identifies an object included in a statement uttered by a speaker as an extraction object, and generates a display image regarding the extraction object,
a utterance data acquisition unit that acquires utterance data of the utterance uttered to the user by the utterer;
A plurality of object data are stored in advance, and the plurality of object data and the statement data acquired by the statement data acquisition section are compared, and among the statement data, data that matches the target object data is selected from the statement data. an object extraction unit that extracts an object as an extraction object;
a visual field image acquisition unit that acquires an image including at least a visual field image corresponding to the visual field of the user;
a target object determination unit that determines whether the extraction target object extracted by the target object extraction unit is included in the visual field image;
a display image generation unit that acquires extraction target information that is information regarding the position of the extraction target and generates the display image that includes the extraction target information that is different from the visual field image;
The display image generation unit determines a display mode of the display image regarding the extraction target based on a determination result by the target object determination unit as to whether the extraction target is included in the visual field image, and determining the display mode that is different depending on whether or not the extraction target is included in the visual field image;
A display image generation device characterized by: The display image generation unit displays the extraction target information in the display mode that emphasizes the extraction target itself when the target object determining unit determines that the extraction target is included in the visual field image. The display image generation device according to claim 1, wherein the display image generation device generates the display image according to the present invention. comprising a positional relationship acquisition unit that acquires a relative positional relationship between the extraction target and the user;
The display image generation unit displays the extraction target information in the display mode that displays the positional relationship when the target object determination unit determines that the extraction target is not included in the visual field image. The display image generation device according to claim 1 or 2, wherein the display image generation device generates the display image. The display image generation unit is configured to make the extraction target visible to the user from the visual field image based on a determination result of the target object determination unit as to whether the extraction target is included in the visual field image. The display image generation device according to any one of claims 1 to 3, wherein the display image is generated including information indicating whether or not. an existence/nonexistence determination unit that determines whether or not the extraction target exists within a preset target range when the target object determination unit determines that the extraction target is not included in the visual field image; Prepare,
The display image generation unit determines the display mode of the extraction target information based on a determination result of the presence/absence determining unit as to whether or not the extraction target exists within the target range. The display image generation device according to any one of claims 1 to 4. comprising a peripheral image acquisition unit that acquires a peripheral image that is an area around the user including the visual field image, and stores the acquired peripheral image;
A claim characterized in that the presence/absence determination unit determines whether or not the extraction target exists within the target range based on the current or past peripheral images acquired by the peripheral image acquisition unit. Item 5. The display image generation device according to item 5. comprising a positional relationship acquisition unit that acquires a relative positional relationship between the extraction target and the user;
The display image generation unit is configured to use a reference position set at the user's position or a position in the vicinity of the user as a reference when the presence/absence determining unit determines that the extraction target exists within the target range. The display according to claim 5 or 6, wherein the display image is generated that shows the extraction target information in the display mode that displays the positional relationship including the direction and distance of the position of the extraction target. Image generation device. The extraction target is a POI (Points of Interest) that is a landmark associated with a position on a map,
comprising a POI information storage unit that stores POI information of the POI including at least information regarding the location of the extraction target;
Claims 5 to 7, wherein the presence/absence determination unit determines whether or not the extraction target exists within the target range based on the POI information stored by the POI information storage unit. The display image generation device according to any one of the preceding items. The subject of the statement is a person,
The display image generation device according to any one of claims 1 to 8, wherein the statement data is statement signal data of the statement made by a person to the user. The utterance subject is a utterance device that utters the utterance to the user,
The image generation device according to any one of claims 1 to 8, wherein the comment data is output sentence data indicating the content of an output sentence output as the comment. 11. The object determining unit outputs information on a determination result as to whether or not the extracted object is included in the visual field image to the speaker. The image generation device described. The utterance data acquisition unit acquires information specifying the utterance subject who uttered the utterance to the user,
The display image generating unit generates the display image including the information identifying the speaking subject acquired by the statement data acquisition unit, according to any one of claims 1 to 11. display image generation device. a peripheral image acquisition unit that acquires a peripheral image that is an area around the user including the visual field image, and stores the acquired peripheral image;
a line of sight recognition unit that recognizes the line of sight of the user;
The visual field image acquisition unit acquires the visual field image based on the current peripheral image acquired by the peripheral image acquisition unit and the current line of sight of the user recognized by the line of sight recognition unit. The display image generation device according to any one of claims 1 to 12. A display image generation method using a display image generation device that identifies a target object included in a statement uttered by a speaker as an extraction target object, and generates a display image related to the extraction target object, the method comprising:
a utterance data acquisition step of acquiring utterance data of utterances uttered to the user by the utterer;
a target object extraction step of comparing a plurality of pre-stored target object data and the acquired utterance data, and extracting data that matches the target object data from among the utterance data as the extraction target;
a visual field image acquisition step of acquiring a visual field image corresponding to the user's visual field;
a target object determination step of determining whether the extracted extraction target object is included in the visual field image;
a display image generation step of acquiring extraction target information that is information regarding the position of the extraction target and generating the display image including the extraction target information different from the visual field image;
In the display image generation step, a display mode of the display image regarding the extraction target is determined based on a determination result of whether the extraction target is included in the visual field image in the target object determination step , determining the display mode that is different depending on whether or not the extraction target is included in the visual field image;
A display image generation method characterized by: