JP2021081372A - Display image generator and display image generation method - Google Patents

Abstract

To appropriately generate information relating to the position of an extraction target recognized by the utterance of other than a user, irrespective of whether or not the extraction target is included in the visual field of the user.SOLUTION: A display image generator 1A comprises: an utterance data acquisition unit 12 for acquiring the utterance data of the statement uttered to a user by an uttering entity; a target extraction unit 13 for extracting data out of the utterance data that matches the target data as an extraction target; a visual field image acquisition unit 14A for acquiring the visual field image of the user; a target determination unit 15A for determining whether or not the extraction target that is the extracted target is included in the visual field image; and a display image generation unit 18A for acquiring extraction target information that is information relating to the position of the extraction target and generating a display image that includes the extraction target information. The display image generation unit 18A determines the display mode of the display image relating to the extraction target on the basis of the determination result of whether or not the extraction target is included in the visual field image.SELECTED DRAWING: Figure 1

Description

The present disclosure relates to a display image generation device and a display image generation method.

Techniques are known to generate information about the location of recognized out-of-vehicle objects. For example, Patent Document 1 discloses a technique of identifying an object outside the vehicle that the vehicle occupant is paying attention to by line-of-sight detection and voice recognition, and generating a display image showing the relative position of the identified object with respect to the vehicle. ..

Japanese Unexamined Patent Publication No. 2006-90790

However, the above-mentioned conventional technique presupposes that the user is looking in the direction in which the object exists, and the position of the object regardless of whether or not the object is included in the user's field of view. It cannot generate information about. Further, the above-mentioned conventional technique is to generate information on the position of the object recognized by the user himself / herself, and the user can generate information on the position of the object recognized by a subject other than the user. No consideration is given to producing for.

This disclosure is made in view of such circumstances, and regardless of whether or not the extraction target recognized by a subject other than the user is included in the user's field of view, the extraction target is concerned. It is an object of the present invention to provide a display image generation device and a display image generation method for appropriately generating information regarding the position of.

The display image generation device according to the present disclosure is a display image generation device that identifies an object included in a statement made by a speaking subject as an extraction target and generates a display image related to the extraction object. The display image generation device according to the present disclosure includes a speech data acquisition unit, an object extraction unit, a visual field image acquisition unit, an object determination unit, and a display image generation unit. The remark data acquisition unit acquires the remark data of the remark made to the user by the remark subject. The object extraction unit stores a plurality of object data in advance, compares the plurality of object data with the speech data acquired by the speech data acquisition unit, and selects data that matches the object data among the speech data. Extract as an extraction target. The field-of-view image acquisition unit acquires an image including at least a field-of-view image corresponding to the user's field of view. The object determination unit determines whether or not the extraction target extracted by the object extraction unit is included in the visual field image. The display image generation unit acquires the object information which is the information about the position of the extraction object, and generates the display image including the output object information different from the visual field image. The display image generation unit determines the display mode of the display image related to the extraction target based on the determination result of whether or not the extraction target is included in the visual field image by the object determination unit.

According to the present disclosure, it is possible to appropriately generate information on the position of an object recognized by a subject other than the user, regardless of whether or not the object is included in the user's field of view. ..

It is a block diagram which shows the display image generation apparatus which concerns on 1st Embodiment. It is a figure which shows the user and the passenger who are riding a vehicle with a terminal attached. It is a top view for demonstrating the user's field of view when viewed from above the vehicle. It is a figure which shows the peripheral situation corresponding to the field of view of the user X which the display image was superposed and displayed in the 1st display mode. It is a figure which shows the display image display apparatus which displayed the 1st display image. It is a figure which shows the peripheral situation corresponding to the field of view of the user X which the display image was superposed and displayed in the 2nd display mode. It is a figure which shows the display image display apparatus which displayed the 2nd display image. It is a figure which shows the peripheral situation corresponding to the field of view of the user X which the display image was superposed and displayed in the 3rd display mode. It is a figure which shows the display image display apparatus which displayed the 3rd display image. It is a flowchart which shows the display image generation processing which concerns on 1st Embodiment. It is a block diagram which shows the display image generation apparatus which concerns on 2nd Embodiment. It is a flowchart which shows the display image generation processing which concerns on 2nd Embodiment. It is a block diagram which shows the display image generation apparatus which concerns on 3rd Embodiment. It is a flowchart which shows the display image generation processing which concerns on 3rd Embodiment.

Hereinafter, exemplary embodiments of the present disclosure will be described with reference to the drawings. In the following description, the same or corresponding parts will be designated by the same reference numerals, and duplicate description will be omitted.
[First Embodiment]

FIG. 1 is a block diagram showing a display image generation device 1A according to the first embodiment. FIG. 2 is a diagram showing a user X and a user Y who are riding in a vehicle 2A with a terminal attached. FIG. 3 is a plan view for explaining the field of view Ex of the user X when viewed from above the vehicle 2A. 4A, 5A, and 6A are diagrams showing a peripheral situation corresponding to the visual field Ex of the user X in which the displayed images are superimposed and displayed in each display mode. 4B, 5B, and 6B are diagrams showing a display image display device on which each display image is displayed. As shown in FIGS. 1 to 6, the display image generator 1A extracts the object T included in the remarks made by the speaking subject (that is, referred to in the remarks made by the speaking subject). This is a device that identifies the object Te and generates a display image P related to the extraction target Te.

More specifically, the display image generation device 1A is a device that generates a display image P that is superimposed and displayed on the surrounding situation corresponding to the visual field Ex of the user X. The user X is in the vehicle 2A with the user (speaking subject) Y who is a person, and is visually recognizing the scenery outside the vehicle, for example. User X is wearing a user terminal 3A. The user Y is equipped with the speaking subject terminal 4 (see FIG. 2). In the present embodiment, the display image generation device 1A will be described by exemplifying a situation in which the user Y speaks to the user X.

Here, the "viewing field Ex of the user X" means a visible area that can be visually recognized by the user X. The "visible area" is an effective visual field in which information can be effectively obtained from the outside world when a human uses the eye and a fixed viewpoint (gaze point) is provided near the center of the physiological visual field. For example, the field of view Ex of the user X may be set in all the visible top, bottom, left, and right regions centered on the central axis of the field of view Ex of the user X. FIG. 3 shows a horizontally visible area of the user X when viewed from above the vehicle 2A. The field of view Ex of the user X changes with the movement of the vehicle 2A. For example, in FIG. 3, the current position of the user X is indicated by the current position X1, and the position of the user X after moving from the current location is indicated by the moving position X2. In the following description, the field of view Ex of the user X can be visually recognized by the user X through the transmissive display of the user terminal 3A in a state where the user X wearing the user terminal 3A described later is facing a predetermined direction. It is assumed that it is set in all areas of up, down, left and right. The visual field Eye of the user Y means a visible region that can be visually recognized by the user Y, similarly to the visual field Ex of the user X (see FIG. 2).

The "surrounding situation" means an actual scenery (outside view) outside the vehicle that is visible to the user X in the area around the user X. The surrounding situation is, for example, a landscape (outside view) of a region extending 360 degrees in the horizontal direction centered on the current position of the user X, including the upper and lower parts of the user X. The “surrounding situation corresponding to the field of view Ex of the user X” means the scenery (outside view) outside the vehicle included in the field of view Ex of the user X. In other words, the surrounding situation is the scenery outside the vehicle in the field of view Ex of the user X. As shown in FIG. 3, the movement of the vehicle 2A changes the peripheral situation corresponding to the visual field Ex of the user X.

"Generating a display image" means generating image information to be displayed on a display or the like. When the image information generated by the display image generation device 1A is transmitted to a display or the like by wired communication or wireless communication, the display image P related to the transmitted image information can be displayed on the display. The "display image P" is an image that displays information or the like regarding the extraction target object Te, and more specifically, is an image that displays information regarding the position of the object T. Here, the display image P is superimposed and displayed on the peripheral situation corresponding to the visual field Ex of the user X. The display image P may be, for example, an image including characters indicating whether or not the extraction target Te is included in the visual field image, and the specific extraction target Te included in the visual field image appears to be surrounded by a frame. It may be an image of a rectangular border displayed as follows. The details will be described later. Here, the "field of view image" is an image corresponding to the field of view Ex of the user X. That is, the visual field image is an image obtained by capturing the peripheral situation corresponding to the visual field Ex of the user X. In the present embodiment, the peripheral situation corresponding to the visual field Ex of the user X is the actual scenery outside the vehicle, and the visual field image is the peripheral situation corresponding to the visual field Ex of the user X as the imaging device (field image acquisition device 32). It is an image taken by.

The display image generation device 1A is configured as, for example, a server, and includes a processor (processing device), a memory (storage device), and the like.

The processor may be composed of, for example, a CPU (Central Processing Unit) or an MPU (Micro-Processing Unit). The memory may include at least one of a semiconductor storage device, a magnetic storage device, and an optical storage device. Further, the memory may include a register, a cache memory, a ROM (Read Only Memory) or a RAM (Random Access Memory) used as a main storage device, and the like.

The display image generator 1A, the vehicle 2A, the user terminal 3A, and the speaking subject terminal 4 are connected to each other so as to be able to communicate (transmit and receive) by wire or wirelessly. The functional configuration of the display image generation device 1A will be described later.

The vehicle 2A is a passenger car or the like on which the user X and the user Y are riding. The vehicle 2A may be capable of both manual driving and automatic driving by switching, or may be capable of driving only one of them. The vehicle 2A includes a navigation device 21 and a peripheral imaging device 22. The navigation device 21 sets the travel route of the vehicle 2A to the set destination based on the position information of the vehicle 2A detected by, for example, GPS (Global Positioning System) and the map information, and the travel thereof. It is a device that guides the vehicle 2A along the route. The navigation device 21 stores (holds) the history of the position of the vehicle 2A (for example, the position coordinates detected by GPS) in chronological order. The navigation device 21 may acquire the traveling direction of the vehicle 2A based on the stored history of the position of the vehicle 2A.

The peripheral image pickup device 22 is a device that captures the peripheral situation of the user X and acquires the peripheral image. The "peripheral image" is an image of a region around the user X including the visual field Ex (that is, the visual field image) of the user X and a region including the visual field Ex of the user X. The peripheral image of the user X may be, for example, an image in which a region extending 360 degrees in the horizontal direction around the user X is captured, or an image in which a region including the upper part of the user X is captured. .. Alternatively, the peripheral image of the user X is an area other than the area around the user X that is difficult to be visually recognized by the user X (for example, the area behind the user X in the state of being seated in the seat of the vehicle 2A). There may be. Alternatively, it may be the same region as the region corresponding to the field of view Ex of the user X. The range of the "region including the visual field Ex of the user X" is not particularly limited as long as it is the region including the visual field Ex of the user X.

The peripheral imaging device 22 is composed of, for example, one or a plurality of cameras. The camera of the peripheral imaging device 22 may be provided outside the vehicle interior, such as on the roof of the vehicle 2A, or may be installed inside the vehicle interior, such as behind the windshield. The vehicle 2A transmits the peripheral image of the user X captured by the peripheral image pickup device 22 to the display image generation device 1A. Note that "transmitting an image" means transmitting image data of an image.

The user terminal 3A is a device worn on the head of the user X, and includes a display image display device 31A and a visual field image acquisition device 32. The display image display device 31A has a display capable of displaying the display image P generated by the display image generation device 1A. The display of the display image display device 31A is a transmissive display such as a glasses type or goggles type, and is located immediately in front of the eyes of the user X with the user terminal 3A attached by the user X. Therefore, the user X can visually recognize the peripheral situation corresponding to the visual field Ex of the user X via the display image display device 31A. Further, when the display image P is displayed on the display image display device 31A, the display image P (see FIGS. 4B, 5B, and 6B) is superimposed on the peripheral situation corresponding to the visual field Ex of the user X when viewed from the user X. Will be displayed. That is, the display image display device 31A has a function as an HMD (Head Mounted Display) used in so-called AR (Augmented Reality) technology.

The field-of-view image acquisition device 32 is an image pickup device that acquires a field-of-view image by capturing the peripheral situation corresponding to the field-of-view Ex of the user X. The field image acquisition device 32 is provided on the user terminal 3A so that the user X can take an image of the line-of-sight direction of the user X while the user terminal 3A is attached. The field image acquisition device 32 is provided, for example, on the side of the display image display device 31A. The user terminal 3A transmits the field of view image captured by the field of view image acquisition device 32 to the display image generation device 1A. In addition, "transmitting the field of view image" means transmitting the image data of the field of view image. Further, the visual field image acquisition device 32 may include a sensor for detecting the line-of-sight direction of the user X (not shown), and may transmit information on the line-of-sight direction of the user X detected from the sensor together with the image data of the visual field image.

The speaking subject terminal 4 is a device worn on the head of the user Y, and includes a speaking data acquisition device 41. The speech data acquisition device 41 is a device that acquires the speech made by the user Y to the user X as speech data. The speech data acquisition device 41 is composed of, for example, a microphone. Here, in the speech data acquisition device 41, the speech subject terminal 4 is a headset, and the speech data acquisition device 41 is a microphone provided in the headset. The speech data acquisition device 41 may be an in-vehicle microphone or an earphone. Further, the speaking subject terminal 4 may further include a display image display device 31A and a visual field image acquisition device 32 similar to the user terminal 3A. The “speech data” is data having information about the content of the remark, and here, the remark data is the remark signal data of the remark. The “speech signal data” means a speech signal. It should be noted that the speech data also includes data in which the user Y does not emit anything.

The speech subject terminal 4 transmits the speech acquired by the speech data acquisition device 41 to the display image generation device 1A. At this time, the speaking subject terminal 4 further transmits information specifying that the speaking subject terminal 4 is worn by the user Y (information specifying the user Y) to the display image generation device 1A. The "information that identifies that the speaking subject terminal 4 is attached by the user Y" is information associated with the user Y, for example, the ID of the speaking subject terminal 4 associated with the user Y. It may be an (Identification) number. In addition, "transmitting a remark" means transmitting the remark signal data (details will be described later) of the remark.

Next, the functional configuration of the display image generation device 1A will be described. The display image generation device 1A includes a peripheral image acquisition unit 11, a speech data acquisition unit 12, an object extraction unit 13, a field image acquisition unit 14A, an object determination unit 15A, an existence / absence determination unit 16A, a positional relationship acquisition unit 17A, and a display. It has an image generation unit 18A.

The peripheral image acquisition unit 11 acquires and stores the peripheral image transmitted from the vehicle 2A. The peripheral image acquisition unit 11 acquires the peripheral image of the user X and stores it in chronological order. More specifically, the peripheral image acquisition unit 11 acquires the peripheral image of the user X by receiving the peripheral image of the user X captured by the peripheral image pickup device 22 of the vehicle 2A from the vehicle 2A. The peripheral image acquisition unit 11 stores the acquired peripheral image of the user X in chronological order. That is, the peripheral image acquisition unit 11 acquires the current peripheral image of the user X and stores (accumulates) the acquired peripheral image as a past peripheral image. The peripheral image acquisition unit 11 may delete the stored information on the past peripheral image at a preset timing.

The remark data acquisition unit 12 acquires the remark data of the remarks made by the user Y to the user X. More specifically, the speech data acquisition unit 12 receives the speech signal data of the user Y's speech acquired by the speech data acquisition device 41 of the speech subject terminal 4 from the speech subject terminal 4, so that the user Y Acquires the speech data of the speech issued to the user X by. The speech data acquisition unit 12 determines whether or not the speech data includes the speech of the user Y. That is, when the user Y does not speak, it is determined that the speech data does not include the speech of the user Y.

In addition, the speech data acquisition unit 12 acquires information that identifies the user Y who has made a speech to the user X. For example, the speech data acquisition unit 12 receives information identifying the user Y from the speech subject terminal 4.

The object extraction unit 13 extracts a character string representing a pre-stored object T included in the speech related to the speech data based on the speech data acquired by the speech data acquisition unit 12. More specifically, the object extraction unit 13 stores character strings (object data) representing a plurality of object Ts in advance, and converts the character strings representing the plurality of objects T and the speech data (object data). (A type of speech data) is compared, and a character string (data) that matches the character string representing the object T among the character strings converted from the speech data is extracted as the extraction target Te. The "object T" is an object that actually exists. The object may be, for example, the type of the object expressed by a general noun (bicycle, street light, building, etc.) or the name of the object expressed by a proper noun (Mt. Fuji, Parliament building, etc.). .. In addition, the object may be limited in terms of its attributes, characteristics, etc. (for example, a blue bicycle, the summit of Mt. Fuji, etc.). The object extraction unit 13 has a storage unit and stores in advance a general noun, a proper noun, an attribute, or a feature representing the object T. The object extraction unit 13 extracts a general noun, a proper noun, an attribute, or a feature representing the object T stored in advance from the speech data acquired by the speech data acquisition unit 12.

As an example, a case where the user Y makes a statement to the user X that "there is a bicycle over there" will be described. In this case, the object extraction unit 13 extracts the extraction target Te from the remarks made by the user Y based on the remark data of the remarks made by the user Y by the remark data acquisition unit 12. Here, it is assumed that the object extraction unit 13 stores in advance that the word "bicycle" represents the type of the object T (bicycle). The object extraction unit 13 extracts the extraction object Te called "bicycle" from the remarks made by the user Y. In some cases, the extraction target Te may not be extracted from the content of the user Y's remarks.

The object extraction unit 13 extracts, for example, by speech recognition (speech recognition), character strings representing a plurality of pre-stored objects T referred to in the speech relating to the speech data. Here, as "speech recognition", a known speech recognition technique can be applied. For example, the object extraction unit 13 recognizes the speech voice signal as a character string based on the speech data by the speech recognition, and extracts the extraction target Te from the recognized character string and the character string representing the plurality of objects T. To do.

The visual field image acquisition unit 14A acquires an image including at least a visual field image which is an image corresponding to the visual field Ex of the user X. The "image including at least the visual field image" may be an image having the same range as the visual field image, or may be an image having a wider range than the visual field image. The visual field image acquisition unit 14A acquires the visual field image by receiving the visual field image captured by the visual field image acquisition device 32 of the user terminal 3A from the visual field image acquisition device 32. Further, the visual field image acquisition unit 14A may acquire information on the line-of-sight direction of the user X from the visual field image acquisition device 32.

The object determination unit 15A determines whether or not the extraction target Te is included in the visual field image of the visual field Ex of the user X acquired by the visual field image acquisition unit 14A. As described above, the “extraction target object Te” is the one that matches the remark data from the plurality of object T stored in the object extraction unit 13. Here, the object extraction unit 13 extracts the extraction object Te called "bicycle".

The object determination unit 15A determines whether or not the extraction target Te is included in the visual field image, for example, by image recognition. Here, as "image recognition", a known image recognition technique can be applied. For example, the object determination unit 15A can detect identification information such as the name, type, shape, color, and direction of an object included in the image as image recognition, a machine learning model, a deep learning model, and OpenCV (Open Source Computer). An image processing algorithm using Vision Library) may be applied.

For example, the object determination unit 15A detects identification information of a plurality of objects included in the field image, and data (object type, etc.) representing the extraction target Te acquired by the object extraction unit 13 and the field of view. The detected identification information of a plurality of objects included in the image is compared with the detected identification information. After that, the object determination unit 15A determines the visual field image based on whether or not the data representing the extraction target Te and the plurality of objects included in the visual field image match in at least one of the type and the name. It is determined whether or not the extraction target Te is included in. Further, the object determination unit 15A uses OCR (Optical Character Recognition) as image recognition to recognize the character content of the signboard included in the field image, and the object extraction unit 13 has acquired the extracted object Te. Compared with the data expressing the name and the content of the recognized signboard, it is based on whether or not the name of the extraction target Te and a part of the content of at least one signboard in the signboard included in the field image match. It may be determined whether or not the field image includes the extraction target Te.

The object determination unit 15A outputs the information of the determination result as to whether or not the extraction object Te is included in the visual field image to the display image display device 31A of the user X. The “determination result of whether or not it is included in the visual field image” means the information of the determination result of whether or not the extraction target Te is visible to the user X (within the visual field Ex of the user X). Here, the object determination unit 15A outputs the determination result information to the user terminal 3A of the user X. It should be noted that the information of the determination result is also output to the speaking subject terminal 4 of the user Y who is the speaking subject.

The existence / non-existence determination unit 16A determines whether or not the extraction target Te exists within the preset target range when the object determination unit 15A determines that the extraction target Te is not included in the visual field image. To do. Specifically, the presence / absence determination unit 16A determines whether or not the extraction target Te exists within the target range based on the current or past peripheral image acquired by the peripheral image acquisition unit 11. The “target range” is a predetermined range set in advance around the position of the user X or the vehicle 2A. For example, the target range may be a predetermined range that can be visually recognized by the user X around the position of the user X or the vehicle 2A (the range shown by the chain double-dashed line in FIG. 3). The range may be a circular range, for example 50 kilometers, from User X or vehicle 2A, or may be a range of any shape other than circular. The target range may be a range in which the extraction target Te can be visually recognized by the user X according to the size of the extraction target Te. For example, if the extraction target Te is Mt. Fuji, the target range is the user X or the vehicle. It may be set in the range from the position (center) of 2A to 300 kilometers. In this example, the target range is a range with a radius of 300 kilometers from the center.

First, the presence / absence determination unit 16A determines whether or not the extraction target Te is included in the current peripheral image and the past peripheral image of the user X acquired by the peripheral image acquisition unit 11 and stored in time series. .. More specifically, the presence / absence determination unit 16A acquires the current peripheral image and the past peripheral image stored by the peripheral image acquisition unit 11, and extracts the acquired current peripheral image and the past peripheral image into the extracted object. Determine if Te is included. The existence / non-existence determination unit 16A may execute the determination by, for example, image recognition. The existence / non-existence determination unit 16A causes the object determination unit 15A to execute image recognition processing of the current peripheral image acquired and stored by the peripheral image acquisition unit 11 and a plurality of images included in the past peripheral image, and executes the processing. Based on the result, the determination may be executed.

Further, the presence / absence determination unit 16A detects various objects in a plurality of images included in the current peripheral image and the past peripheral image stored by the peripheral image acquisition unit 11, and the name, type, and shape of the object. , Color, direction, and other identification information is detected, and one or more image tags are assigned to the detected identification information to generate and store a tagged image. After that, the existence / non-existence determination unit 16A has speech data representing the extraction target object Te acquired by the object extraction unit 13, and a peripheral image that matches at least one of the names and types of the objects among the plurality of image tags. Based on whether or not, it is determined whether or not the acquired current peripheral image and the past peripheral image include the extraction target Te.

Further, when the presence / absence determination unit 16A determines that the acquired current peripheral image and the past peripheral image do not include the extraction target Te, the extraction target Te is within the preset target range. It is determined that it does not exist in.

Next, when the existence / non-existence determination unit 16A determines that the acquired current peripheral image and the past peripheral image include the extraction target Te, the position where the extraction target Te exists is within the target range. It is determined whether or not it is. When the extraction target Te is included in the current peripheral image, the existence / non-existence determination unit 16A can acquire the direction and distance from the user X or the vehicle 2A to the extraction target Te by a known method. For example, the presence / absence determination unit 16A estimates the direction and distance from the user X or the vehicle 2A to the extraction target Te based on the current peripheral image of the user X acquired by the peripheral image acquisition unit 11, and the extraction target object. It may be determined whether or not the position where Te exists is within the target range. Alternatively, the presence / absence determination unit 16A uses RADAR (Radio Detection and Ranging) or LIDAR (Light Detection and Ranging) provided in the vehicle 2A (not shown) from the user X or the vehicle 2A to the extraction target Te. The direction and distance may be measured to determine whether or not the position where the extraction target Te exists is within the target range.

When the extraction target Te is not included in the current peripheral image, the presence / absence determination unit 16A acquires the last peripheral image including the extraction target Te from the peripheral image acquisition unit 11 in chronological order. Next, the presence / absence determination unit 16A calculates the relative direction and distance between the current user X or the vehicle 2A and the user X or the vehicle 2A at the time of imaging based on the actual vehicle position history acquired from the navigation device 21. Next, the presence / absence determination unit 16A determines the direction and distance from the user X or the vehicle 2A to the extraction target Te based on the relative direction and distance thereof and the relative direction and distance from the user X or the vehicle 2A to the extraction target Te. Estimate the distance. Subsequently, the presence / absence determination unit 16A may determine whether or not the distance from the user X or the vehicle 2A to the extraction target Te is within the target range.

The positional relationship acquisition unit 17A acquires the relative positional relationship between the extraction target Te and the user X. The "positional relationship" may be represented by the direction and distance of the position of the extraction target Te with reference to the reference position (for example, the center position of the vehicle 2A) set at the position of the user X or the position near the user X. Alternatively, the information that the extraction target Te does not exist within the preset target range may be represented. The positional relationship acquisition unit 17A may acquire the direction and distance from the user X or the vehicle 2A to the extraction target Te from the existence / non-existence determination unit 16A. Further, the positional relationship acquisition unit 17A may estimate the direction and distance from the user X or the vehicle 2A to the extraction target Te based on the current or past peripheral image acquired by the peripheral image acquisition unit 11. Further, the positional relationship acquisition unit 17A may estimate the direction and distance from the user X or the vehicle 2A to the extraction target Te by a radar or a rider provided on the vehicle 2A. Further, the positional relationship acquisition unit 17A may acquire information from the existence / non-existence determination unit 16A that the extraction target Te does not exist within the preset target range.

The positional relationship acquisition unit 17A calculates the direction of the extraction target Te with respect to the line-of-sight direction of the user X. The positional relationship acquisition unit 17A may estimate the user's line-of-sight direction based on the visual field image of the user X acquired from the visual field image acquisition unit 14A and the peripheral image of the user X acquired from the peripheral image acquisition unit 11. Further, the positional relationship acquisition unit 17A may acquire the line-of-sight direction of the user X from the visual field image acquisition unit 14A. The positional relationship acquisition unit 17A estimates the extraction target Te direction with respect to the user X's line-of-sight direction based on the calculated direction from the user X or vehicle 2A to the extraction target Te and the user X's line-of-sight direction. Further, the direction of the extraction target Te with respect to the line-of-sight direction of the user X may be two types, left rear in the line-of-sight direction and right rear in the line-of-sight direction.

The display image generation unit 18A acquires the extraction target information and generates the display image P including the extraction target information. “Extraction target information” means information regarding the position of the extraction target Te. The extraction target information may be information indicating the position of the extraction target Te itself, information indicating the direction or distance in which the extraction target Te exists, and the extraction target Te may be within a predetermined area. It may be information indicating whether or not it exists.

The display image generation unit 18A determines the display mode of the display image P of the extraction target Te based on the determination result of the object determination unit 15A. The "display mode" is a display mode of an image showing information on an object to be extracted. The display mode may be an image showing the position of the extraction target Te itself, or an image showing the distance and direction of the extraction target Te as seen by the user, and the extraction target Te may be within a predetermined area. It may be an image showing whether or not it exists.

When the object determination unit 15A determines that the extraction target Te is included in the field image, the display image generation unit 18A acquires the field image of the field Ex from the field image acquisition unit 14A and extracts it from the field image. The image of the target object Te is recognized, and the first display image P1 showing the extraction target information is generated in a display mode that emphasizes the extraction target Te itself that is displayed superimposed on the extraction target Te. The "display mode that emphasizes the extraction target object itself" may be, for example, a display mode in which the extraction target object Te is surrounded by a square, a circle, or the like, and is a display mode in which the extraction target object Te is directly pointed by an arrow. It may be (see FIG. 4).

Further, when the display image generation unit 18A determines that the extraction target Te is not included in the visual field image by the object determination unit 15A, the presence / absence determination unit 16A determines that the extraction target Te is within the target range. The display mode of the extraction target information is determined based on whether or not the determination is made. More specifically, when the presence / absence determination unit 16A determines that the extraction target Te exists within the target range, the display image generation unit 18A determines the direction and distance of the position of the extraction target Te with reference to the reference position. A second display image P2 showing the extraction target information is generated in a display mode that displays the including positional relationship (see FIG. 5). The "display mode for displaying the positional relationship" is a display mode for an image showing the direction and distance of the position of the extraction target Te with reference to the reference position. The display image generation unit 18A acquires the positional relationship information including the direction and distance of the position of the extraction target Te with the reference position as a reference by the positional relationship acquisition unit 17A, and uses the acquired reference position as a reference for the extraction target Te. A second display image P2 that displays the positional relationship including the direction and distance of the position is generated. For example, when the extraction target Te is located to the rear left of the field of view Ex of the user X, as shown in FIG. 5, a symbol image indicating the rear left of the field of view Ex of the user X and an image indicating the distance are generated to generate a field of view. Display on the left of the image.

Further, the display image generation unit 18A indicates information that the extraction target Te does not exist in the preset target range when the existence / non-existence determination unit 16A determines that the extraction target Te does not exist in the target range. A third display image P3 is generated (see FIG. 6).

The display image generation unit 18A generates a display image P including information for identifying the speaking subject acquired by the speaking data acquisition unit 12. For example, the display image generation unit 18A generates the first display image P1 to the third display image P3 of "Mentioned by Y." when the speaker subject acquired by the speech data acquisition unit 12 is the user Y. It may be (see FIGS. 4 to 6).

The display image generation unit 18A indicates whether or not the extraction target Te is visible to the user X based on the determination result of whether or not the extraction target Te is included in the visual field image by the object determination unit 15A. A display image P including information is generated (see FIGS. 4 to 6). More specifically, the display image generation unit 18A indicates that the extraction target Te is visible to the user X when the object determination unit 15A determines that the extraction target Te is included in the field image. When the first display image P1 containing the information is generated and the object determination unit 15A determines that the extraction target Te is not included in the visual field image, it indicates that the extraction target Te is not visible to the user X. The second display image P2 and the third display image P3 including the information are generated. For example, the display image generation unit 18A generates a first display image P1 of "Bicycle is visible now." When the object determination unit 15A determines that the extraction target Te is included in the visual field image. It may be good (see FIG. 4). On the other hand, when the object determination unit 15A determines that the extraction object Te is not included in the visual field image, the display image generation unit 18A says "Bicycle is invisible now." The display image P3 may be generated (see FIGS. 5 and 6).

Subsequently, the image generation process executed by the display image generation device 1A will be described. FIG. 7 is a flowchart showing the display image generation process. In the flowchart of FIG. 7, for example, the display image generation process by the display image generation device 1A is started when the vehicle 2A is started.

As shown in FIG. 7, in step S101, the display image generation device 1A acquires the peripheral image of the user X by the peripheral image acquisition unit 11. The peripheral image acquisition unit 11 acquires the peripheral image captured by the peripheral image pickup device 22 of the vehicle 2A. After that, the display image generation device 1A proceeds to step S102.

In step S102, the display image generation device 1A acquires the speech data of the speech issued to the user X by the user (subject of speech) Y by the speech data acquisition unit 12. The utterance data acquisition unit 12 acquires the utterance data of the utterance made to the user X by the user Y acquired from the utterance data acquisition device 41 of the utterance subject terminal 4. As described above, the speech data also includes data in which the user Y does not emit anything. Further, the speech data acquisition unit 12 acquires information for identifying the passenger Y and transmits it to the display image generation device 1A. Then, the process proceeds to step S103.

In step S103, the display image generation device 1A determines whether or not the speech data includes the speech of the user (subject of speech) Y by the speech data acquisition unit 12. If it is determined that the user Y's remark is included, the process proceeds to step S104. If it is determined that the user Y's remark is not included, the process proceeds to the end.

In step S104, the display image generation device 1A determines whether or not the object extraction unit 13 can extract the extraction target Te that matches the object T from the statement data. If it is determined that the extraction target Te can be extracted, the process proceeds to step S105. If it is determined that the extraction target Te cannot be extracted, the process proceeds to the end.

In step S105, the display image generation device 1A acquires the visual field image of the user X by the visual field image acquisition unit 14A. The visual field image acquisition unit 14A acquires the visual field image of the user X from the visual field image acquisition device 32 of the user terminal 3A worn by the user X. Then, the process proceeds to step S106.

In step S106, in the display image generation device 1A, whether or not the extraction target Te extracted from the object extraction unit 13 by the object determination unit 15A is included in the field image of the user X acquired from the field image acquisition unit 14A. Is determined. If it is determined that the extraction target Te is included in the visual field image of the user X, the process proceeds to step S107. If it is determined that the extraction target Te is not included in the visual field image of the user X, the process proceeds to step S108.

When it is determined that the extraction target Te is included in the visual field image of the user X, in step S107, the display image generation device 1A uses the display image generation unit 18A to emphasize the extraction target Te itself. Image P1 is generated. The display image generation unit 18A acquires the visual field image of the visual field Ex from the visual field image acquisition unit 14A, recognizes the extraction target Te from the visual field image, and emphasizes the extraction target Te itself displayed by superimposing it on the visual field image. A first display image P1 showing information on an object to be extracted is generated in the first display mode (see FIG. 4). The display image generation unit 18A was acquired by the information indicating that the extraction target Te is visible to the user X from the visual field image (“Bicycle is visible now.” In FIG. 4) and the speech data acquisition unit 12. The first display image P1 may further include information for identifying the speaking subject (“Mentioned by Y.” in FIG. 4). The display image generation unit 18A transmits the generated first display image P1 to the display image display device 31A of the user terminal 3A.

When it is determined that the extraction target Te is not included in the visual field image of the user X, in step S108, the display image generation device 1A is currently or acquired by the peripheral image acquisition unit 11 by the presence / absence determination unit 16A. Based on the past peripheral image, it is determined whether or not the extraction target Te exists within the target range. When the existence / non-existence determination unit 16A determines that the extraction target Te does not exist within the target range, the existence / non-existence determination unit 16A proceeds to step S111. When the existence / non-existence determination unit 16A determines that the extraction target Te exists within the target range, the existence / non-existence determination unit 16A proceeds to step S109.

When it is determined that the position where the extraction target Te exists is within the target range, in step S109, the display image generation device 1A uses the positional relationship acquisition unit 17A to determine the position between the extraction target Te and the user X. Get a relationship. The positional relationship acquisition unit 17A is based on the peripheral image of the current or past user X acquired from the peripheral image acquisition unit 11, the distance from the extraction target Te to the user X or the vehicle 2A, and the direction of the user X with respect to the visual field Ex. To estimate. Further, the positional relationship acquisition unit 17A may acquire the distance from the extraction target Te to the user X or the vehicle 2A from the existence / non-existence determination unit 16A. Then, the process proceeds to step S110.

In step S110, the display image generation device 1A displays the positional relationship including the direction and distance of the position of the extraction target Te with reference to the reference position acquired from the positional relationship acquisition unit 17A by the display image generation unit 18A. 2 Display image P2 is generated. The display image generation unit 18A displays a symbol image (arrow in FIG. 5) and a distance (“20 m” in FIG. 5) indicating the direction of the user X with respect to the visual field Ex acquired from the positional relationship acquisition unit 17A. Generates the second display image P2 showing the extraction target information in. The display image generation unit 18A is acquired by the information indicating that the extraction target Te is invisible to the user X from the visual field image (“Bicycle is invisible now.” In FIG. 5) and the speech data acquisition unit 12. The second display image P2 including the information for identifying the speaking subject (“Mentioned by Y.” in FIG. 5) may be generated. The display image generation unit 18A transmits the generated second display image P2 to the display image display device 31A of the user terminal 3A.

When it is determined that the position where the extraction target Te exists is not within the target range, in step S111, the display image generation device 1A uses the positional relationship acquisition unit 17A to determine the position between the extraction target Te and the user X. Get a relationship. Specifically, the positional relationship acquisition unit 17A acquires the positional relationship information in which the extraction target Te does not exist within the preset target range from the existence / non-existence determination unit 16A. Then, the process proceeds to step S112.

In step S112, the display image generation device 1A displays the positional relationship between the extraction target Te and the user X that the extraction target Te acquired from the positional relationship acquisition unit 17A does not exist within the preset target range. The third display image P3 is generated. The display image generation unit 18A indicates that the extraction target Te is invisible to the user X from the visual field image (“Bicycle is invisible now.” In FIG. 6) and the remarks acquired by the remark data acquisition unit 12. A third display image P3 including information for identifying the subject (“Mentioned by Y.” in FIG. 6) is generated. The positional relationship including the direction and distance of the position of the extraction target Te (the positional relationship according to the second display mode) is not displayed. The display image generation unit 18A transmits the generated third display image P3 to the display image display device 31A of the user terminal 3A.

When the above-described processing of the display image generation unit 18A is completed, the display image generation device 1A ends the current processing and repeats the display image generation processing from step S101 again.

As described above, in the present embodiment, the display image generation device 1A that specifies the object T included in the remarks made by the speaking subject as the extraction target Te and generates the display image P related to the extraction target Te is disclosed. To do. The display image generation device 1A includes a speech data acquisition unit 12, an object extraction unit 13, a visual field image acquisition unit 14A, an object determination unit 15A, and a display image generation unit 18A. The remark data acquisition unit 12 acquires the remark data of the remark made to the user X by the user Y who is the remark subject. The object extraction unit 13 stores a plurality of object data (character strings) in advance, compares the plurality of object data with the speech data (character string) acquired by the speech data acquisition unit 12, and speaks data. Of these, the data that matches the object data is extracted as the extraction target Te. The visual field image acquisition unit 14A acquires an image including at least the visual field image of the user X. The object determination unit 15A determines whether or not the extraction target Te extracted by the object extraction unit 13 is included in the visual field image. The display image generation unit 18A acquires the extraction target information which is the information regarding the position of the extraction target Te, and generates the display image P including the extraction target information different from the visual field image. Further, the display image generation unit 18A determines the display mode of the display image P regarding the extraction target Te based on the determination result of whether or not the extraction target Te is included in the visual field image by the object determination unit 15A.

As a result, the display image generation device 1A can identify the extraction target Te recognized by the subject (user Y) other than the user X by the speech data acquisition unit 12 and the object extraction unit 13. The display image generation device 1A can obtain a determination result of whether or not the extraction target Te is included in the visual field image by the visual field image acquisition unit 14A and the object determination unit 15A. Then, the display image generation unit 18A determines the display mode of the display image P regarding the extraction target Te based on the determination result of the object determination unit 15A. As a result, the display image generation device 1A relates to the position of the extraction target Te recognized by a subject other than the user X regardless of whether or not the extraction target Te is included in the field of view Ex of the user X. Information can be appropriately generated (FIGS. 4 to 6).

Further, in the above-described embodiment, the display image generation unit 18A emphasizes the extraction target Te itself when the target determination unit 15A determines that the extraction target Te is included in the visual field image. The first display image P1 showing the extraction target information is generated. As a result, when the display image generation device 1A determines that the extraction target Te is included in the visual field image of the user X by the object determination unit 15A, the user X can specify the extraction target Te. (S107 in FIG. 7).

Further, in the above-described embodiment, the display image generation device 1A includes a positional relationship acquisition unit 17A that acquires a relative positional relationship between the extraction target Te and the user X. When the object determination unit 15A determines that the extraction object Te is not included in the visual field image, the display image generation unit 18A shows the extraction object information in a display mode for displaying the positional relationship. Generate P2. As a result, when the display image generation device 1A determines that the extraction target Te is not included in the visual field image by the object determination unit 15A, the positional relationship acquisition unit 17A determines that the extraction target Te and the user X are relative to each other. Get the positional relationship. The display image generation device 1A generates a second display image P2 showing the extraction target information in a display mode that displays the acquired positional relationship. As a result, the display image generation device 1A can appropriately generate information regarding the position of the extraction target Te even when the object T is not included in the field of view Ex of the user X.

Further, in the above-described embodiment, the display image generation unit 18A determines whether or not the extraction target Te is included in the visual field image by the object determination unit 15A, and the extraction target Te is determined by the user X. A display image P (first display image P1 to third display image P3) including information indicating whether or not the image is visible is generated. As a result, the display image generation device 1A generates a display image P including information indicating whether or not the extraction target Te is visible to the user X from the visual field image based on the determination result of the object determination unit 15A. To do. As a result, the display image generation device 1A can easily grasp the information as to whether or not the extraction target Te is visible to the user X.

Further, in the above-described embodiment, when the display image generation device 1A determines by the object determination unit 15A that the extraction target Te is not included in the visual field image, the extraction target Te is a preset target. The presence / absence determination unit 16A for determining whether or not it exists within the range is provided. The display image generation unit 18A determines the display mode of the extraction target information based on the determination result of whether or not the extraction target Te exists within the target range. As a result, the display image generation device 1A determines whether or not the extraction target Te is visible to the user X by determining the display mode of the extraction target information based on the determination result of the presence / absence determination unit 16A. A display image P (second display image P2, third display image P3) including the information to be shown is generated. As a result, the display image generation device 1A can appropriately generate information regarding the position of the extraction target Te regardless of whether or not the extraction target Te exists in the target range.

Further, in the above-described embodiment, the display image generation device 1A includes a peripheral image acquisition unit 11 that acquires a peripheral image and stores the acquired peripheral image. The existence / non-existence determination unit 16A determines whether or not the extraction target Te exists within the target range based on the current or past peripheral image acquired by the peripheral image acquisition unit 11. As a result, the presence / absence determination unit 16A can determine in more detail whether or not the extraction target Te exists within the target range based on the acquired current or past peripheral image.

Further, in the above-described embodiment, the display image generation device 1A includes a positional relationship acquisition unit 17A that acquires a relative positional relationship between the extraction target Te and the user X. When the presence / absence determination unit 16A determines that the extraction target Te exists within the target range, the display image generation unit 18A displays the positional relationship including the direction and distance of the position of the extraction target Te with reference to the reference position. A display image P (second display image P2, third display image P3) showing information on the object to be extracted is generated in the display mode. As a result, when the presence / absence determination unit 16A determines that the extraction target Te exists within the target range, the display image generation device 1A determines that the extraction target Te position is based on the reference position by the positional relationship acquisition unit 17A. Get direction and distance. Next, the display image generation device 1A can generate positional relationship information including the direction and distance of the position of the extraction target Te with reference to the reference position. As a result, when the presence / absence determination unit 16A determines that the extraction target Te exists within the target range, the display image generation device 1A can grasp the positional relationship of the extraction target Te.

Further, in the above-described embodiment, the speaking subject is a person (user Y), and the speaking data is the speaking signal data of the speaking. As a result, the display image generation device 1A can acquire the speech signal data of the speech from the transmitting subject who is a person. As a result, in the display image generation device 1A, even if the speaking subject is a person, regardless of whether or not the extraction target Te recognized by the subject other than the user X is included in the field of view of the user X. Information regarding the position of the extraction target Te can be appropriately generated.

Further, in the above-described embodiment, the object determination unit 15A outputs the information of the determination result as to whether or not the extraction object Te is included in the visual field image of the user X to the user Y who is the main speaker. As a result, the display image generation device outputs the determination result of whether or not the extraction target Te is included in the visual field image of the user X to the user Y who is the speaking subject by the object determining unit 15. Information on whether or not the user X can visually recognize the object can be acquired, and how the topic proceeds can be determined depending on whether or not the user X can visually recognize the object.

Further, in the above-described embodiment, the speech data acquisition unit 12 acquires information that identifies the user Y who has made a speech to the user X. The display image generation unit 18A generates a display image P (first display image P1 to third display image P3) including information for identifying the user Y acquired by the speech data acquisition unit 12. As a result, the display image generation device 1A can acquire the information that identifies the user Y by the speech data acquisition unit 12, and can generate the display image P that includes the information that identifies the user Y by the display image generation unit 18A. As a result, the user X can grasp the user Y.

Further, in the above-described embodiment, the display image generation device 1A is a display image generation method that identifies the extraction target Te included in the remarks made by the remark subject and generates the display image P related to the extraction target Te. To disclose. The display image generation device 1A executes a speech data acquisition step, an object extraction step, a visual field image acquisition step, an object determination step, and a display image generation step. The remark data acquisition step acquires the remark data of the remark made to the user X by the user Y who is the remark subject (S103 in FIG. 7). The object extraction step compares a plurality of object data (character strings) stored in advance with the acquired speech data (character string), and extracts data that matches the object data among the speech data. Extract as Te (S104 in FIG. 7). The visual field image acquisition step acquires the visual field image of the user X (S105 in FIG. 7), and the object determination step determines whether or not the extracted extracted object Te is included in the visual field image (S106 in FIG. 7). ). The display image generation step acquires the extraction target information which is the information regarding the position of the extraction target Te, and generates the display image P including the extraction target information different from the visual field image (S107, S110, FIG. 7). S112). Further, in the display image generation step, the display mode of the display image P relating to the extraction target Te is determined based on the determination result of whether or not the extraction target Te in the object determination step is included in the visual field image (FIG. 7 S107, S110, S112).

As a result, the display image generation device 1A can identify the extraction target Te recognized by the subject (user Y) other than the user X by the speech data acquisition step object extraction step. The display image generation device 1A can obtain a determination result of whether or not the extraction target object Te is included in the visual field image by the visual field image acquisition step and the object determination step. Then, in the display image generation step, the display mode of the display image P regarding the extraction target Te is determined based on the determination result of the object determination step. As a result, the display image generation device 1A relates to the position of the extraction target Te recognized by a subject other than the user X regardless of whether or not the extraction target Te is included in the field of view Ex of the user X. Information can be appropriately generated (FIGS. 4 to 6).
[Second Embodiment]

FIG. 8 is a block diagram showing a display image generation device 1B according to the second embodiment. In the present embodiment, the display image generation device 1B capable of executing the display image generation process using POI (Point of Interest) information will be described. Here, the "POI" means a specific place such as a store, a facility, or a famous place of interest on a map in which a name and location information (latitude / longitude) are registered in the POI information storage unit 19. Further, as an example of the first embodiment, the remark "There is a bicycle over there" made by the user Y to the user X, and as an example in the second embodiment, "There is a convenience store over there." ". Then, the object extraction unit 13 extracts the extraction object Te called "convenience store" from the remarks made by the user Y. In the second embodiment, the same description as in the first embodiment will be omitted or simplified.

In FIG. 8, the display image generation device 1B does not include the peripheral image acquisition unit 11 as compared with the display image generation device 1A according to the first embodiment, and the visual field image acquisition unit replaces the visual field image acquisition unit 14A. 14B is provided, an object determination unit 15B is provided instead of the object determination unit 15A, an existence / absence determination unit 16B is provided instead of the existence / absence determination unit 16A, and a positional relationship acquisition unit 17A is used instead. The difference is that the positional relationship acquisition unit 17B is provided, the display image generation unit 18B is provided instead of the display image generation unit 18A, and the POI information storage unit 19 is further provided. It is otherwise the same.

The display image generator 1B, the vehicle 2B, the user terminal 3B, and the speaking subject terminal 4 are connected to each other so as to be able to communicate (transmit and receive) by wire or wirelessly.

The vehicle 2B is different from the vehicle 2A according to the first embodiment in that it is not provided with the peripheral imaging device 22, and is the same in other respects.

The user terminal 3B is different from the user terminal 3A according to the first embodiment in that it includes a display image display device 31B instead of the display image display device 31A, and is the same in other respects. Is.

The speaking subject terminal 4 is the same as the speaking subject terminal 4 according to the first embodiment.

The POI information storage unit 19 stores POI information including at least information regarding the position of the object T included in the map information. This "POI information" includes at least the name of the landmark which is the POI, the usage classification of the landmark, the feature information of the landmark, the image of the landmark, and the location information of the landmark. Landmarks are buildings, parks, commercial facilities, retail stores (convenience stores, etc.), and the like. The POI information storage unit 19 may acquire POI information from the outside of the vehicle 2B by communication, or may acquire landmark information stored in the navigation device 21 from the navigation device 21. The POI information storage unit 19 may update the POI information of the area where the vehicle 2B is located in real time according to the acquired position of the vehicle 2B. Further, when the route search is performed by the navigation device 21, the POI information storage unit 19 may acquire the POI information on the route downloaded by the navigation device 21.

The field image acquisition unit 14B is the same as the object determination unit 15A according to the first embodiment.

The object determination unit 15B is different from the object determination unit 15A according to the first embodiment in the following points, but is the same except for the following points. The object determination unit 15B acquires POI information from the POI information storage unit 19, and determines whether or not the extraction object Te acquired by the object extraction unit 13 is a POI. Then, it is determined whether or not the image of the extraction target Te is included in the visual field image acquired by the visual field image acquisition unit 14B. If the extraction target Te is not a POI, or if the extraction target Te is not included in the visual field image, the object determination unit 15B determines that the image of the extraction target Te is not included in the visual field image. judge.

The existence / non-existence determination unit 16B is different from the object determination unit 15A according to the first embodiment in the following points (points of using POI information), but is the same in other respects. The existence / non-existence determination unit 16B determines whether or not the extraction target Te exists within the preset target range when the object determination unit 15B determines that the extraction target Te is not included in the visual field image. To do. Specifically, the presence / absence determination unit 16B determines whether or not the extraction target Te exists within the target range based on the POI information.

First, the presence / absence determination unit 16B determines whether or not the extraction target Te is included in the POI information acquired by the POI information storage unit 19. More specifically, the presence / absence determination unit 16B acquires the POI information acquired by the POI information storage unit 19, and whether the acquired POI information includes the extraction target Te acquired by the object extraction unit 13. Judge whether or not.

Further, when it is determined that the acquired POI information does not include the extraction target Te, the existence / non-existence determination unit 16B determines that the extraction target Te does not exist within the preset target range.

Next, the presence / absence determination unit 16B determines that the acquired POI information includes the extraction target Te (here, for example, a convenience store), and the extraction target is based on the acquired POI information. It is determined whether or not the position where Te exists is within the target range. When the existence / non-existence determination unit 16B includes the extraction target Te in the POI information, the extraction target Te included in the position information of the vehicle 2B acquired from the navigation device 21 and the POI information stored in the POI information storage unit 19. The distance from the vehicle 2B to the extraction target Te is calculated using the position information of. In addition, the presence / absence determination unit 16B determines whether or not the extraction target Te is within the preset target range based on the calculated distance.

The positional relationship acquisition unit 17B acquires the relative positional relationship between the extraction target Te and the user X. The positional relationship acquisition unit 17B acquires the position information of the extraction target Te by the POI information storage unit 19, acquires the position of the vehicle 2B from the navigation device 21, and obtains the position information of the acquired extraction target Te and the vehicle 2B. The direction and distance from the vehicle 2B to the extraction target Te may be calculated based on the position information.

The display image generation unit 18B acquires the extraction target information and generates the display image P including the extraction target information.

The display image generation unit 18B determines the display mode of the display image P of the extraction target Te based on the determination result of the object determination unit 15B. The meanings of terms such as "information on the object to be extracted" are the same as those in the first embodiment. Further, the "Bicycle" displayed in FIGS. 4 to 6 is referred to as a "Convenience store" in the second embodiment. When the object determination unit 15B determines that the extraction target Te is POI and the extraction target Te is included in the visual field image, the display image generation unit 18B generates the first display image P1. To do. In this case, the display image generation unit 18B acquires the visual field image of the visual field Ex from the visual field image acquisition unit 14B, recognizes the extraction target Te from the visual field image, and superimposes the extraction target Te on the extraction target Te to display the extraction target. A first display image P1 showing information on the object to be extracted is generated in a display mode that emphasizes the object Te itself (see FIG. 4). When the object determination unit 15B determines that the extraction object Te is not a POI, or when it is determined that the extraction object Te is not included in the visual field image, the first embodiment is performed. The display image P1 is not generated.

Further, when the display image generation unit 18B determines that the extraction target Te is not included in the visual field image by the object determination unit 15B, the POI information includes the extraction target Te and extracts the extraction target Te. The display mode of the extraction target information is determined based on whether or not the presence / absence determination unit 16B determines that the target object Te exists within the target range. More specifically, the display image generation unit 18B acquires the extraction target Te when the POI information includes the extraction target Te and the presence / absence determination unit 16B determines that the extraction target Te exists within the target range. A second display showing the extraction target information in a display mode in which the extraction target Te displays the positional relationship including the direction and the distance to the reference position based on the position information of the extraction target Te and the position information of the vehicle 2B. Image P2 is generated. The display image generation unit 18B acquires the positional relationship information including the direction and distance of the extraction target Te from the reference position by the positional relationship acquisition unit 17B, and the acquired position where the extraction target Te includes the direction and distance with respect to the reference position. A second display image P2 displaying the relationship is generated (see FIG. 5).

Further, the display image generation unit 18B determines that the extraction target object Te is not included in the POI information, or the existence / non-existence determination unit 16B determines that the extraction target object Te does not exist within the target range. A third display image P3 showing information that Te does not exist within the preset target range is generated (see FIG. 6).

Further, the display image generation unit 18B may generate a display image P (first display image P1 to third display image P3) including information for identifying the speaking subject acquired by the speech data acquisition unit 12 (FIG. 4 to 6).

Further, the display image generation unit 18B determines whether or not the extraction target object Te is visible to the user X based on the determination result of whether or not the extraction target object Te is included in the visual field image by the object determination unit 15B. Display images P (first display images P1 to third display images P3) including information indicating the above are generated (see FIGS. 4 to 6).

Subsequently, the image generation process executed by the display image generation device 1B will be described. FIG. 9 is a flowchart showing a display image generation process. In the flowchart of FIG. 9, for example, the display image generation process by the display image generation device 1B is started when the vehicle 2B is started.

As shown in FIG. 9, in step S201, the POI information storage unit 19 acquires and stores POI information from the outside or the vehicle 2B. After that, the display image generation device 1B proceeds to step S202.

In step S202, the display image generation device 1B acquires the speech data of the voice uttered to the user X by the user (subject of speech) Y by the speech data acquisition unit 12. The speech data acquisition unit 12 acquires speech data of the voice emitted to the user X by the user Y acquired from the speech data acquisition device 41 of the speech subject terminal 4. As described above, the speech data also includes data in which the user Y does not emit anything. Further, the speech data acquisition unit 12 acquires information that identifies the user Y and transmits it to the display image generation device 1B. Then, the process proceeds to step S203.

In step S203, the display image generation device 1B determines whether or not the speech data includes the speech of the user (subject of speech) Y by the speech data acquisition unit 12. If it is determined that the remark of the user Y is included, the process proceeds to step S204. If it is determined that the user Y's remark is not included, the process proceeds to the end.

In step S204, the display image generation device 1B determines whether or not the object extraction unit 13 can extract the extraction target Te that matches the object T from the statement data. If it is determined that the extraction target Te can be extracted, the process proceeds to step S205. If it is determined that the extraction target Te cannot be extracted, the process proceeds to the end.

In step S205, the display image generation device 1B acquires the visual field image of the user X by the visual field image acquisition unit 14B. The visual field image acquisition unit 14B acquires the visual field image of the user X from the visual field image acquisition device 32 of the user terminal 3B worn by the user X. Then, the process proceeds to step S206.

In step S206, the display image generation device 1B determines whether or not the extraction target object Te is a POI by the object determination unit 15B. Further, the display image generation device 1B determines whether or not the extraction target object Te extracted from the object extraction unit 13 by the object determination unit 15B is included in the visual field image of the user X acquired from the visual field image acquisition unit 14B. judge. If it is determined that the extraction target Te is not a POI, or if it is determined that the extraction target Te is not included in the visual field image of the user X, the process proceeds to step S208. If it is determined that the extraction target Te is POI and the extraction target Te is included in the visual field image of the user X, the process proceeds to step S207. Here, for example, if the extraction target Te is a convenience store and the convenience store is stored in the POI information storage unit 19 as POI information, it is determined that the extraction target Te is a POI. Further, for example, if the extraction target Te is a running bicycle, it is determined that the extraction target Te is not a POI because it is not stored in the POI information storage unit 19 as POI information.

When it is determined that the extraction target Te is POI and the extraction target Te is included in the visual field image of the user X, in step S207, the display image generation device 1B is the display image generation unit. The first display image P1 that emphasizes the extraction target Te itself is generated by 18B. The display image generation unit 18B acquires the visual field image of the visual field Ex from the visual field image acquisition unit 14B, recognizes the extraction target Te from the visual field image, and emphasizes the extraction target Te itself displayed by superimposing it on the visual field image. A first display image P1 showing information on an object to be extracted is generated in the first display mode. The display image generation unit 18B is the first display that further includes information indicating that the extraction target Te is visible to the user X from the visual field image and information for identifying the speaking subject acquired by the speaking data acquisition unit 12. Image P1 may be generated. The display image generation unit 18B transmits the generated first display image P1 to the display image display device 31B of the user terminal 3B.

When it is determined that the extraction target Te is not a POI, or when it is determined that the extraction target Te is not included in the visual field image of the user X, in step S208, first, the display image generation device 1B , The presence / absence determination unit 16B determines whether or not the extraction target Te is included in the POI information based on the POI information stored by the POI information storage unit 19. Further, when it is determined that the POI information includes the extraction target Te, the display image generation device 1B extracts the POI information by the presence / absence determination unit 16B based on the POI information stored by the POI information storage unit 19. It is determined whether or not the object Te exists within the target range. If it is determined that the extraction target Te is not included in the POI information, or if it is determined that the extraction target Te does not exist within the target range, the process proceeds to step S211. If it is determined that the extraction target Te is included in the POI information and it is determined that the extraction target Te exists within the target range, the process proceeds to step S209. Here, for example, if the extraction target Te is a convenience store and the convenience store is stored in the POI information storage unit 19 as POI information, it is determined that the extraction target Te is included in the POI information. .. Further, for example, if the extraction target Te is a running bicycle, it is determined that the extraction target Te is not included in the POI information because it is not stored in the POI information storage unit 19 as POI information.

When it is determined that the POI information includes the extraction target Te and the position where the extraction target Te exists is within the target range, in step S209, the display image generation device 1B , The positional relationship acquisition unit 17B acquires the positional relationship between the extraction target Te and the user X. The positional relationship acquisition unit 17B calculates the direction and distance from the extraction target Te to the user X or the vehicle 2B based on the acquired position information of the extraction target Te and the position information of the vehicle 2B. Then, the process proceeds to step S210.

In step S210, the display image generation device 1B is extracted from the vehicle 2B based on the position information of the extraction target Te acquired from the positional relationship acquisition unit 17B and the position information of the vehicle 2B by the display image generation unit 18B. A second display image P2 that displays the positional relationship including the direction and distance to Te is generated. The display image generation unit 18B displays a symbol image indicating the direction of the user X with respect to the field of view Ex acquired from the positional relationship acquisition unit 17B and a second display image P2 showing the extraction target information in the second display mode of displaying the distance. Generate. The display image generation unit 18B provides a second display image P2 including information indicating that the extraction target Te is invisible from the user X and information for identifying the speaking subject acquired by the speaking data acquisition unit 12. It may be generated. The display image generation unit 18B transmits the generated second display image P2 to the display image display device 31B of the user terminal 3B.

If it is determined that the POI information does not include the extraction target Te, or if it is determined that the position where the extraction target Te exists is not within the target range, in step S211, the display image generation device 1B Acquires the positional relationship between the extraction target Te and the user X by the positional relationship acquisition unit 17B. Specifically, the positional relationship acquisition unit 17B acquires the positional relationship information in which the extraction target Te does not exist within the preset target range from the existence / non-existence determination unit 16B. After that, the process proceeds to step S212.

In step S212, the display image generation device 1B displays the positional relationship between the extraction target Te and the user X that the extraction target Te acquired from the positional relationship acquisition unit 17B does not exist within the preset target range. The third display image P3 is generated. The display image generation unit 18B is a third display image P3 including information indicating that the extraction target Te is invisible to the user X from the visual field image and information for identifying the speaker subject acquired by the speech data acquisition unit 12. To generate. The positional relationship including the direction and distance of the position of the extraction target Te (the positional relationship according to the second display mode) is not displayed. The display image generation unit 18B transmits the generated third display image P3 to the display image display device 31B of the user terminal 3B.

When the above-described processing of the display image generation unit 18B is completed, the display image generation device 1B ends the current processing and repeats the display image generation processing from step S201 again.

As described above, the present embodiment includes a POI information storage unit 19 that stores POI information including at least information regarding the position of the extraction target Te. The existence / non-existence determination unit 16B determines whether or not the extraction target Te exists within the target range based on the POI information stored by the POI information storage unit 19. As a result, the presence / absence determination unit 16B can reliably determine whether or not the extraction target Te exists within the target range based on the POI information stored in the POI information storage unit 19.
[Third Embodiment]

FIG. 10 is a block diagram showing the display image generation device 1C according to the third embodiment. In the present embodiment, the display image generation device 1C capable of executing the display image generation process using the user terminal 3C, which is a display device installed in the vehicle 2C, will be described. In the second embodiment, the same description as in the first embodiment will be omitted or simplified.

In FIG. 10, the display image generation device 1C includes a field image acquisition unit 14C instead of the field image acquisition unit 14A as compared with the display image generation device 1A according to the first embodiment, that is, an object determination unit. An object determination unit 15C is provided instead of the 15A, an existence determination unit 16C is provided instead of the existence determination unit 16A, and a positional relationship acquisition unit 17C is provided instead of the positional relationship acquisition unit 17A. The difference is that the display image generation unit 18C is provided instead of the display image generation unit 18A, and the line-of-sight recognition unit 20 is provided, and they are the same in other respects.

The display image generator 1C, the vehicle 2C, the user terminal 3C, and the speaking subject terminal 4 are connected to each other so as to be able to communicate (transmit and receive) by wire or wirelessly.

The vehicle 2C is different from the vehicle 2A according to the first embodiment in that it is provided with the posture acquisition device 23, and is the same in other respects.

Compared with the user terminal 3A according to the first embodiment, the user terminal 3B does not include the visual field image acquisition device 32 and includes the display image display device 31C instead of the display image display device 31A. It is different in, and is the same in other respects.

The speaking subject terminal 4 is the same as the speaking subject terminal 4 according to the first embodiment.

The posture acquisition device 23 acquires image information including the face image of the user X. The posture acquisition device 23 captures an image including the face image of the user X from the in-vehicle camera installed in the vehicle 2C.

The line-of-sight recognition unit 20 recognizes the line of sight of the user X. The "line of sight" is a line-of-sight direction that passes through the centers of both eyes of the user X and indicates the face orientation of the user X. The line-of-sight recognition unit 20 acquires image information including the face image of the user X from the posture acquisition device 23, and recognizes the line-of-sight direction of the user X.

The visual field image acquisition unit 14C acquires a visual field image based on the real-time peripheral image acquired by the peripheral image acquisition unit 11 and the line of sight of the user X recognized by the line of sight recognition unit 20. More specifically, the field image acquisition unit 14C acquires the line-of-sight direction of the user X from the line-of-sight recognition unit 20 and estimates the field-of-view Ex of the user X. The visual field image acquisition unit 14C acquires a real-time image of the vehicle peripheral area from the peripheral image acquisition unit 11, cuts out a region corresponding to the user X's visual field Ex estimated from the image of the vehicle peripheral area, and acquires the visual field image. Here, the "region corresponding to the estimated user X's field of view Ex" is, for example, 60 degrees above and 70 degrees below the vertical field of view, and 100 degrees to the left and right in the horizontal field of view, respectively, without moving the eyes. Let it be an area.

The object determination unit 15C determines whether or not the extraction target Te is included in the visual field image of the visual field Ex of the user X acquired by the visual field image acquisition unit 14C. The object determination unit 15C may determine by the same method as the object determination unit 15A according to the first embodiment.

The existence / non-existence determination unit 16C determines whether or not the extraction target Te exists within the preset target range when the object determination unit 15C determines that the extraction target Te is not included in the visual field image. To do. The presence / absence determination unit 16C may determine the presence / absence determination unit 16C by the same method as the presence / absence determination unit 16A according to the first embodiment.

The positional relationship acquisition unit 17C acquires the relative positional relationship between the extraction target Te and the user X. The positional relationship acquisition unit 17C may estimate the direction and distance from the user X or the vehicle 2C to the extraction target Te by the same method as the positional relationship acquisition unit 17A according to the first embodiment. Further, the positional relationship acquisition unit 17C acquires information from the existence / non-existence determination unit 16C that the extraction target Te does not exist within the preset target range by the same method as the positional relationship acquisition unit 17A according to the first embodiment. You may.

The display image generation unit 18C acquires the extraction target information and generates the display image P including the extraction target information.

The display image generation unit 18C determines the display mode of the display image P of the extraction target Te as in the first embodiment, based on the determination result of the object determination unit 15C. The meanings of terms such as "information on the object to be extracted" are the same as those in the first embodiment. When the object determination unit 15C determines that the extraction target Te is included in the field image, the display image generation unit 18C acquires the field image of the field Ex from the field image acquisition unit 14C and extracts it from the field image. The object Te is image-recognized, and the first display image P1 showing the extraction target information is generated in a display mode that emphasizes the extraction target Te itself displayed by superimposing it on the extraction target Te (see FIG. 4). ..

Further, when the display image generation unit 18C determines that the extraction target Te is not included in the visual field image by the object determination unit 15C, the presence / absence determination unit 16C determines that the extraction target Te exists within the target range. Based on whether or not it is determined, the display mode of the extraction target information is determined as in the first embodiment. More specifically, when the presence / absence determination unit 16C determines that the extraction target Te exists within the target range, the display image generation unit 18C determines the direction and distance of the position of the extraction target Te with reference to the reference position. A second display image P2 showing the extraction target information is generated in a display mode that displays the including positional relationship (see FIG. 5). The display image generation unit 18C acquires the positional relationship information including the direction and distance of the position of the extraction target Te with the reference position as a reference by the positional relationship acquisition unit 17C, and uses the acquired reference position as a reference for the extraction target Te. A second display image P2 that displays the positional relationship including the direction and distance of the position is generated.

Further, the display image generation unit 18C indicates information that the extraction target Te does not exist in the preset target range when the existence / non-existence determination unit 16C determines that the extraction target Te does not exist in the target range. A third display image P3 is generated (see FIG. 6).

Further, the display image generation unit 18C displays the display image P (first display image P1 to third display image P3) including the information for identifying the speaker acquired by the speech data acquisition unit 12 as in the first embodiment. Generate (see FIGS. 4 to 6).

Further, the display image generation unit 18C determines whether or not the extraction target object Te is visible to the user X based on the determination result of whether or not the extraction target object Te is included in the visual field image by the object determination unit 15C. Display images P (first display images P1 to third display images P3) including information indicating the above are generated (see FIGS. 4 to 6).

Subsequently, the image generation process executed by the display image generation device 1C will be described. FIG. 11 is a flowchart showing the display image generation process. In the flowchart of FIG. 11, for example, the display image generation process by the display image generation device 1C is started when the vehicle 2C is started.

As shown in FIG. 11, in step S301, the display image generation device 1C acquires the peripheral image of the user X by the peripheral image acquisition unit 11. The peripheral image acquisition unit 11 acquires the peripheral image captured by the peripheral image pickup device 22 of the vehicle 2C. After that, the display image generation device 1C proceeds to step S302.

In step S302, the display image generation device 1C acquires the speech data of the voice uttered to the user X by the user (subject of speech) Y by the speech data acquisition unit 12. The speech data acquisition unit 12 acquires speech data of the voice emitted to the user X by the user Y acquired from the speech data acquisition device 41 of the speech subject terminal 4. As described above, the speech data also includes data in which the user Y does not emit anything. Further, the speech data acquisition unit 12 acquires information that identifies the user Y and transmits it to the display image generation device 1C. Then, the process proceeds to step S303.

In step S303, the display image generation device 1C determines whether or not the speech data includes the speech of the user (subject of speech) Y by the speech data acquisition unit 12. If it is determined that the remark of the user Y is included, the process proceeds to step S304. If it is determined that the user Y's remark is not included, the process proceeds to the end.

In step S304, the display image generation device 1C determines whether or not the object extraction unit 13 can extract the extraction target Te that matches the object T from the statement data. If it is determined that the extraction target Te can be extracted, the process proceeds to step S305. If it is determined that the extraction target Te cannot be extracted, the process proceeds to the end.

In step S305, the display image generation device 1C recognizes the line of sight of the user X by the line of sight recognition unit 20. The line-of-sight recognition unit 20 acquires image information including the face image of the user X from the posture acquisition device 23, and recognizes the line-of-sight direction of the user X based on the acquired image information. Then, the process proceeds to step S306.

In step S306, the display image generation device 1C acquires the visual field image of the user X by the visual field image acquisition unit 14C. The field image acquisition unit 14C acquires the line-of-sight direction of the user X from the line-of-sight recognition unit 20 and estimates the field-of-view Ex of the user X. The visual field image acquisition unit 14C acquires a real-time image of the vehicle peripheral area from the peripheral image acquisition unit 11, cuts out a region with respect to the visual field Ex of the user X estimated from the image of the vehicle peripheral area, and acquires the visual field image. Then, the process proceeds to step S307.

In step S307, in the display image generation device 1C, whether or not the extraction target Te extracted from the object extraction unit 13 by the object determination unit 15C is included in the field image of the user X acquired from the field image acquisition unit 14C. Is determined. If it is determined that the extraction target Te is included in the visual field image of the user X, the process proceeds to step S308. If it is determined that the extraction target Te is not included in the visual field image of the user X, the process proceeds to step S309.

When it is determined that the extraction target Te is included in the visual field image of the user X, in step S308, the display image generation device 1C uses the display image generation unit 18C to emphasize the extraction target Te itself. Image P1 is generated. The display image generation unit 18C acquires the visual field image of the visual field Ex from the visual field image acquisition unit 14C, recognizes the extraction target Te from the visual field image, and emphasizes the extraction target Te itself displayed by superimposing it on the visual field image. A first display image P1 showing information on an object to be extracted is generated in the first display mode. The display image generation unit 18C is the first display that further includes information indicating that the extraction target Te is visible to the user X from the visual field image and information for identifying the speaking subject acquired by the speaking data acquisition unit 12. Image P1 may be generated. The display image generation unit 18C transmits the generated first display image P1 to the display image display device 31C of the user terminal 3C.

When it is determined that the extraction target Te is not included in the visual field image of the user X, in step S309, the display image generation device 1C is currently or acquired by the peripheral image acquisition unit 11 by the presence / absence determination unit 16C. Based on the past peripheral image, it is determined whether or not the extraction target Te exists within the target range. When the existence / non-existence determination unit 16C determines that the extraction target Te does not exist within the target range, the existence / non-existence determination unit 16C proceeds to step S312. When the existence / non-existence determination unit 16C determines that the extraction target Te exists within the target range, the existence / non-existence determination unit 16C proceeds to step S310.

When it is determined that the position where the extraction target Te exists is within the target range, in step S310, the display image generation device 1C uses the positional relationship acquisition unit 17C to position the extraction target Te and the user X. Get a relationship. The positional relationship acquisition unit 17C is based on the peripheral image of the current or past user X acquired from the peripheral image acquisition unit 11, the distance from the extraction target Te to the user X or the vehicle 2C, and the direction of the user X with respect to the visual field Ex. To estimate. Further, the positional relationship acquisition unit 17C may acquire the distance from the extraction target Te to the user X or the vehicle 2C from the existence / non-existence determination unit 16C. Then, the process proceeds to step S311.

In step S311, the display image generation device 1C displays the positional relationship including the direction and distance of the position of the extraction target Te with reference to the reference position acquired from the positional relationship acquisition unit 17C by the display image generation unit 18C. 2 Display image P2 is generated. The display image generation unit 18C displays a symbol image indicating the direction of the user X with respect to the field of view Ex acquired from the positional relationship acquisition unit 17C and a second display image P2 showing the extraction target information in the second display mode of displaying the distance. Generate. The display image generation unit 18C provides a second display image P2 including information indicating that the extraction target Te is invisible from the user X and information for identifying the speaking subject acquired by the speaking data acquisition unit 12. It may be generated. The display image generation unit 18C transmits the generated second display image P2 to the display image display device 31C of the user terminal 3C.

When it is determined that the position where the extraction target Te exists is not within the target range, in step S312, the display image generation device 1C uses the positional relationship acquisition unit 17C to determine the position between the extraction target Te and the user X. Get a relationship. Specifically, the positional relationship acquisition unit 17C acquires the positional relationship information in which the extraction target Te does not exist within the preset target range from the existence / non-existence determination unit 16C. Then, the process proceeds to step S313.

In step S313, the display image generation device 1C displays the positional relationship between the extraction target Te and the user X that the extraction target Te acquired from the positional relationship acquisition unit 17C does not exist within the preset target range. The third display image P3 is generated. The display image generation unit 18C includes a third display image P3 including information indicating that the extraction target Te is invisible to the user X from the visual field image and information for identifying the speaking subject acquired by the speaking data acquisition unit 12. To generate. The positional relationship including the direction and distance of the position of the extraction target Te (the positional relationship according to the second display mode) is not displayed. The display image generation unit 18C transmits the generated third display image P3 to the display image display device 31C of the user terminal 3C.

When the above-described processing of the display image generation unit 18C is completed, the display image generation device 1C ends the current processing and repeats the display image generation processing from step S301 again.

As described above, in the present embodiment, the display image generation device 1C includes a peripheral image acquisition unit 11 that acquires and stores a peripheral image, and a line-of-sight recognition unit 20 that recognizes the line of sight of the user X. The field-of-view image acquisition unit 14C acquires a field-of-view image based on the current peripheral image acquired by the peripheral image acquisition unit 11 and the current line-of-sight of the user X recognized by the line-of-sight recognition unit 20. As a result, the display image generation device 1C acquires the peripheral image which is an image of the region including the field of view Ex of the user X by the peripheral image acquisition unit 11, recognizes the line of sight of the user X by the line of sight recognition unit 20, and acquires the peripheral image. It is possible to acquire a visual field image according to the line of sight of the user X including the peripheral image. As a result, the line-of-sight recognition unit 20 can acquire the field-of-view image of the user X even if the user terminal 3C does not have the field-of-view image acquisition device 32.

Although the display image generation device and the display image generation method of the present disclosure have been described based on the above-described embodiments, the specific configuration is not limited to each of these embodiments and is within the scope of claims. As long as the gist of the invention according to each of the above claims is not deviated, design changes and additions are permitted.

In each embodiment, both user X and user Y have shown an example of being in a vehicle, but the present invention is not limited to this. For example, one or both of the user X and the user Y who is the speaking subject may exist (position) outside the vehicles 2A to C (that is, a place away from the vehicles 2A to C). In this case, the user terminal of user X or the server to which the user terminal of user X can be connected has a configuration of a speech data acquisition unit, an object extraction unit, an object determination unit, and a display image generation unit. You need to have at least. The field of view image acquisition unit is included in, for example, the field of view image acquisition device of the user terminal. Then, the peripheral image obtained by the peripheral imaging device may be used as a visual field image corresponding to the visual field Ex of the user X, or the user X may have the peripheral imaging device and transmit it to the user terminal. Further, when the user X is outside the vehicle, the posture acquisition device 23 is installed near the user terminal 3C or the user X, and acquires the face orientation information of the user X by the face image or the sensor of the user X. Then, the line-of-sight recognition unit 20 recognizes the line-of-sight direction of the user X from the face image or face orientation information of the user X acquired by the posture acquisition device 23. The field-of-view image acquisition unit generates a field-of-view image of the user X based on the peripheral image captured by the peripheral image pickup device 22 and the line-of-sight direction of the user X recognized by the line-of-sight recognition unit 20. When the user X is outside the vehicle, the line-of-sight recognition unit 20 is assumed to be owned by the user terminal of the user X or the server to which the user terminal of the user X can be connected. Then, even when the user X is outside the vehicle, the user terminal generates a display image P relating to the extraction target Te included in the speech made by the speaking subject. Then, the display image P is displayed on the display image display device.

In each embodiment, the object determination unit outputs information on the determination result of whether or not the extraction target Te is included in the visual field image of the user X to the speaker terminal 4 of the user Y who is the speaker. Although shown, it is not limited to this. For example, as the information to be output to the user Y, the field image, the display image P, the peripheral image, and the like of the user X may be output. Further, when the user Y is particularly present outside the vehicle, the image is displayed on the user Y's speaking subject terminal 4 or VR (Virtual Reality, image display device). By displaying the image to the user Y who is the main speaker in this way, the user Y can obtain information on the visible area and the line-of-sight direction of the user X, so that the topic of the user X and the user Y progresses. It becomes easier to decide.

Further, the peripheral image captured by the peripheral imaging device 22 is not limited to the one described in each of the above embodiments, and may be, for example, a visual field image corresponding to the visual field Ex of the user X. Here, for example, when the user Y who is the speaking subject exists outside the vehicle of the vehicle 2A, the speaking subject terminal 4 displays a part or all of the peripheral images captured by the peripheral imaging device 22. May be done. As a result, it is possible to determine how to proceed with the topic between the user X and the user Y.

Further, although the display image display devices 31A to 31C of the user terminals 3A to 3C have shown an example of using a transmissive display, a head-up display installed in the vehicles 2A to 2C may be used. For example, the head-up display is set at the lower position of the front window of the vehicles 2A to 2C, and displays an image on the windshield with a light device. In this case, the image displays the display image P corresponding to the field of view of Ex of the user X generated by the display image generation units 18A to 18C.

Further, the speaking subject may be a speaking device that speaks to the user X instead of a person. In the case of a speaking device, the speaking data is output sentence data. The output sentence data may be voice data in which the speaking device outputs the output sentence (character string) as voice, or may be an output sentence (character string). Therefore, the display image generation devices 1A to 1C can acquire the output sentence data from the speech device that makes a speech to the user X by the speech data acquisition device. Further, in this case, the "speech uttered by the speaking subject" is the "speech uttered (output) by the speaking device". Further, the display image generation units 18A to 18C may acquire information for specifying a speaking device that emits a voice to the user X, and may generate a display image P such as "Mentioned by Speech output device." As a result, the extraction target information included in the speech of the speaking device can be displayed in an appropriate display mode for the user X. Specifically, the speaking device may be a so-called interactive agent device capable of voice dialogue with the user X.

Further, in the above, the speaking subject is only one user Y or only one speaking device, but the target of the speaking subject may be a plurality. For example, the target of the speaking subject may be two or more passengers (users), or one passenger (user) and one speaking device. In this case, the remark data acquisition unit 12 acquires information that identifies the remark subject who made a remark to the user X. Next, the display image generation units 18A to 18C generate the display image P including the information for identifying the speaking subject acquired by the speaking data acquisition unit 12. As a result, the display image generation devices 1A to 1C acquire the information for identifying the speaking subject by the speech data acquisition unit 12, and generate the display image P including the information for identifying the speaking subject by the display image generation units 18A to 18C. be able to. As a result, when there are a plurality of subjects of the speaking subject, the user X can clearly grasp the speaking subject.

In the above, an example of having the speech data acquisition unit 12 and the speech data acquisition device 41 is shown, but if the speech data acquisition unit 12 has the function of the speech data acquisition device 41, the speech data acquisition device 41 is not provided. You may. Further, although an example having the visual field image acquisition units 14A and 14B and the visual field image acquisition device 32 is shown, if the visual field image acquisition units 14A and 14B have the functions of the visual field image acquisition device 32, the visual field image acquisition device 32 can be used. You don't have to prepare. Further, although an example having the line-of-sight recognition unit 20 and the posture acquisition device 23 is shown, if the line-of-sight recognition unit 20 has the function of the posture acquisition device 23, the posture acquisition device 23 may not be provided. Furthermore, although an example having the peripheral image acquisition unit 11 and the peripheral image pickup device 22 is shown, if the peripheral image acquisition unit 11 has the function of the peripheral image pickup device 22, the peripheral image pickup device 22 may not be provided. ..

In the second embodiment, the object determination unit 15B determines whether or not the extraction target Te is POI, and whether the image of the extraction target Te is included in the visual field image acquired by the visual field image acquisition unit 14B. An example of determining whether or not to use is shown, but the present invention is not limited to this. For example, the object determination unit does not determine whether the extraction target is POI, but only determines whether the image of the extraction target is included in the field image acquired by the field image acquisition unit. good. In this determination, even if the extraction target is not POI, if it is determined that the extraction target is included in the visual field image, the first display image is generated.

1A, 1B, 1C Display image generator 11 Peripheral image acquisition unit 12 Speech data acquisition unit 13 Object extraction unit 14A, 14B, 14C Field image acquisition unit 15A, 15B, 15C Object determination unit 16A, 16B, 16C Presence / absence determination unit 17A, 17B, 17C Positional relationship acquisition unit 18A, 18B, 18C Display image generation unit 19 POI information storage unit 20 Line-of-sight recognition unit 2A, 2B, 2C Vehicle 21 Navigation device 22 Peripheral image pickup device 23 Posture acquisition device 3A, 3B, 3C User Terminals 31A, 31B, 31C Display image display device 32 Field image acquisition device 4 Speech-based terminal 41 Speech data acquisition device

Claims (14)

A display image generator that identifies an object included in a statement made by a speaker as an extraction object and generates a display image related to the extraction object.
A speech data acquisition unit that acquires speech data of the speech issued to the user by the speech subject, and a speech data acquisition unit.
A plurality of object data are stored in advance, the plurality of object data are compared with the speech data acquired by the speech data acquisition unit, and the data that matches the object data among the speech data is selected as described above. An object extraction unit to be extracted as an extraction object,
A field image acquisition unit that acquires an image including at least a field image corresponding to the user's field of view, and a field image acquisition unit.
An object determination unit that determines whether or not the extraction object extracted by the object extraction unit is included in the visual field image, and an object determination unit.
A display image generation unit that acquires extraction target information that is information on the position of the extraction target and generates the display image that includes the extraction target information that is different from the field image is provided.
The display image generation unit determines the display mode of the display image regarding the extraction target based on the determination result of whether or not the extraction target is included in the visual field image by the object determination unit. A featured display image generator. When the object determination unit determines that the extraction object is included in the field image, the display image generation unit displays the extraction object information in the display mode that emphasizes the extraction object itself. The display image generation device according to claim 1, wherein the display image is generated. A positional relationship acquisition unit for acquiring the relative positional relationship between the extraction target and the user is provided.
When the object determination unit determines that the extraction object is not included in the field image, the display image generation unit shows the extraction object information in the display mode for displaying the positional relationship. The display image generation device according to claim 1 or 2, wherein the display image is generated. The display image generation unit makes the extraction target visible to the user from the field image based on the determination result of whether or not the extraction target is included in the field image by the object determination unit. The display image generation device according to any one of claims 1 to 3, wherein the display image including information indicating whether or not the image is generated is generated. When the object determination unit determines that the extraction target is not included in the visual field image, the presence / absence determination unit that determines whether or not the extraction target exists within the preset target range is provided. Prepare,
The display image generation unit is characterized in that the display mode of the extraction target information is determined based on the determination result of whether or not the extraction target is within the target range by the presence / absence determination unit. The display image generator according to any one of claims 1 to 4. A peripheral image acquisition unit that acquires a peripheral image that is a peripheral area of the user including the visual field image and stores the acquired peripheral image is provided.
The presence / absence determination unit determines whether or not the extraction target is within the target range based on the current or past peripheral image acquired by the peripheral image acquisition unit. Item 5. The display image generator according to item 5. A positional relationship acquisition unit for acquiring the relative positional relationship between the extraction target and the user is provided.
When the presence / absence determination unit determines that the extraction target is within the target range, the display image generation unit uses a reference position set at the user's position or a position in the vicinity of the user as a reference. The display according to claim 5 or 6, wherein the display image showing the extraction target information is generated in the display mode for displaying the positional relationship including the direction and distance of the position of the extraction target. Image generator. The extraction target is a POI (Points of Interest), which is a landmark associated with a position on a map.
A POI information storage unit for storing POI information of the POI including at least information on the position of the extraction target is provided.
Claims 5 to 7 are characterized in that the presence / absence determination unit determines whether or not the extraction target is within the target range based on the POI information stored by the POI information storage unit. The display image generator according to any one of the items up to. The subject of the statement is a person
The display image generation device according to any one of claims 1 to 8, wherein the speech data is speech signal data of the speech issued to the user by a person. The speaking subject is a speaking device that makes the speaking to the user.
The image generation device according to any one of claims 1 to 8, wherein the remark data is output sentence data indicating the content of the output sentence output as the remark. The object determination unit according to any one of claims 1 to 10, wherein the object determination unit outputs information on the determination result of whether or not the extraction object is included in the visual field image to the speaking subject. The image generator described. The remark data acquisition unit acquires information that identifies the remark subject who made the remark to the user, and obtains information.
The display image generation unit is described in any one of claims 1 to 11, wherein the display image generation unit generates the display image including the information that identifies the speech subject acquired by the speech data acquisition unit. Display image generator. A peripheral image acquisition unit that acquires a peripheral image that is a peripheral area of the user including the visual field image and stores the acquired peripheral image.
A line-of-sight recognition unit that recognizes the user's line of sight is provided.
The visual field image acquisition unit acquires the visual field image based on the current peripheral image acquired by the peripheral image acquisition unit and the current line of sight of the user recognized by the line-of-sight recognition unit. The display image generation device according to any one of claims 1 to 12. It is a display image generation method by a display image generation device that specifies an object included in a statement made by a speaker as an extraction object and generates a display image related to the extraction object.
The remark data acquisition step of acquiring the remark data of the remark made to the user by the remark subject, and
An object extraction step of comparing a plurality of object data stored in advance with the acquired remark data and extracting data that matches the object data among the remark data as the extraction target.
A field image acquisition step of acquiring a field image corresponding to the user's field of view, and
An object determination step for determining whether or not the extracted object to be extracted is included in the visual field image, and an object determination step.
The display image generation step of acquiring the extraction target information which is the information about the position of the extraction target and generating the display image including the extraction target information different from the field image is included.
In the display image generation step, the display mode of the display image relating to the extraction target is determined based on the determination result of whether or not the extraction target is included in the visual field image in the object determination step. A display image generation method characterized by.

Images