JP2006343829A - Device and method for photographing around vehicle - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To easily specifying an object intended by a user from an image obtained by photographing the surroundings of a vehicle even if a direction of the vehicle is changed during voice recognition. <P>SOLUTION: A method for photographing around vehicle captures images photographed by a plurality of cameras around the vehicle at a start of the voice recognition (S106), calculates a variation amount of a direction of its own vehicle between the start time and an end time of the voice recognition (S112), determines priorities of images in which the objects recognized through the voice recognition are estimated to be contained by using a table with the priorities defined therein on the basis of the timing to capture the variation amount of the direction of its own vehicle and images (S114), and displays a plurality of image area candidates onto a display part in accordance with the determined priorities to make the user select the intended object when the image areas estimated as the objects are detected more than one (S212). <P>COPYRIGHT: (C)2007,JPO&INPIT

Description

  The present invention relates to a vehicle periphery photographing apparatus that photographs a vehicle periphery using a plurality of cameras mounted on the vehicle, and a vehicle periphery photographing method that photographs a vehicle periphery using a plurality of cameras mounted on the vehicle.

2. Description of the Related Art Conventionally, as a vehicle periphery photographing device that photographs a vehicle periphery using a camera mounted on the vehicle, a device that can recognize a user's speech and control a photographing direction and a camera operation has been proposed ( For example, see Patent Document 1).
JP-A-8-277736 (paragraph numbers 0019 to 0021)

  However, as described in Japanese Patent Application Laid-Open No. 2004-228561, when the situation around the vehicle is photographed by recognizing the user's voice, the following problem occurs.

  For example, before the intersection, the user looks at the sign in front and speaks “What is the sign in front?” And turns right at the intersection. When the image around the vehicle is taken in response to the above, the “sign” may be included in the image taken by the camera photographing the left side of the vehicle, not the image photographed by the camera photographing the front of the vehicle.

  In this way, when the situation around the vehicle is photographed by voice recognition, there is a situation in which the target object intended by the user cannot be specified if the direction of the vehicle changes while the voice of the user's utterance is recognized. It is done.

  Therefore, when a plurality of image areas estimated as objects recognized by voice recognition are searched from captured images around the vehicle, these image areas are displayed on the display screen, and the user selects an intended object. A method is conceivable. However, if a plurality of image areas are uniformly displayed on the display screen, there is a problem that it is difficult to specify an intended object.

  The present invention has been made in view of the above problem, and an object of the present invention is to make it easy to specify an object intended by a user from an image area obtained by photographing the periphery of a vehicle.

  In order to achieve the above object, according to the first aspect of the present invention, the amount of change in the direction of the vehicle at the start and end of the voice recognition function for recognizing the voice spoken by the user is calculated, and Based on the amount of change in the direction of the own vehicle, the direction of the object recognized by the voice recognition, and the timing of capturing the image. The image capture is performed in the order in which the objects recognized by the voice recognition are estimated to be included using a table in which the priority of the images estimated to include the objects recognized by the voice recognition is defined. Determining a priority of the image captured by the means, detecting an image region estimated as the object recognized by the voice recognition from the captured image, and a plurality of image regions When detected, the plurality of image area candidates are displayed on a display unit in accordance with the determined priority, and an object intended by the user is selected from the image areas displayed on the display unit. Yes.

  Therefore, the display unit displays image region candidates that are estimated to include objects recognized by voice recognition in accordance with the priority, so that the object intended by the user can be easily specified. Can do.

  Further, the predetermined timing based on the user's utterance can be at the start of speech recognition or at the end of speech recognition, as in the second aspect of the invention.

  Here, the image around the vehicle captured at the start of speech recognition is not only an image obtained by capturing images of the periphery of the vehicle with a plurality of cameras at the start of speech recognition. A moving image captured by a plurality of cameras is stored in a storage medium, and an image obtained by capturing an image at the start of voice recognition from the storage medium at the end of voice recognition is also included.

  In the inventions according to claims 4 to 6, the situation at the time of photographing around the vehicle is acquired, and the color detection range recognized by voice recognition is corrected according to the obtained situation at the time of photographing around the vehicle, Based on the corrected color detection range, an image region estimated as an object recognized by speech recognition is detected from an image captured by the image capturing means.

  Therefore, it is possible to specify the target object intended by the user even when the color of the captured image is different from the actual color depending on the situation at the time of shooting around the vehicle.

  In the invention according to claim 7, the image data of the image area of the object is transmitted to the server via the communication means, and the detailed information of the object searched by the server is acquired based on the transmitted image data. The detailed information of the acquired object is provided to the user.

  As described above, the image data of the image area of the object is transmitted to the server, and the detailed information of the object searched by the server is acquired based on the transmitted image data. Even if it is not stored in the internal storage medium, it is possible to obtain detailed information of the object from the server and provide it to the user.

(First embodiment)
A schematic configuration of an information providing system according to the first embodiment of the present invention is shown in FIG. The information providing system includes a navigation device 1 as a vehicle periphery imaging device and a server 2. Thus, in this embodiment, the vehicle periphery imaging device is configured using the navigation device 1.

  The navigation device 1 includes a remote control sensor 10, a touch panel 11, a voice recognition unit 12, a voice synthesis unit 13, a display device 14, a display 14a, a position detector 15, an external storage device 16, a communication device connection device 17, and a control circuit 18. ing.

  The control circuit 18 is connected to a vehicle-mounted camera 19 that captures the situation around the vehicle, the speech recognition unit 12 is connected to a microphone 12a, and the speech synthesis unit 13 is connected to a speaker 13a.

  The remote control sensor 10 outputs a signal received from the remote control 10 a that transmits a radio signal using infrared rays or the like to the control circuit 18 based on a user operation.

  The touch panel 11 is provided so as to overlap the display screen of the display 14 a and outputs a signal corresponding to a user's touch operation to the control circuit 18.

  The voice recognition unit 12 recognizes the user's voice from the voice signal input from the microphone 12 a, extracts a language included in the voice, and outputs the language to the control circuit 18.

  The voice synthesizer 13 generates a voice signal for voice output based on the signal input from the control circuit 18, and outputs the generated voice signal to the speaker 13a.

  The display device 14 displays the video on the display unit of the display 14 a such as an LED or LCD in accordance with the video signal from the control circuit 18.

  The position detector 15 includes sensors such as a GPS receiver, a gyro sensor, a vehicle speed sensor, and an acceleration sensor. The position detector 15 is used to specify the position and direction (orientation) of the vehicle based on the characteristics of each sensor. The positioning signal is output to the control circuit 18.

  The external storage device 16 controls data reading and data writing to an external storage medium such as a CD-ROM, DVD, or memory card, if possible. Examples of information stored in the external storage medium include map matching data and map data.

  The communication device connection device 17 is a device for connecting the communication device 3. The communication device 3 is connected to the communication device connection device 17, and bidirectional communication is performed between the navigation device 1 and the server 2 via the communication device 3. Note that a mobile phone is used as the communication device 3.

  The control circuit 18 is configured by a computer having a memory such as a ROM and a RAM, a CPU, and the like, and executes various processes according to programs stored in the ROM. Specifically, there are a current position calculation process for calculating the current position based on a signal input from the position detector 15 and a map display process for displaying the vehicle position mark on the current position on the map. The control circuit 18 includes a timer for measuring time, and manages the date and time by this timer.

  The in-vehicle camera 19 has three cameras that photograph the front side of the vehicle, the right side of the vehicle, and the left side of the vehicle. Each piece of image data photographed by these cameras is input as a moving image to the control circuit 18, and the control circuit 18 takes it into a memory as a still image when necessary.

  The navigation device 1 according to the present embodiment recognizes a user's utterance by the voice recognition unit 12, captures the vehicle periphery by the in-vehicle camera 19 when the voice recognition is completed, and is recognized by voice recognition from the captured image. The image area of the object is specified, and the specified image data is transmitted to the server 2 via the communication device 3 connected to the communication device connection device 17.

  The server 2 includes a database (DB) 20 in which various detailed information such as animals and plants, signs, cars, and the like are stored. In this database 20, for example, detailed information such as images of animals and plants, names, and places of origin are stored for animals and plants, and detailed information such as images of cars, vehicle model names, manufacturer names, and model years are stored for automobiles. Yes.

  When the server 2 receives the image data from the navigation device 1, the server 2 searches the database 20 for an image that matches the object included in the received image data, and transmits detailed information on the searched object to the navigation device 1. For example, when image data of an oncoming vehicle is received from the navigation device 1, an image that matches the image of the oncoming vehicle included in the image data is searched from the database 20. Detailed information such as the car model name, manufacturer name, and model year is transmitted to the navigation device 1.

  The navigation device 1 displays detailed information on the object provided from the server 2 on the display 14a.

  Next, speech recognition by the speech recognition unit 12 will be described. The speech recognition unit 12 extracts a language included in the user's speech using morphological analysis of natural language, and relative orientation (front, right front, upper left, lower right, left rear, etc.), target for each extracted language Recognize object features (color, shape, size, etc.), target names (proprietary nouns, facilities, signboards, animals and plants, etc.), and interrogative words (what, when, where, who, why, how).

  Various databases such as a color database (color DB), a shape database (shape DB), and a relative orientation database (relative orientation DB) as shown in FIG. 2 are stored in the external storage medium of the external storage device 16. The voice recognition unit 12 recognizes the relative orientation, the feature of the target, the name of the target, the interrogative, etc. using these databases.

  For example, in response to the question “What is the name of a blue oncoming car” by the user, “blue”, “opposite”, “car”, “name”, “what” are extracted, and the database shown in FIG. “Blue” is a representation of the color contained in the color database, the classification is “blue”, “opposite” is a representation of the relative orientation contained in the relative orientation database, and the classification is “front”. Recognize target names, question words, etc.

  In addition, the voice recognition unit 12 starts voice recognition processing in response to a user's voice input, and ends a series of voice recognition processing when there is no voice input for a predetermined time (for example, 5 seconds). Then, when there is a user's voice input again, a series of voice recognition processing is performed until a state in which no voice input is continued for a predetermined time or longer.

  As described above, when shooting the situation around the vehicle by voice recognition, if the direction of the vehicle changes between the start of voice recognition for recognizing the user's voice and the end of voice recognition, the voice recognition There may be a case where the object intended by the user is not included in the image obtained by capturing the recognized direction (relative orientation). For this reason, the navigation device 1 calculates the amount of change in the direction of the vehicle from the start of the speech recognition to the end of the speech recognition, and prioritizes the image that is estimated to include the target object intended by the user. The display unit displays image area candidates that are estimated to include objects recognized by voice recognition according to the determined priority, and identifies the intended object from the image area candidates. It is easy.

  The voice recognition unit 12, the voice synthesis unit 13, and the control circuit 18 of the navigation device 1 are configured by a computer. Next, the processing of this computer will be described with reference to FIG. The navigation device 1 starts operating when power is supplied from the battery via the ignition switch, and the computer periodically performs the processing shown in FIG. 3 in parallel with the processing such as the current location calculation processing described above. The computer is connected to the Internet from the communication device connection device 17 via a mobile phone at regular intervals, acquires the current weather from a site on the Internet, and stores it in a memory.

  First, the absolute azimuth (D1) of the host vehicle is calculated based on the positioning signal input from the position detector 15, and the calculated absolute azimuth (D1) is stored in the memory (S100). Note that the absolute orientation in the present embodiment is calculated as a value from 0 ° to 360 ° in the counterclockwise direction with respect to the north.

  Next, it is determined whether there is an audio input based on whether there is an audio signal input from the microphone 12a (S102).

  If there is no audio signal input from the microphone 12a, it is determined as “None” in S102, the process returns to S100, the latest absolute direction (D1) is calculated again, and the calculated latest absolute direction (D1) is stored in the memory. Remember.

  Further, when an audio signal is input from the microphone 12a, “Yes” is determined in S102, and voice recognition for recognizing the user's voice is started. When voice recognition is started in this way, the absolute direction (D1) stored in the memory becomes the direction of the host vehicle at the start of voice recognition.

  Next, the orientation, characteristics (color, shape), interrogative, etc. of the object are extracted by voice recognition. Specifically, the orientation, characteristics (color, shape), interrogation, etc. of the object included in the user's voice are recognized from the language extracted by voice recognition (S104).

  Then, when a silent state without a voice input continues for a certain time or more and a series of voice recognition is finished, an image around the vehicle photographed by the in-vehicle camera 19 is captured and stored in the memory (S106).

  Next, the situation around the vehicle at the time of image capture is acquired. Specifically, the situation around the vehicle such as date information and weather at the time of image capture is acquired. The date / time information is acquired from the timer, and the current weather is read from the memory (S108).

  Next, the absolute azimuth (D2) of the host vehicle is calculated based on the positioning signal input from the position detector 15, and the calculated absolute azimuth (D2) is stored in the memory (S110).

  Next, the amount of change in the direction of the own vehicle from when the speech is taken to the time when the image is captured, that is, the amount of change in the direction of the own vehicle from when speech recognition is started until the end of speech recognition is calculated. As shown in FIG. 4, the amount of change in direction is calculated as a difference (D1-D2) between the absolute direction (D1) of the own vehicle calculated in S100 and the absolute direction (D2) of the own vehicle calculated in S110 ( S112).

  Next, it is estimated that the object around the vehicle is included from the images obtained by photographing the object around the vehicle based on the relative azimuth of the object recognized by the voice recognition and the amount of change in the azimuth calculated in S112. The priority of the image area (camera image) to be determined is determined (S114).

  As shown in FIG. 5, the external storage medium of the external storage device 16 includes the object recognized by the voice recognition with respect to the amount of change in the direction of the own vehicle and the direction of the object recognized by the voice recognition. A table in which the priority of the estimated image is defined is stored, and this table is used to determine the priority of the image area including the object intended by the user.

  For example, when “front> right, left” is written, the priority of the image area captured by the camera that captures the right side of the vehicle and the image captured by the camera that captures the left side of the vehicle is the same. This means that the priority of the image taken by the camera to be taken is the highest.

  In addition, when “Previous> Right> Left” is described, the priority of the image captured by the camera that captures the front of the vehicle is the highest, and then the priority of the image captured by the camera that captures the right side of the vehicle. This means that the priority of the image captured by the camera that captures the left side of the vehicle is the lowest.

  Next, image area detection processing is performed for detecting the image area of the object for each image from the situation at the time of image capture acquired in S108, the priority determined in S114 and the characteristics of the object recognized by voice recognition (S200). ).

  FIG. 6 shows a flowchart of the image area detection process. In this image area detection process, the color of the image captured due to the influence of time zone, weather, etc. is corrected by correcting the color range using a color range conversion table stored in advance in an external storage medium. Even if it is different from the above, the object intended by the user can be specified.

  FIG. 7 shows a color range conversion table (time zone) for correcting the color range due to the influence of the time zone. For example, in the morning and evening time zones, even white objects may appear reddish due to the effects of sunrise or sunset, so the color detection range should be expanded so that the red range is expanded relative to the colors recognized by voice recognition. It is to be corrected. Also, in the night time zone, the surroundings may be dark and even white objects may appear gray, so the color detection range should be corrected to reduce the brightness with respect to the colors recognized by voice recognition. It has become.

  FIG. 8 shows a color range conversion table (weather) for correcting the color range due to the influence of the weather. As shown in the figure, in the case of cloudy, rainy or snowy, the surroundings become dark, so the color detection range is corrected so as to reduce the brightness with respect to the color recognized by voice recognition.

  In the image region detection process shown in FIG. 6, first, an image (image region) having the highest priority is read (S202).

  Next, the correction content of the matching color range is specified from the image capture status (date and weather) acquired in S108 and the color range conversion table, and the specified correction is performed for the color of the object recognized by voice recognition. Perform color range correction based on the content. Specifically, the correction contents in the date and time (time zone) at the time of image capture are specified by referring to the color range conversion table (time zone), and the weather at the time of image capture by referring to the color range conversion table (weather) In step S204, the correction content is determined, and the color of the object recognized by voice recognition is corrected based on the specified correction content.

  For example, even if the color of the object recognized by voice recognition is white, the red range is expanded with respect to white in the evening time zone, and the color range including not only white but also pink is corrected. Even if the color of the object recognized by voice recognition is white, in the case of rain, it is corrected to a gray color range rather than white so as to reduce the brightness.

  Next, matching of the image area estimated as the object from the read image is performed based on the situation at the time of image capture and the characteristics of the object recognized by the speech recognition. Specifically, using the color range corrected in S204, an image area of a color included in the corrected color range is searched from the read image. For example, when “signpost” is recognized by voice recognition, a shape that matches the feature of the sign such as a circle or a triangle is added to the search condition from the read image, and image region matching is performed (S206). .

  Here, when a matching image area is detected from the read image, the detected image area is stored in the RAM as a candidate for the object (S208), and whether or not matching determination has been performed for all images. Is determined (S210).

  As described above, when it is determined that the above-described processing S202 to S208 is repeated in order from the image with the highest priority and the matching determination is performed on all the images in S210 (YES in S210), next, in the memory. If it is determined whether there are a plurality of stored object candidates and it is determined that there are a plurality of object candidates, the object candidates are displayed on the display 14a according to the priority, and the intended object is displayed to the user. The candidate selected by the operation of the user's touch panel 11 or the remote controller 10a from the image candidates displayed on the display 14a is specified as an object intended by the user (S212).

  For example, numbers are assigned in descending order of priority, displayed in order from the left to the right of the display 14a in order of high priority, and the intended object is selected by the user. When it is determined that there are a plurality of candidate objects in the same image, the priorities of the candidates previously stored in the memory are increased, and are displayed on the display 14a in descending order of priority.

  In S212, if there is one candidate for the target stored in the memory, the target stored in the memory is displayed on the display 14a, and the target stored in the memory is specified as the target intended by the user. To do.

  In S212, if there is no candidate for the object stored in the memory, the display 14a displays that the object intended by the user is not searched, and the process is terminated.

  Returning to the description of FIG. 3, in S300, together with the inquiry content from the user, the image area detected by the image area detection processing in S200, that is, the image data of the object specified by the user's operation is transmitted to the server 2. Before transmitting image data to the server 2, the image is cut out so as to be the minimum image including the target object, and the cut out image is transmitted to the server 2. The inquiry content from the user is transmitted to the server 2 as a language recognized by voice recognition for recognizing the user's voice (S300).

  When the server 2 receives the inquiry content and image data from the user, the server 2 searches the database 20 for a matching image based on the received inquiry content and image data from the user, and transmits the search result to the navigation device 1. Specifically, an object to be matched is searched from the database 20 based on the received inquiry content and image data from the user, and detailed information of the searched object is transmitted to the navigation device 1.

  When the computer of the navigation device 1 receives the detailed information of the searched object from the server 2 (S302), the computer displays the detailed information of the received object on the display 14a and outputs the sound from the speaker 13a (S304). Return to the process.

  According to the above configuration, at the start of the voice recognition function for recognizing the voice uttered by the user, images obtained by photographing the periphery of the vehicle with a plurality of cameras are captured (S106), and the voice recognition function for recognizing the voice uttered by the user. The amount of change in the direction of the vehicle at the start and the end is calculated (S112), the amount of change in the direction of the vehicle, the direction of the object recognized by the voice recognition, and the image captured by the image capturing means In the order in which it is estimated that the object recognized by the speech recognition is included using the table in which the priority of the image estimated to include the object recognized by the speech recognition based on the timing of The priority of the captured image is determined (S114), and an image region estimated as a target recognized by voice recognition is detected from the captured image (S200), When an image region is detected, a plurality of image region candidates are displayed on the display unit according to the determined priority, and an object intended by the user is selected and selected from the image regions displayed on the display unit (S212). )

  Therefore, the display unit displays image area candidates that are estimated to include the object recognized by the voice recognition according to the priority, so that the object intended by the user can be easily specified.

(Second Embodiment)
Next, an information providing system according to the second embodiment will be described. Although the navigation apparatus 1 in the first embodiment has shown an example of capturing an image obtained by photographing an object around the vehicle at the end of voice recognition, the navigation apparatus 1 in the present embodiment is a moving image around the vehicle for a certain period of time. Is stored in the memory, and the image stored in the memory is read out at the end of the speech recognition. Hereinafter, the same parts as those of the first embodiment are denoted by the same reference numerals, description thereof will be omitted, and different parts will be mainly described.

  FIG. 9 shows the processing of the computer of the navigation device 1 according to this embodiment.

  First, an image around the vehicle taken by the in-vehicle camera 19 is captured and stored in the memory (S400).

  Next, the situation around the vehicle at the time of image capture is acquired. Specifically, the situation around the vehicle such as date information and weather at the time of image capture is acquired (S402).

  Next, the absolute azimuth (D3) of the host vehicle is calculated based on the positioning signal input from the position detector 15, and the calculated absolute azimuth (D3) is stored in the memory (S404).

  Next, it is determined whether there is an audio input based on whether there is an audio signal input from the microphone 12a (S406).

  If there is no audio signal input from the microphone 12a, it is determined as “None” in S406, the process returns to S400, the latest absolute direction (D3) is calculated again, and the calculated latest absolute direction (D3) is stored in the memory. Remember.

  When a voice signal is input from the microphone 12a, “Yes” is determined in S406, and voice recognition for recognizing the user's voice is started. Note that the absolute direction (D3) stored in the memory is the direction of the vehicle at the start of speech recognition.

  Next, the orientation, characteristics (color, shape), interrogative, etc. of the object are extracted by voice recognition. Specifically, the orientation, characteristics (color, shape), interrogation, etc. of the object included in the user's voice are recognized from the language extracted by voice recognition (S408).

  When a silent state without a voice input continues for a certain time or longer, when a series of voice recognition is finished, the absolute direction (D4) of the host vehicle is calculated based on the positioning signal input from the position detector 15 and the calculated absolute The direction (D4) is stored in the memory (S410).

  Next, the amount of change in the direction of the own vehicle from when the speech is taken to the time when the image is captured, that is, the amount of change in the direction of the own vehicle from when speech recognition is started until the end of speech recognition is calculated. As shown in FIG. 4, the amount of change in direction is calculated as the difference (D4-D3) between the absolute direction (D4) of the vehicle calculated in S410 and the absolute direction (D3) of the vehicle calculated in S404. (S412).

  Next, the priority of the image estimated to include the object recognized by the voice recognition is defined with respect to the amount of change in the direction of the vehicle and the direction of the object recognized by the voice recognition shown in FIG. Based on the determined table, the priority of the image area estimated to include the object intended by the user is determined (S414).

  In the first embodiment, an image obtained by photographing the periphery of the vehicle at the end of voice recognition is captured. In this embodiment, an image obtained by photographing the periphery of the vehicle at the start of speech recognition is captured. The priority may be determined using a different table. However, by changing the calculation method of the azimuth change amount, the priority can be determined using the same table (shown in FIG. 5) as in the first embodiment. I try to make a decision.

  That is, in the first embodiment, in the process of S112 in FIG. 3, the azimuth changes as the difference (D1-D2) between the absolute azimuth (D1) at the start of speech recognition and the absolute azimuth (D2) at the end of speech recognition. In this embodiment, the amount of change in orientation is calculated as the difference (D4-D3) between the absolute orientation (D4) at the end of speech recognition and the absolute orientation (D3) at the start of speech recognition. Thus, the priority is determined using the same table (shown in FIG. 5) as in the first embodiment.

  Next, an image area estimated as an object is detected based on the situation at the time of image capture, the priority of the image area, and the characteristics of the object recognized by voice recognition (S200).

  Thereafter, processing similar to S300 to S304 in FIG. 3 is performed, and detailed information on the object is provided to the user.

  Therefore, for example, as shown in FIG. 10 (a), the user starts to speak before the intersection, “What is that blue car? When the vehicle completes a right turn at a right-angled intersection as shown in FIG. 10B, a series of voice recognition ends, and the user speaks before the intersection. The direction of the vehicle at the start is stored in the memory as the direction (D3) of the vehicle at the start of voice recognition, and the direction of the vehicle when the vehicle at the end of the voice recognition has finished turning right at the intersection It is stored in the memory as the vehicle direction (D4). In this case, the azimuth change amount θ is calculated as θ = D4−D3 = −90 °.

  As shown in FIG. 5, when the relative azimuth is “left” and the azimuth change amount is “−60 ° or more”, the priorities are in the order of “front> right, left”.

  Therefore, as shown in FIG. 10A, an image area detection process is performed for preferentially detecting the image of the object from the captured image of the camera that captures the front of the vehicle.

  As shown in FIG. 11 (a), the user starts speaking before the intersection, and utters “... In the case where a series of speech recognition is finished when the vehicle has turned right at a right-angled intersection as shown in FIG. 3), the direction change amount θ is calculated as θ = D4−D3 = −90 °.

  As shown in FIG. 5, when the relative azimuth is “right” and the azimuth change amount is “−60 ° or more”, the priorities are in the order of “right> front> left”.

  Therefore, as shown in FIG. 11 (a), the image of the object is detected in the order of the photographed image of the camera photographing the right side of the vehicle, the photographed image of the camera photographing the front of the vehicle, and the photographed image of the camera photographing the left side of the vehicle. The image area detection process to be performed is performed.

  Also, as shown in FIG. 12 (a), the user starts speaking just before approaching a road that is 45 ° to the right, and “... is visible in the front ...” in the middle of the right turn at the intersection. When a series of voice recognition ends while the vehicle is traveling on a road that is 45 ° to the right as shown in FIG. 12B, the direction change amount θ is θ = D4−D3 = Calculated as -45 °.

  As shown in FIG. 5, when the relative azimuth is “front” and the azimuth change amount is “−30 ° or more and less than −60 °”, the priorities are in the order of “front> left> right”.

  Accordingly, as shown in FIG. 12 (a), the image of the object is detected in the order of a captured image of the camera that captures the front of the vehicle, a captured image of the camera that captures the left side of the vehicle, and a captured image of the camera that captures the right side of the vehicle. The image area detection process to be performed is performed.

  Also, as shown in FIG. 13A, the user starts speaking just before reaching the road that is 45 ° to the right, and in the middle of the right turn at the intersection, “... looks to the right ...” When a series of voice recognition ends while the vehicle is traveling on a road that is 45 ° to the right as shown in FIG. 13B, the direction change amount θ is θ = D4−D3 = Calculated as -45 °.

  As shown in FIG. 5, when the relative azimuth is “front” and the azimuth change amount is “−30 ° or more and less than −60 °”, the priorities are in the order of “front> left> right”.

  Therefore, as shown in FIG. 13 (a), the image of the object is detected in the order of the photographed image of the camera photographing the front of the vehicle, the photographed image of the camera photographing the left side of the vehicle, and the photographed image of the camera photographing the right side of the vehicle. The image area detection process to be performed is performed.

  As described above, in the present embodiment, a moving image around the vehicle for a certain time is stored in the memory, and the image stored in the memory is read out when the speech recognition ends. When shooting the situation around the vehicle after the voice recognition is completed, for example, in response to the user's inquiry “What is that oncoming vehicle?”, The oncoming vehicle passes, and the oncoming vehicle appears in the image taken by the in-vehicle camera 19. However, as in this embodiment, a moving image around the vehicle for a certain period of time is stored in the memory, and the image stored in the memory is read out at the end of the voice recognition. , Never miss the image of the oncoming vehicle.

  Although the navigation apparatus 1 according to the present embodiment has shown an example in which a moving image around the vehicle for a certain time is stored in the memory, a still image obtained by periodically capturing images around the vehicle is stored in the memory as needed. It may be.

(Third embodiment)
Next, an information providing system according to the third embodiment will be described. The navigation device 1 in the second embodiment has shown an example in which moving images around the vehicle for a certain period of time are stored in the memory, and the image stored in the memory is read out at the end of the speech recognition. The navigation device 1 captures an image obtained by photographing an object around the vehicle at the start of voice recognition. Hereinafter, the same parts as those of the second embodiment are denoted by the same reference numerals, description thereof will be omitted, and different parts will be mainly described.

  FIG. 14 shows the processing of the computer of the navigation device 1 according to this embodiment.

  First, it is determined whether there is an audio input based on whether there is an audio signal input from the microphone 12a (S500).

  If no audio signal is input from the microphone 12a, it is determined as “none” in S500, and the process returns to S500.

  When an audio signal is input from the microphone 12a due to the user's utterance, “Yes” is determined in S500, voice recognition for recognizing the user's voice is started, and an image around the vehicle captured by the in-vehicle camera 19 is displayed. Store it in the capture memory (S502).

  Also, the latest absolute direction (D5) is calculated in response to the start of voice recognition, and the calculated latest absolute direction (D5) is stored in the memory (S506).

  Next, the orientation, characteristics (color, shape), interrogative, etc. of the object are extracted by voice recognition. Specifically, the orientation, characteristics (color, shape), interrogation, etc. of the object included in the user's voice are recognized from the language extracted by voice recognition (S508).

  Then, when the silent state with no voice input continues for a certain time or more and the series of voice recognition is finished, the latest absolute direction (D6) is calculated, and the calculated latest absolute direction (D6) is stored in the memory (S510). ).

  Next, the amount of change in the direction of the own vehicle from when the utterance is taken until the image is captured is calculated. As shown in FIG. 4, the direction change amount is calculated as a difference (D6−D5) between the absolute direction (D6) of the own vehicle calculated in S510 and the absolute direction (D5) of the own vehicle calculated in S506. (S512).

  Thereafter, the same processing as S414 to S304 in FIG. 9 is performed, and detailed information of the object is provided to the user.

  As described above, in this embodiment, an image obtained by capturing an object around the vehicle is captured at the start of voice recognition. In the second embodiment, it is necessary to provide a relatively large-capacity memory in order to store a moving image around the vehicle for a certain period of time, and a load is continuously applied to the computer. In addition, a still image obtained by capturing an object around the vehicle at the start of voice recognition is taken into the memory, so that the memory capacity can be reduced and the load on the computer can be reduced.

(Other embodiments)
In the above embodiment, in the processing of S212, the example in which the target candidates are displayed side by side on the display 14a in the descending order of priority has been described, but for example, the timing may be shifted on the display 14a in order of the priority. Good.

  Moreover, although the example which used the mobile telephone as the communication apparatus 3 was shown in the said embodiment, you may use the communication apparatus which communicates, such as VICS and DSRC. Further, the navigation device 1 may be configured to incorporate the communication device 3.

  In the above embodiment, an example is shown in which an image obtained by capturing the periphery of the vehicle with a plurality of cameras at the start or end of speech recognition is captured based on the user's utterance. You may make it capture the image image | photographed with the camera.

  Further, in the above-described embodiment, the timing for capturing an image of the periphery of the vehicle is different as at the end of voice recognition and at the start by changing the calculation method of the amount of change in orientation as in S112 and S412. Even in this case, the example of determining the priority using the same table (shown in FIG. 5) is shown. However, the calculation method of the direction change amount is unified, and the periphery of the vehicle is photographed at the end of the speech recognition. The priority may be determined using separate tables depending on whether an image is captured or an image obtained by capturing the periphery of the vehicle at the start of voice recognition.

  The correspondence between the configuration in the above embodiment and the configuration in the claims will be described. Each processing in S112, S412, and S512 corresponds to a direction change amount calculation unit, and the processing in S106, S400, and S502 is an image capture. Each process of S114, S414, and S514 corresponds to a priority determination unit, the external storage medium of the external storage device 16 corresponds to a storage unit, and the process of S200 corresponds to an image area detection unit. The process of S212 corresponds to the selection means, the processes of S100, S404, and S506 correspond to the first azimuth calculation means, the processes of S110, S410, and S510 correspond to the second azimuth calculation means, and S108, S402. , S504 corresponds to status acquisition means, S204 corresponds to color correction means, and the communication device connection device 17 and the communication device 3 communicate with each other. Corresponds to the processing of S300 corresponds to the transmission means, the processing of S302 corresponds to the acquiring means, the processing of S304 corresponds to the information providing means.

It is a figure showing a schematic structure of an information service system concerning one embodiment of the present invention. It is a chart for demonstrating a color database, a shape database, and a relative orientation database. It is a flowchart which shows the process of the computer of the navigation apparatus 1 in 1st Embodiment. It is explanatory drawing about the variation | change_quantity of an azimuth | direction. It is a figure which shows the table in which the priority of the image area estimated that the target object is contained was defined. 4 is a flowchart of image area detection processing by a computer of the navigation device 1. It is a chart which shows an example of a color range conversion table (time slot). It is a chart which shows an example of a color range conversion table (weather). It is a flowchart which shows the process of the computer of the navigation apparatus 1 in 2nd Embodiment. It is a figure for demonstrating the action | operation in 2nd Embodiment. It is a figure for demonstrating the action | operation in 2nd Embodiment. It is a figure for demonstrating the action | operation in 2nd Embodiment. It is a figure for demonstrating the action | operation in 2nd Embodiment. It is a flowchart which shows the process of the computer of the navigation apparatus 1 in 3rd Embodiment.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 ... Navigation apparatus, 2 ... Server, 10 ... Remote control sensor, 11 ... Touch panel,
12 ... voice recognition unit, 12a ... microphone, 13 ... voice synthesis unit, 13a ... speaker,
14 ... display device, 14a ... display, 15 ... position detector, 16 ... external storage device,
DESCRIPTION OF SYMBOLS 17 ... Communication apparatus connection apparatus, 18 ... Control circuit, 19 ... Car-mounted camera, 20 ... Database.

Claims (9)

A vehicle periphery photographing device that photographs a vehicle periphery using a plurality of cameras mounted on the vehicle,
Direction change amount calculation means for calculating the amount of change in the direction of the vehicle at the start and end of the voice recognition function for recognizing the voice spoken by the user;
Image capturing means for capturing images obtained by photographing the periphery of the vehicle with a plurality of cameras at a predetermined timing based on the user's utterance;
The object recognized by the voice recognition based on the amount of change in the direction of the own vehicle, the direction of the object recognized by the voice recognition, and the predetermined timing at which the image is captured by the image capturing means. Storage means for storing a table in which priority levels of images estimated to be included are defined;
Priority determining means for determining the priority of the images captured by the image capturing means in the order in which it is estimated that the object recognized by the speech recognition is included using the table stored in the storage means. When,
Image area detecting means for detecting an image area estimated as an object recognized by the voice recognition from the images captured by the image capturing means;
When a plurality of image areas are detected by the image area detection unit, the plurality of image area candidates are displayed on the display unit according to the priority determined by the priority determination unit, and displayed on the display unit A vehicle periphery photographing apparatus comprising: selection means for selecting an object intended by a user from an image area. 2. The vehicle periphery photographing apparatus according to claim 1, wherein the predetermined timing is when voice recognition starts or when voice recognition ends. First azimuth calculating means for calculating the azimuth of the vehicle at the start of the voice recognition;
Second azimuth calculating means for calculating the azimuth of the vehicle at the end of the voice recognition,
The azimuth change amount calculating means starts and ends the voice recognition based on the azimuth of the own vehicle calculated by the first azimuth calculating means and the azimuth of the own vehicle calculated by the second azimuth calculating means. The vehicle periphery photographing apparatus according to claim 1, wherein a change amount of the direction of the own vehicle at the time is calculated. Situation acquisition means for acquiring a situation at the time of photographing around the vehicle;
Color correction means for correcting the color detection range recognized by the voice recognition according to the situation at the time of shooting around the vehicle acquired by the situation acquisition means;
The image area detection means is an image estimated as an object recognized by the speech recognition from images captured by the image capture means based on the color detection range corrected by the color correction means. The vehicle periphery photographing apparatus according to any one of claims 1 to 3, wherein an area is detected. The situation acquisition means acquires a time zone at the time of shooting an object around the vehicle,
5. The vehicle periphery photographing apparatus according to claim 4, wherein the color correction unit corrects a color detection range recognized by the voice recognition in accordance with a time zone acquired by the situation acquisition unit. The situation acquisition means acquires the weather at the time of photographing the object around the vehicle,
6. The vehicle periphery photographing apparatus according to claim 4, wherein the color correction unit corrects a color detection range recognized by the voice recognition according to the weather acquired by the situation acquisition unit. Communication means for communicating with a server having a search function for searching for detailed information;
Transmitting means for transmitting image data of the image area of the object selected by the selecting means to the server via the communication means;
Obtaining means for obtaining detailed information of the object searched by the server based on the image data transmitted from the transmitting means via the communication means;
The vehicle periphery photographing apparatus according to claim 1, further comprising: an information providing unit that provides a user with detailed information about the object acquired by the acquiring unit. A vehicle periphery photographing method for photographing a periphery of a vehicle using a plurality of cameras mounted on the vehicle,
Calculate the amount of change in the direction of the vehicle at the start and end of the voice recognition function that recognizes the voice spoken by the user,
Capture images taken by a plurality of cameras around the vehicle at a predetermined timing based on the user's utterance,
Image priority estimated to include the object recognized by the voice recognition based on the amount of change in the direction of the vehicle, the direction of the object recognized by the voice recognition, and the timing of capturing the image Determining the priority of the images captured by the image capturing means in the order in which it is estimated that the object recognized by the speech recognition is included using the table defined by
Detecting an image area estimated as an object recognized by the voice recognition from the captured image;
When a plurality of image areas are detected, the plurality of image area candidates are displayed on a display unit according to the determined priority, and an object intended by the user is selected from the image areas displayed on the display unit A vehicle periphery photographing method. 9. The vehicle periphery photographing method according to claim 8, wherein the predetermined timing is when voice recognition starts or when voice recognition ends.

Images