JP2023162002A - Electronic apparatus, method, and program - Google Patents

Abstract

To simplify determination of positional deviation on a route of a movable body.SOLUTION: An electronic apparatus 100 comprises, for example: a position calculation unit 154 that acquires first appearance position information on an appearance position of a traffic light indicated by an image picked up by an imaging unit 110 mounted on a movable body; a storage unit 140 that stores information on a reference imaging position that is a position on a predetermined route where the movable body picks up an image of the traffic light; and a determination unit 155 that determines a state of deviation of the movable body from the predetermined route on the basis of the first appearance position information and second appearance position information that is the position of the traffic light in the image picked up at the reference imaging position.SELECTED DRAWING: Figure 2

Description

TECHNICAL FIELD This application relates to electronic devices, methods, and programs.

BACKGROUND ART Conventionally, in-vehicle equipment mounted on a moving object uses a technique for specifying the position of the moving object. Patent Document 1 discloses that objects on a road are detected based on images taken by a camera mounted on a moving object, and are detected according to the movement of the moving object, and the distance between the objects and the object on a map is calculated. It is disclosed that the position of a moving object is determined based on the following.

Japanese Patent Application Publication No. 2015-175690

In the prior art, a plurality of different intervals must be detected in order to determine the current position of a mobile object from the intervals at which traffic lights are present. For this reason, in the conventional technology for determining the current position of a moving object, there is a need to simplify the determination of positional deviation in the route of the moving object.

An electronic device according to one aspect includes a position calculation unit that acquires first appearance position information regarding an appearance position of a traffic light indicated by an image captured by an imaging unit mounted on a moving body; a storage unit that stores information on a reference imaging position that is a position at which a body images the traffic light; and a second appearance position information that is the position of the traffic light in the first appearance position information and the image taken at the reference imaging position. and a determination unit that determines a deviation state of the moving body from the predetermined route based on the following.

In a method according to one aspect, a computer acquires first appearance position information regarding the appearance position of a traffic light indicated by an image captured by an imaging unit mounted on a moving object, and storing information on a reference imaging position, which is a position where the body images the traffic light, in a storage unit; and storing information on the first appearance position information and a second appearance, which is the position of the traffic light in the image taken at the reference imaging position. and determining a deviation state of the mobile body from the predetermined route based on the position information.

According to one aspect, a program causes a computer to obtain first appearance position information regarding the appearance position of a traffic light indicated by an image captured by an imaging unit mounted on a moving body, and to acquire first appearance position information regarding the appearance position of a traffic light shown in an image captured by an imaging unit mounted on a moving object, and storing information on a reference imaging position, which is a position where the body images the traffic light, in a storage unit; and storing information on the first appearance position information and a second appearance, which is the position of the traffic light in the image taken at the reference imaging position. and determining a deviation state of the mobile body from the predetermined route based on the position information.

FIG. 1 is a diagram for explaining an overview of an electronic device according to an embodiment. FIG. 2 is a diagram showing an example of the configuration of the electronic device shown in FIG. 1. FIG. 3 is a diagram for explaining an example of change information on the appearance position of a traffic light. FIG. 4 is a diagram for explaining an example of a deviation state based on the change information and reference information shown in FIG. 3. FIG. 5 is a diagram for explaining an example of a delay based on the change information and reference information shown in FIG. 3. FIG. 6 is a flowchart illustrating an example of a processing procedure executed by the control unit of the electronic device according to the embodiment. FIG. 7 is a diagram for explaining an example of the traveling environment of the electronic device according to the embodiment. FIG. 8 is a diagram for explaining an example of the operation of the electronic device according to the embodiment. FIG. 9 is a diagram for explaining an example of evaluation of the electronic device according to the embodiment. FIG. 10 is a diagram illustrating an example of a system configuration of an electronic device according to a modification of the embodiment.

A plurality of embodiments for implementing electronic devices, methods, programs, etc. according to the present application will be described in detail with reference to the drawings. Note that the present application is not limited by the following explanation. Furthermore, the constituent elements in the following description include those that can be easily assumed by those skilled in the art, those that are substantially the same, and those that are within the so-called equivalent range. In the following description, similar components may be denoted by the same reference numerals. Furthermore, duplicate explanations may be omitted.

FIG. 1 is a diagram for explaining an overview of an electronic device according to an embodiment. As shown in FIG. 1, the electronic device 100 is mounted on a moving body 1000. Electronic device 100 includes an on-vehicle device mounted on mobile object 1000. The mobile object 1000 includes, for example, a vehicle that can be moved by automatic driving, manual driving by a driver, or the like. Vehicles include, for example, buses, trucks, cars, and the like. In this embodiment, a case will be described in which the mobile object 1000 is a vehicle that automatically travels along the cruise route 1000R, but the present invention is not limited to this. The cruise route 1000R is an example of a periodic route that travels around a predetermined route at a predetermined time, and is an example of a predetermined route. The mobile object 1000 may include an airplane, a ship, a motorcycle, a drone, and the like.

In the example shown in FIG. 1, a mobile object 1000 is traveling in a travel lane 2000 on a cruise route 1000R. The driving lane 2000 has one or more lanes 2100 in a traveling direction 2000D. The mobile object 1000 is an automatic cruise vehicle that repeatedly travels on the same cruise route. In the driving lane 2000, traffic lights 3000 are installed near intersections, crosswalks, and the like. Therefore, on the cruise route 1000R, the mobile body 1000 moves through a location where the traffic light 3000 is present in front of the mobile body 1000 in the traveling direction D. In this embodiment, the electronic device 100 provides a technique for simplifying the determination of the current position of the moving body 1000 based on the traffic light 3000 indicated by the image data of the foreground of the moving body 1000.

FIG. 2 is a diagram showing an example of the configuration of the electronic device 100 shown in FIG. 1. As shown in FIG. As shown in FIG. 2, the electronic device 100 includes an imaging section 110, a sensor section 120, a communication section 130, a storage section 140, and a control section 150. The control section 150 is electrically connected to the imaging section 110, the sensor section 120, the communication section 130, the storage section 140, and the like.

Note that in this embodiment, a configuration in which the electronic device 100 includes an imaging section 110, a sensor section 120, a communication section 130, a storage section 140, and a control section 150 will be described, but the present invention is not limited to this. For example, the electronic device 100 may include a communication section 130, a storage section 140, and a control section 150, and may be configured to acquire information from an imaging device, a sensor, etc. outside the device.

The imaging unit 110 is provided near the windshield of the moving body 1000, on the roof, etc., so as to be able to image the foreground of the moving body 1000. The imaging unit 110 can electronically capture image data using an image sensor such as a CCD (Charge Coupled Device Image Sensor) or a CMOS (Complementary Metal Oxide Semiconductor). The imaging direction of the imaging unit 110 includes a direction in which the traffic light 3000 can be imaged in the traveling direction 2000D of the moving body 1000. The imaging unit 110 can image the foreground of the moving body 1000 in real time at a predetermined frame rate, and can supply the captured image data to the control unit 150.

The sensor unit 120 detects sensor information that can identify the state of the moving body 1000. The sensor unit 120 can use a sensor such as a position sensor or a gyro sensor, for example. Examples of the position sensor include a sensor that obtains a position in absolute coordinates, such as a GPS (Global Positioning System) receiver. The sensor unit 120 can supply sensor information including the position of the moving body 1000 (the position of the electronic device 100), angular velocity, etc. to the control unit 150. Thereby, the control unit 150 can acquire self-position information of the mobile body 1000 based on the sensor information.

The communication unit 130 can communicate with other communication devices, for example. The communication unit 130 can support various communication standards. The communication unit 130 can transmit and receive various data via, for example, a wired or wireless network. The communication unit 130 can supply the received data to the control unit 150. The communication unit 130 can transmit data to a destination specified by the control unit 150. The communication unit 130 can support CAN (Controller Area Network) communication. When the mobile object 1000 has a function of estimating its own position using an on-vehicle sensor, the communication unit 130 can receive self-position information from the mobile object 1000 and supply it to the control unit 150. In-vehicle sensors include, for example, GPS, IMU (Internal Measurement Unit), and the like. The self-location information includes information that allows identification of the self-location, movement direction, etc. of the mobile body 1000 that is a management target of the electronic device 100.

The storage unit 140 can store programs and data. The storage unit 140 is also used as a work area for temporarily storing processing results of the control unit 150. The storage unit 140 may include any non-transitory storage medium such as a semiconductor storage medium and a magnetic storage medium. Storage unit 140 may include multiple types of storage media. The storage unit 140 may include a combination of a portable storage medium such as a memory card, an optical disk, or a magneto-optical disk, and a storage medium reading device. The storage unit 140 may include a storage device used as a temporary storage area, such as a RAM (Random Access Memory).

The storage unit 140 can store various data such as a program 141, cruise data 142, change information 143, appearance position information 144, support information 145, evaluation information 146, and image data D10. The program 141 is a program that causes the control unit 150 to execute functions for realizing processes related to various operations of the electronic device 100. The cruise data 142 includes data regarding the cruise route 1000R (periodic route) of the mobile object 1000, the cruise schedule, the position of the traffic light 3000 on the cruise route 1000R, and the like. The cruise data 142 is data obtained from a database, a server, etc. via the communication unit 130. In this embodiment, the cruise data 142 includes cruise route information D21 and one or more reference information D22. The cruise route information D21 has information regarding the cruise route of the mobile object 1000. The reference information D22 is provided in association with the traffic light 3000 on the cruise route 1000R, and includes the reference appearance position of the traffic light 3000 on the image, the position and lane of the mobile object 1000 at the traffic light 3000, and the state of the traffic light 3000 at the time of passage. has information that allows identification. The reference information D22 includes information on a reference imaging position, which is a position where the moving object 1000 images the traffic light 3000 on the cruise route 1000R.

The storage unit 140 may store machine learning data, for example. Here, the machine learning data may be data generated by machine learning. Machine learning data may include parameters generated by machine learning. Furthermore, machine learning may be based on AI (Artificial Intelligence) technology that enables specific tasks to be executed through training. More specifically, machine learning may be a technique in which an information processing device such as a computer learns a large amount of data and automatically constructs algorithms or models that perform tasks such as classification and/or prediction. In this specification, machine learning may be included as part of AI. In this specification, machine learning may include supervised learning in which features or rules of input data are learned based on correct data. Furthermore, machine learning may include unsupervised learning in which features or rules of input data are learned without correct data. Furthermore, machine learning may include reinforcement learning, which learns the characteristics or rules of input data by giving rewards or punishments. Furthermore, in this specification, machine learning may be any combination of supervised learning, unsupervised learning, and reinforcement learning.

The concept of machine learning data in this embodiment may include an algorithm that outputs a predetermined inference (estimate) result using an algorithm learned on input data. This embodiment uses a linear regression that predicts the relationship between a dependent variable and an independent variable, a neural network (NN) that is a mathematical model of neurons in the human nervous system, and a minimum square that calculates by squaring the error. Other suitable algorithms can be used, such as multiplication, decision trees that solve the problem in a tree structure, and regularization that transforms data in a predetermined manner. This embodiment may use deep neural network learning, which is a type of neural network. Deep neural network learning is a type of neural network, and a neural network that generally has a deep structure with one or more middle layers is called deep learning. Deep learning is widely used as an algorithm for AI.

The change information 143 is information that can identify a change in the appearance position of the same traffic light 3000 indicated by the image data D10. The change information 143 includes, for example, the appearance position of the traffic light 3000 in the image, a direction vector indicating a change in position, a trajectory, and the like. The appearance position information 144 includes information that allows identification of a change in the appearance position of the traffic light 3000 in an image in the driving lane 2000 in which the moving object 1000 is traveling, information that allows identification of the position of the moving object 1000 that captured the traffic light 3000, and the like. The support information 145 includes information that can support operation based on the route deviation status, delay status, etc. of the moving body 1000. The evaluation information 146 includes information that can identify the evaluation result of the travel based on the path deviation state, delay state, etc. of the moving object 1000. The image data D10 is data that allows identification of an image captured by the imaging unit 110. The storage unit 140 can store a plurality of image data D10 in chronological order.

Control unit 150 is a calculation processing device. The arithmetic processing unit includes, for example, a CPU (Central Processing Unit), a SoC (System-on-a-Chip), an MCU (Micro Control Unit), an FPGA (Field-Programmable Gate Array), and a coprocessor. including but not limited to these Not limited. The control unit 150 can comprehensively control the operation of the electronic device 100 and realize various functions.

Specifically, the control unit 150 can execute instructions included in the program 141 stored in the storage unit 140 while referring to information stored in the storage unit 140 as necessary. The control unit 150 controls the functional units according to data and instructions, thereby realizing various functions. The functional units include, for example, the sensor unit 120 and the communication unit 130, but are not limited thereto.

The control unit 150 includes functional units such as an image recognition unit 151, an acquisition unit 152, an estimation unit 153, a position calculation unit 154, a determination unit 155, a detection unit 156, a movement support unit 157, and an evaluation unit 158. By executing the program 141, the control unit 150 controls the functions of the image recognition unit 151, the acquisition unit 152, the estimation unit 153, the position calculation unit 154, the determination unit 155, the detection unit 156, the motion support unit 157, the evaluation unit 158, etc. Realize the department. The program 141 causes the control unit 150 of the electronic device 100 to function as an image recognition unit 151, an acquisition unit 152, an estimation unit 153, a position calculation unit 154, a determination unit 155, a detection unit 156, an operation support unit 157, and an evaluation unit 158. This is a program for

The image recognition unit 151 recognizes the traffic light 3000 from the foreground image of the moving object 1000 indicated by the image data D10 captured by the imaging unit 110. The image recognition unit 151 has a function of recognizing the presence or absence of the traffic light 3000 in an image, the state of the traffic light 3000, etc. using, for example, a learning model that has learned the traffic light 3000, an image recognition program, or the like. The state of the traffic light 3000 includes, for example, whether the traffic light 3000 is lit, the color of the lit signal, and the like. The image recognition unit 151 performs image recognition every time the image data D10 is supplied from the imaging unit 110. For example, when capturing an image of the foreground from the moving object 1000, the traffic light 3000 is likely to be captured in the upper half of the image. For this reason, the image recognition unit 151 may be configured to perform recognition processing for the traffic light 3000 on the area where the traffic light 3000 may be imaged in the image indicated by the image data D10, or may be configured to perform recognition processing on the traffic light 3000 for all areas of the image. A configuration may also be adopted in which recognition processing for the traffic light 3000 is performed. The image recognition unit 151 stores the recognition result of the image data D10 in the storage unit 140. The recognition result includes, for example, whether the image shows the traffic light 3000 or not, and if so, the state of the traffic light 3000.

The acquisition unit 152 acquires self-position information of the moving body 1000 based on the sensor information that the sensor unit 120 supplies to the control unit 150. The acquisition unit 152 recognizes the current location of the mobile object 1000 based on the acquired self-location information. The acquisition unit 152 may acquire the self-position using a highly accurate map using a technique such as SLAM (Simultaneous Localization and Mapping), for example. The acquisition unit 152 has a function of acquiring a velocity pulse signal, an angular velocity signal, an acceleration signal, etc. from the mobile object 1000 and estimating the autonomous navigation position. The acquisition unit 152 can acquire self-location information based on the satellite position and autonomous navigation position indicated by the self-location information. Self-location information can identify road links and intersection names, but the accuracy of the position of the lane 2100 in which the vehicle is traveling is uncertain.

The estimation unit 153 estimates change information 143 of the appearance position of the traffic light 3000 in the image based on a plurality of image data D10 obtained by capturing images of the same traffic light 3000 at different timings by the imaging unit 110. The estimation unit 153 estimates change information 143 indicating a change in the appearance position of the traffic light 3000 in the image based on a plurality of image data D10 obtained by capturing images of the same traffic light 3000 at different timings. The estimation unit 153 can provide the estimated change information 143 to the position calculation unit 154.

The position calculation unit 154 acquires appearance position information 144 (first appearance position information) regarding the appearance position of the traffic light 3000 indicated by the image D11 of the image data D10 captured by the imaging unit 110 mounted on the mobile object 1000. The position calculation unit 154 determines the location of the traffic light in the image D11 in the driving lane 2000 in which the mobile body 1000 is traveling, based on the appearance position of the traffic light 3000 indicated by the plurality of image data D10 captured by the imaging unit 110 mounted on the mobile body 1000. Appearance position information 144 that can identify 3000 changes is calculated. The position calculation unit 154 calculates the appearance position of the traffic light 3000 recognized by the image recognition unit 151 in the image indicated by the image data D10, and calculates the appearance position where the change from the appearance position of the continuous traffic light 3000 indicated by the change information 143 can be identified. Calculate position information 144. By associating the self-position information with the appearance position information 144 of the traffic light 3000 in the image D11, the position calculation unit 154 can associate the coordinates indicated by the appearance position information 144 with the coordinates indicated by the self-position information. The appearance position information 144 may be information that associates the coordinates indicated by the appearance position information 144 with the coordinates indicated by the self-location information. For example, the self-position estimated by the mobile object 1000 is information in which the position of the lane 2100 is uncertain even if the driving lane 2000 can be specified. Therefore, the position calculation unit 154 makes it possible to calculate the position of the lane 2100, which is uncertain based on the own position, by referring to the own position and the change in the appearance position of the traffic light 3000. The position calculation unit 154 uses, for example, a machine learning program, a lookup table, a calculation program, etc., to determine the moving body corresponding to the change information 143 of the appearance position of the traffic light 3000 and the reference information D22 indicating the driving lane 2000 at that position. 1000 in the driving lane 2000 is calculated.

The determination unit 155 determines the deviation state of the mobile object 1000 from the cruise route 1000R based on the first appearance position information and the second appearance position information, which is the position of the traffic light 3000 in the image D11 captured at the reference imaging position. . The determination unit 155 compares the appearance position information 144 calculated by the position calculation unit 154 with the reference information D22 of the moving object 1000 stored in the storage unit 140, and determines the deviation state of the cruise route 1000R (route) of the moving object 1000. Determine. The determination unit 155 determines the state of deviation between the position of the mobile object 1000 in the vicinity of the traffic light 3000 and the reference position indicated by the cruise route 1000R. An example of determining the deviation state will be described later. The determination unit 155 stores the deviation state of the cruise route 1000R in the storage unit 140 in association with the reference information D22 of the traffic light 3000. Thereby, the determination unit 155 can store the determination result corresponding to the traffic light 3000 in the storage unit 140.

The detection unit 156 detects the lighting state of the traffic light 3000 from the image data D10. The lighting state of the traffic light 3000 includes the lighting timing of the traffic light. The detection unit 156 detects the lighting timing of the traffic light 3000 based on the state of the traffic light 3000 recognized by the image recognition unit 151. The detection unit 156 can detect, for example, the color of the signal that the traffic light 3000 is lit, the lighting or non-lighting of a predetermined signal, etc. as the lighting timing. For example, when the mobile object 1000 moves along the cruise route 1000R at the scheduled time, the state of the traffic light 3000 at the time of passing the traffic light 3000 is likely to be the same. Therefore, the detection unit 156 can provide the detection result for determining the delay of the moving body 1000 by detecting the lighting timing of the traffic light 3000 from the image data D10.

The determining unit 155 compares the lighting timing of the traffic light 3000 detected by the detecting unit 156 with the reference lighting timing of the reference information D22 stored in the storage unit 140 to determine the delay state of the mobile object 1000. The delay state of the mobile object 1000 includes, for example, a state in which the movement of the mobile object 1000 is maintained as it is, moves faster, moves slower, etc. with respect to the blinking timing of the traffic light 3000 on the cruise route 1000R. The determination unit 155 stores the delay state of the mobile object 1000 in the storage unit 140 in association with the reference information D22 of the traffic light 3000.

The operation support unit 157 executes a process for supporting the operation of the mobile object 1000 based on at least one of the deviation state and delay state of the cruise route 1000R determined by the determination unit 155. Processes that support the operation of the mobile object 1000 include, for example, processing for notifying the mobile object 1000, a management device that manages the mobile object 1000, etc. of the deviation state of the cruise route 1000R, and changing the cruise route 1000R based on the deviation state. This includes processing for instructing, processing for resolving delay states, etc.

The evaluation unit 158 evaluates the travel results such as the travel trajectory and the cruise result of the mobile body 1000 based on at least one of the deviation state and delay state of the cruise route 1000R of the mobile body 1000 determined by the determination unit 155. The evaluation unit 158 evaluates the traveling trajectory, cruising time, etc. of the mobile object 1000 based on the deviation state of one or more traffic lights 3000 on the cruising route 1000R, and stores evaluation information 146 indicating the evaluation result in the storage unit 140. The evaluation information 146 includes, for example, information such as the degree of coincidence between the cruise route 1000R and the route actually traveled, and the delay status of the plurality of traffic lights 3000. The evaluation information 146 is information that can be used to review or change the cruise of the mobile object 1000 in the case of automatic driving. The evaluation information 146 is information that can be used for guidance, evaluation, etc. of the driver of the mobile object 1000 in the case of manual driving. The evaluation unit 158 can provide the evaluation information 146 to the mobile object 1000, an external management device, a server, etc. via the communication unit 130. The evaluation unit 158 can use the evaluation results of the running trajectory to improve the running trajectory. For example, when the mobile object 1000 is driven manually, the evaluation unit 158 outputs the evaluation result of the travel trajectory as an evaluation rank or score for use in driver education. For example, when the mobile object 1000 is autonomously driving, the evaluation unit 158 outputs the evaluation result of the travel trajectory as information applicable to improving the automatic driving algorithm.

The functional configuration example of the electronic device 100 according to the present embodiment has been described above. Note that the above configuration described using FIG. 2 is just an example, and the functional configuration of the electronic device 100 according to the present embodiment is not limited to the example. The functional configuration of the electronic device 100 according to this embodiment can be flexibly modified according to specifications and operation.

(Example of determining deviation state and delay state)
FIG. 3 is a diagram for explaining an example of the change information 143 of the appearance position of the traffic light 3000. FIG. 4 is a diagram for explaining an example of a deviation state based on the change information 143 shown in FIG. 3 and the reference information D22. FIG. 5 is a diagram for explaining an example of a delay based on the change information 143 and the reference information D22 shown in FIG. 3.

As shown in FIG. 3, when the electronic device 100 acquires the image data D10 captured by the imaging unit 110 at time T1, it recognizes the appearance position P11 and the appearance position P21 of the traffic light 3000 in the image D11 indicated by the image data D10. . The appearance position P11 is the position of a traffic light 3000 that exists near the mobile body 1000 in the driving lane 2000 in which the mobile body 1000 is moving. The appearance position P21 is the position of the traffic light 3000 that is located far in front of the mobile object 1000. The electronic device 100 stores the recognized appearance position P11 and appearance position P21 in the change information 143 of the storage unit 140 in association with the image data D10 at time T1.

Thereafter, at time T2, electronic device 100 has moved to a position ahead of the position at time T1. When the electronic device 100 acquires the image data D10 captured by the imaging unit 110, it recognizes the appearance position P12 and the appearance position P22 of the traffic light 3000 in the image D11 indicated by the image data D10. The appearance position P21 is the position of a traffic light 3000 that exists near the mobile body 1000 in the travel lane 2000 in which the mobile body 1000 is moving. The appearance position P22 is the position of the traffic light 3000 that is located far in front of the mobile object 1000.

The electronic device 100 calculates the appearance position information 144 including a vector directed from the appearance position P11 at time T1 indicated by the change information 143 to the appearance position P12 at time T2. The electronic device 100 calculates appearance position information 144 including a vector from appearance position P21 at time T1 to appearance position P22 at time T2. In the example shown in FIG. 3, since the appearance position P21 and the appearance position P22 are far from the mobile object 1000, the electronic device 100 does not calculate the appearance position information 144 of the appearance position P22.

The electronic device 100 identifies the traffic light 3000 indicated by the image data D10 from the cruise data 142, and extracts the reference information D22 corresponding to the traffic light 3000 from the storage unit 140. The electronic device 100 may identify a traffic light 3000 near the self-position of the mobile body 1000 from the cruise data 142, or may identify it from the cruise data 142 based on the order in which the traffic light 3000 is passed.

As shown in FIG. 4, in the image D12 showing the map corresponding to the image data D10, the electronic device 100 distinguishes between the vector 144V indicated by the calculated appearance position information 144 and the reference vector D22A or D22B indicated by the reference information D22. The deviation state G is determined by comparing. For example, when the reference information D22 indicates the reference vector D22A and the vector 144V overlaps with the vector, the electronic device 100 determines that the reference vector D22A and the vector 144V do not deviate from each other.

For example, when the reference information D22 indicates the reference vector D22A and the vector 144V deviates to the right with respect to the vector, the electronic device 100 determines the deviation state G between the reference vector D22A and the vector 144V. In this case, the moving object 1000 is shown to be moving slightly to the right from the cruise route 1000R. For example, when the reference information D22 indicates the reference vector D22B and the vector 144V deviates to the left with respect to the vector, the electronic device 100 determines the deviation state G between the reference vector D22B and the vector 144V. In this case, the moving object 1000 is shown to be moving slightly to the left from the cruise route 1000R. Thereby, the electronic device 100 can contribute to identifying the deviation of the driving position of the lane 2100 in the driving lane 2000, the lane, etc. by determining the deviation state G in the driving lane 2000 in which the mobile object 1000 is traveling.

Note that in the example shown in FIG. 4, the deviation state G is determined as the state at the appearance position P12 at time T2, but the present invention is not limited to this. The deviation state G may be determined, for example, by determining the state where the distance between vectors is the maximum or minimum, or may be determined as the average of the states. Furthermore, the electronic device 100 can calculate the magnitude of the deviation state G as the deviation amount and associate it with the determination result.

Further, as shown in FIG. 5, the reference information D22 includes a determination area D22C, a determination area D22D, which is obtained by converting the relationship between the reference lighting timing at which the traffic light 3000 blinks and the appearance position of the traffic light 3000 onto the map of the image D12. and information indicating the determination area D22E. The blinking of the traffic light 3000 is a timing when it is possible to drive while paying attention to other traffic. The determination area D22C is an area through which the moving object 1000 can pass before the traffic light 3000 switches by continuing to drive. The determination area D22D is an area through which the moving object 1000 can pass before the traffic light 3000 switches by running in a hurry. The determination area D22E is an area where the moving object 1000 is decelerated and waits for the next lighting of the traffic light 3000, since the traffic light 3000 is switched.

The electronic device 100 determines the delay state based on the vector 144V indicated by the estimated appearance position information 144 and the determination region D22C, determination region D22D, and determination region D22E indicated by the reference information D22. For example, when the appearance position P12 of the traffic light 3000 in the image D12 shown by the image data D10 is the appearance position P12, the electronic device 100 indicates that the movement of the mobile object 1000 is to be maintained as it is because the appearance position P12 is located in the determination area D22C. It is determined to be in a delayed state. For example, when the appearance position P13 of the traffic light 3000 in the image D11 shown by the image data D10 is the appearance position P13, the electronic device 100 is in a delay state indicating that the movement of the mobile object 1000 is accelerated because the appearance position P13 is located in the determination area D22D. It is determined that For example, when the appearance position P14 of the traffic light 3000 in the image D11 shown by the image data D10 is the appearance position P14, the electronic device 100 is in a delay state indicating that the movement of the mobile object 1000 is slowed down because the appearance position P14 is located in the determination area D22E. It is determined that In this way, the electronic device 100 can support correction of the cruising state of the mobile object 1000 by determining the delay state with respect to the blinking timing of the traffic light 3000.

FIG. 6 is a flowchart illustrating an example of a processing procedure executed by the control unit 150 of the electronic device 100 according to the embodiment. The processing procedure shown in FIG. 6 is realized by the control unit 150 executing the program 141. The processing procedure shown in FIG. 6 is repeatedly executed by the control unit 150 when the mobile object 1000 is cruising.

As shown in FIG. 6, the control unit 150 of the electronic device 100 recognizes the traffic light 3000 from the image data D10 acquired from the imaging unit 110 (step S100). For example, the control unit 150 executes recognition processing to recognize the traffic light 3000 in the image D11 indicated by the acquired image data D10, and stores the recognition result in the storage unit 140. When the process of step S100 is completed, the control unit 150 advances the process to step S101.

The control unit 150 determines whether the traffic light 3000 is recognized from the image D11 (step S101). For example, if the recognition result in step S100 indicates that the traffic light 3000 has been recognized, the control unit 150 determines that the traffic light 3000 has been recognized from the image D11. When the control unit 150 determines that the traffic light 3000 is not recognized from the image D11 (No in step S101), the control unit 150 ends the processing procedure shown in FIG. 6. Further, when the control unit 150 determines that the traffic light 3000 is recognized from the image D11 (Yes in step S101), the process proceeds to step S102.

The control unit 150 determines whether the traffic light 3000 has been recognized (step S102). For example, the control unit 150 determines that the traffic light 3000 has been recognized when the traffic light 3000 has been recognized in the previous processing procedure, when the change information 143 associated with the traffic light 3000 is present, etc. When the control unit 150 determines that the traffic light 3000 has not been recognized, that is, it is the first recognition of the traffic light 3000 (No in step S102), the process proceeds to step S103.

The control unit 150 stores change information 143 indicating the appearance position of the traffic light 3000 in the image D11 in the storage unit 140 (step S103). For example, the control unit 150 stores change information 143 including the previous appearance position and recognition time of the traffic light 3000 in the storage unit 140. When the process of step S103 is completed, the control unit 150 ends the processing procedure shown in FIG. 6.

Further, when the control unit 150 determines that the traffic light 3000 has been recognized (Yes in step S102), the process proceeds to step S104. The control unit 150 estimates change information 143 of the appearance position of the traffic light 3000 in the image D11 (step S104). For example, the control unit 150 estimates change information 143 indicating a vector between the appearance position of the traffic light 3000 in the image D11 shown by the image data D10 and the previous appearance position of the same traffic light 3000. After storing the estimated change information 143 in the storage unit 140, the control unit 150 advances the process to step S105.

The control unit 150 acquires self-location information (step S105). For example, the control unit 150 acquires self-position information estimated by the mobile object 1000 via the communication unit 130 and stores it in the storage unit 140. When the process of step S105 is completed, the control unit 150 advances the process to step S106.

Based on the appearance position of the traffic light 3000 indicated by the image data D10, the control unit 150 calculates appearance position information 144 that can identify a change in the appearance position of the traffic light 3000 in the driving lane 2000 (step S106). For example, the control unit 150 uses the self-position indicated by the self-location information as a reference, and includes the past appearance position including the previous appearance indicated by the change information 143, the current appearance position of the traffic light 3000 in the image D11, and a vector 144V indicating the change thereof. Appearance position information 144 is calculated. When the process of step S106 is completed, the control unit 150 advances the process to step S107.

The control unit 150 compares the appearance position information 144 and the reference information D22 and determines the deviation state G and delay state of the route of the mobile object 1000 (step S107). For example, the control unit 150 extracts the reference information D22 corresponding to the traffic light 3000 from the cruise route 1000R of the mobile object 1000. The control unit 150 then compares the vector indicated by the appearance position information 144 and the reference vector indicated by the reference information D22 to determine the deviation state G. The control unit 150 compares the appearance position indicated by the appearance position information 144 with the determination area indicated by the reference information D22 to determine the delay state. When the control unit 150 stores the determination results of the deviation state G and the delay state in the storage unit 140, the control unit 150 advances the process to step S108.

The control unit 150 executes a process to support the operation of the mobile object 1000 based on the determined deviation state G and delay state (step S108). For example, the control unit 150 notifies the management device of the mobile object 1000 of the deviation state G of the cruise route 1000R by executing a process to support the operation of the mobile object 1000, and controls the cruise route 1000R based on the deviation state G. Direct changes. For example, the control unit 150 issues an instruction to change the operation plan so as to eliminate the delay state by executing processing to support the operation of the mobile object 1000. When the process of step S108 is completed, the control unit 150 advances the process to step S109.

The control unit 150 evaluates the driving result of the mobile body 1000 based on the deviation state G and the delay state of the cruise route 1000R of the mobile body 1000 (step S109). For example, the control unit 150 evaluates the travel trajectory, cruising time, etc. of the mobile object 1000 based on the deviation state G and delay state of the traffic light 3000, and stores evaluation information 146 indicating the evaluation result in the storage unit 140. When the process of step S109 ends, the control unit 150 ends the processing procedure shown in FIG. 6.

In the processing procedure shown in FIG. 6, when the traffic light 3000 is recognized from the image D11, the running of the mobile object 1000 is evaluated in step S109, but the present invention is not limited to this. For example, the processing procedure shown in FIG. 6 can be changed so that the processing in step S9 is executed when the mobile object 1000 finishes moving on the cruise route 1000R.

(Operation of electronic equipment)
FIG. 7 is a diagram for explaining an example of the traveling environment of the electronic device 100 according to the embodiment. FIG. 8 is a diagram for explaining an example of the operation of the electronic device 100 according to the embodiment. FIG. 9 is a diagram for explaining an example of evaluation of the electronic device 100 according to the embodiment.

The example shown in FIG. 7 shows a cruise route 1000R along which the mobile body 1000 carrying the electronic device 100 moves. A plurality of traffic lights 3000 are installed in the driving lane 2000 of the cruise route 1000R. When starting a cruise, the electronic device 100 stores the reference information D22 for each of the plurality of traffic lights 3000 on the cruise route 1000R in the storage unit 140 by storing cruise data 142 indicating the cruise route 1000R in the storage unit 140.

The moving object 1000 is moving in the traveling direction 2000D of the driving lane 2000 and is approaching the traffic light 3000. In this case, as shown in FIG. 8, the electronic device 100 recognizes the traffic light 3000 in the image D11 shown by the image data D10, and generates appearance position information 140 based on the change information 143 of the traffic light 3000 from the plurality of continuous image data D10. Calculate. The electronic device 100 compares the appearance position information 144 with the reference information D22 corresponding to the traffic light 3000 to determine the path deviation state G of the moving object 1000. In the example shown in FIG. 8, the electronic device 100 determines that the deviation state G of the change information 143 with respect to the reference vector D22A indicated by the reference information D22 of the traffic light 3000 is shifted to the right.

The electronic device 100 provides the mobile object 1000, the management device, etc. with support information 145 including the determination result of the deviation state with respect to the cruise route 1000R. The electronic device 100 can include, for example, support information 145 including information such as a moving direction and a route for the mobile object 1000 to return to the cruise route 1000R based on the cruise data 142, the deviation state, and the like. As a result, for example, the moving object 1000 that is automatically traveling can change its moving direction, moving speed, etc. so as to return to the cruise route 1000R based on the support information 145 that can resolve the fact that it is leaning to the right. I can do it.

The electronic device 100 determines the delay state with respect to the reference lighting timing indicated by the reference information D22 of the traffic signal 3000. As shown in FIG. 5, when the appearance position P12 indicated by the change information 143 exists in the determination area D22C of the reference information D22, the electronic device 100 enters a delay state indicating that the movement of the mobile object 1000 is to be maintained as it is. judge. In this case, there is no need to correct the cruise of the mobile object 1000, so the electronic device 100 continues the process.

Furthermore, when the appearance position P12 indicated by the change information 143 exists in the determination area D22D of the reference information D22, the electronic device 100 determines that the moving body 1000 is in a delay state indicating that the movement of the moving body 1000 is to be accelerated. In this case, the electronic device 100 provides the mobile object 1000, the management device, etc. with support information 145 instructing to increase the moving speed. As a result, the moving speed of the moving object 1000 increases within the legal speed limit, and the moving object 1000 can pass through the traffic light 3000 without stopping.

Further, when the appearance position P12 indicated by the change information 143 exists in the determination area D22E of the reference information D22, the electronic device 100 determines that the moving object 1000 is in a delay state indicating that the movement of the moving object 1000 is slowed down. In this case, the electronic device 100 provides the mobile object 1000, the management device, etc. with support information 145 instructing to slow down the moving speed. Thereby, the mobile object 1000 can drive safely by decelerating and waiting for the next green light from the traffic light 3000.

In order to support the operation of the mobile object 1000 based on the deviation state G, the electronic device 100 calculates the deviation value of the mobile object 1000 from the cruise route 1000R, and sends the support information 145 including the deviation value to the mobile object 1000. Provided to management equipment, etc. Thereby, the mobile body 1000 can correct the traveling position of the mobile body 1000 in the travel lane 2000 based on the deviation value of the support information 145.

The electronic device 100 evaluates the travel of the mobile body 1000 based on the determined path deviation state G and delay state of the mobile body 1000. The electronic device generates evaluation information 146 indicating the evaluation result for each traffic light, and stores it in the storage unit 140 in association with the cruise data 142. For example, as shown in FIG. 9, the electronic device 100 generates evaluation information 146 indicating evaluation results such as a tendency to shift to the right, a tendency to lag behind the reference lighting timing, and the like. For example, when the mobile body 1000 is in manual operation, the electronic device 100 can instruct the driver while driving by displaying the evaluation information 146 on the panel, display, etc. of the mobile body 1000. For example, when the mobile body 1000 is in automatic operation, the electronic device 100 can support the evaluation of the cruise route 1000R of the mobile body 1000 by providing the evaluation information 146 to the management device of the mobile body 1000.

The electronic device 100 according to the embodiment can determine the path deviation state of the moving body 1000 by simply calculating the change in the appearance position of the traffic light 300 in the image D11 shown by the plurality of image data D10. Thereby, the electronic device 100 can improve the accuracy of determining positional deviation on the cruise route 1000R even in a moving environment where the detection accuracy of the self-position of the mobile object 1000 is low and only image data D10 can be obtained.

(Modified example)
Although the electronic device 100 described above is mounted on the mobile body 1000, the present invention is not limited thereto. The electronic device 100 may be realized by, for example, a server device provided outside the mobile object 1000. FIG. 10 is a diagram illustrating an example of a system configuration of an electronic device 100 according to a modification of the embodiment. As shown in FIG. 10, the electronic device 100 is configured to be able to communicate with a mobile object 1000 via a network 400. The electronic device 100 may be realized by, for example, a personal computer, a tablet terminal, a smartphone, or the like. Electronic device 100 acquires image data D10 from imaging unit 110 of mobile object 1000 via network 400. The electronic device 100 determines the path deviation state and delay state of the mobile object 1000 based on the acquired image data D10, and provides support information 145 and evaluation information 146 based on the determination results to the mobile object 1000 via the network 400. do. The electronic device 100 may provide the mobile object 1000 with information indicating the determination results of the deviation state and the delay state.

The specific embodiments have been described in order to provide a complete and clear disclosure of the technology as claimed. However, the appended claims should not be limited to the above-mentioned embodiments, but include all modifications and substitutions that can be created by a person skilled in the art within the scope of the basic matters presented in this specification. should be configured to embody a specific configuration. Those skilled in the art can make various changes and modifications to the contents of the present disclosure based on the present disclosure. Accordingly, these variations and modifications are included within the scope of this disclosure. For example, in each embodiment, each functional unit, each means, each step, etc. may be added to other embodiments so as not to be logically contradictory, or each functional unit, each means, each step, etc. of other embodiments may be added to other embodiments to avoid logical contradiction. It is possible to replace it with Further, in each embodiment, it is possible to combine or divide a plurality of functional units, means, steps, etc. into one. Further, each embodiment of the present disclosure described above is not limited to being implemented faithfully to each described embodiment, but may be implemented by combining each feature or omitting a part as appropriate. You can also do that.

100 Electronic device 110 Imaging section 120 Sensor section 130 Communication section 140 Storage section 141 Program 142 Cruise data 143 Change information 144 Appearance position information 145 Support information 146 Evaluation information 150 Control section 151 Image recognition section 152 Acquisition section 153 Estimation section 154 Position calculation section 155 Judgment unit 156 Detection unit 157 Operation support unit 158 Evaluation unit 1000 Moving object 1000R Cruising route 2000 Driving lane 2100 Lane 3000 Traffic light D10 Image data D21 Cruising route information D22 Reference information G Displacement state P11, P12 Appearance position

Claims (8)

a position calculation unit that acquires first appearance position information regarding the appearance position of a traffic light indicated by an image captured by an imaging unit mounted on the mobile body;
a storage unit that stores information on a reference imaging position, which is a position at which the moving body images the traffic light on a predetermined route;
a determination unit that determines a deviation state of the moving body from the predetermined route based on the first appearance position information and second appearance position information that is the position of the traffic light in the image captured at the reference imaging position; ,
Electronic equipment equipped with The electronic device according to claim 1,
further comprising a detection unit that detects a lighting state of the traffic light from the image,
The storage unit has information on a reference lighting state of the traffic light that serves as a reference for the mobile object,
The determination unit compares the detected lighting state of the traffic light with the reference lighting state to determine a delay state of the mobile object. The electronic device according to claim 2,
The mobile body is a mobile body that travels on a periodic route,
The reference imaging position and the reference lighting state, which serve as a reference for the moving object, are values set at the traffic light on the periodic route. The electronic device. The electronic device according to claim 3,
The electronic device further includes an operation support unit that executes a process for supporting the operation of the mobile body based on at least one of the calculated deviation state and the delay state. The electronic device according to claim 4,
further comprising an acquisition unit that acquires self-location information of the mobile body based on sensor information of the mobile body,
The position calculation unit is configured to calculate a third appearance position in which a change in the appearance position of the traffic light in the image can be identified in a travel lane in which the mobile object travels, based on the appearance position of the traffic light in the image and the self-position information. Electronic equipment that calculates information. The electronic device according to claim 5,
The electronic device further includes an evaluation unit that evaluates a running result of the mobile body based on at least one of the deviation state and the delay state of the route of the mobile body determined by the determination unit. The computer is
Obtaining first appearance position information regarding the appearance position of a traffic light indicated by an image captured by an imaging unit mounted on the mobile object;
storing information on a reference imaging position, which is a position at which the moving body images the traffic light on a predetermined route, in a storage unit;
Determining a deviation state of the mobile body from the predetermined route based on the first appearance position information and second appearance position information that is the position of the traffic light in the image captured at the reference imaging position;
method including. to the computer,
Obtaining first appearance position information regarding the appearance position of a traffic light indicated by an image captured by an imaging unit mounted on the mobile object;
storing information on a reference imaging position, which is a position at which the moving body images the traffic light on a predetermined route, in a storage unit;
determining a deviation state of the mobile body from the predetermined route based on the first appearance position information and second appearance position information that is the position of the traffic light in an image captured at the reference imaging position;
A program to run.

Images