JP6726075B2 - Image processing method, image processing apparatus and program - Google Patents

Description

The present invention relates to an image processing method, an image processing device and a program.

In recent years, general object recognition by machine learning technology using neural networks has been attracting attention because of its high performance.

However, in general object recognition by neural network, in order to bring out high recognition performance, it is necessary to perform learning processing using a large amount of images in which the names and types of objects to be recognized are added as annotations (correct answer information). There is.

Further, in machine learning, it is known that accuracy is improved if large-scale data (big data) is provided as learning data.

One method of collecting big data is to use outsourcing to a third party such as crowdsourcing. Crowdsourcing is a mechanism to request an unspecified number of people (workers) to perform simple tasks (tasks) through the Internet at low cost. Therefore, if data collection of big data is performed using crowdsourcing, tasks for individual data making up big data can be distributed and requested to a large number of workers, so that big data can be transmitted efficiently (relatively). It is cheap and can be collected in a short time.

For example, Patent Document 1 discloses a technique for realizing crowdsourcing with high work accuracy by a small number of people.

JP, 2013-197785, A

However, even if the technique disclosed in Patent Document 1 is used, when a high degree of recognition is required for the annotation work, there is a problem in that the annotation work is likely to cause individual differences among crowdsourcing workers. An example of a case in which a high degree of recognition is required for annotating work is, for example, annotating a dangerous area that is likely to be dangerous because a person crosses the front of the vehicle. Therefore, when a high degree of recognition is required for the work of adding annotations, there is a problem that the quality of the learning data obtained by crowdsourcing varies. When machine learning is performed using big data composed of learning data having variations in quality, the learning accuracy does not improve.

The present disclosure has been made in view of the above circumstances, and an object of the present disclosure is to provide an image processing method, an image processing device, and a program that can suppress variation in quality of learning data.

In order to achieve the above object, a task image processing method according to an aspect of the present invention is provided with a first annotation indicating two or more first areas, at least one of which is a person area, and is mounted on a vehicle. A plurality of time-sequential images captured by an in-vehicle camera, wherein the two or more first regions are present in the travel route of the vehicle, and the distance between the first regions is less than or equal to a threshold value. In the acquisition step of acquiring a plurality of images including at least the image of the first area, and in the plurality of images acquired in the acquisition step, the two or more first regions are sequentially traced back from the last time image in the time series. A determination step of determining each position, and a first image at the first first time when it is determined that the position of each of the two or more first regions is not in the travel route in the determination step among the plurality of images. And a second annotation indicating the second region determined in the determining step, and a determining step of determining a region between the two or more first regions in the identified first image as a second region. And a grant step of granting.

Note that these general or specific aspects may be realized by a recording medium such as a system, a method, an integrated circuit, a computer program or a computer-readable CD-ROM, and the system, the method, the integrated circuit, the computer. It may be realized by any combination of the program and the recording medium.

According to the present invention, it is possible to realize an image processing method and the like that can suppress variations in quality of learning data.

FIG. 1 is a diagram showing an example of a functional configuration of the image processing apparatus according to the first embodiment. FIG. 2 is a diagram showing an example of a plurality of images acquired by the annotation unit according to the first embodiment. FIG. 3 is an explanatory diagram of image processing performed by the annotation unit according to the first embodiment on the plurality of images shown in FIG. FIG. 4 is an explanatory diagram of an example of image processing performed by the annotation unit according to the first embodiment on the plurality of images shown in FIG. FIG. 5 is a diagram showing an example of a detailed functional configuration of the narrowing unit shown in FIG. FIG. 6 is an explanatory diagram of a first narrowing-down method of the narrowing-down portion according to the first embodiment. FIG. 7 is an explanatory diagram of a second narrowing-down method of the narrowing portion according to the first embodiment. FIG. 8 is a flowchart showing the operation of the narrowing unit of the image processing apparatus according to the first embodiment. FIG. 9 is a flowchart showing the operation of the annotation unit of the image processing apparatus according to the first embodiment. FIG. 10 is an explanatory diagram of the effect of the first embodiment. FIG. 11A is a diagram showing an example of a second area determined by the annotation unit according to the first embodiment. FIG. 11B is a diagram showing an example of the second region determined by the annotation unit in the first modification. FIG. 12 is a diagram illustrating an example of a plurality of images acquired by the annotation unit according to the second modification. FIG. 13 is a diagram illustrating an example of the second area determined by the annotation unit in the second modification. FIG. 14 is a diagram illustrating an example of a plurality of images acquired by the annotation unit according to the modified example 3. FIG. 15 is a diagram illustrating an example of the second region determined by the annotation unit in the modified example 3. FIG. 16 is a diagram illustrating an example of the second annotation added by the annotation unit in the first example of the modified example 4. FIG. 17 is a diagram illustrating an example of the second annotation added by the annotation unit in the second example of the modified example 4. FIG. 18 is a diagram illustrating an example of a detailed configuration of the determination unit according to the second embodiment. FIG. 19 is a flowchart showing the operation of the determination unit of the image processing apparatus according to the second embodiment. FIG. 20 is a diagram showing an example of a plurality of images acquired by the acquisition unit according to the second embodiment. FIG. 21 is an explanatory diagram of image processing performed by the determination unit according to the second embodiment on the plurality of images shown in FIG. FIG. 22 is a diagram showing an example of the functional configuration of the image processing apparatus according to the third embodiment.

An image processing method according to an aspect of the present invention is provided with a first annotation that indicates two or more first areas, at least one of which is a person area, and the time series of images taken by an on-vehicle camera mounted on a vehicle. A plurality of consecutive images, the plurality of images including at least one image in which the two or more first regions are present in the traveling route of the vehicle and the distance between the first regions is equal to or less than a threshold value; An acquisition step of acquiring, and a determination step of determining the position of each of the two or more first regions while tracing back the time in sequence from the last time image in the plurality of images acquired in the acquisition step. , The first image at the first first time when it is determined that the position of each of the two or more first regions is not in the travel route in the determination step among the plurality of images A determining step of determining an area between the two or more first areas in the image as a second area; and an assigning step of assigning a second annotation indicating the second area determined in the determining step. ..

In this way, the plurality of images captured by the vehicle-mounted camera can be mechanically attached with the second annotation indicating the second region that requires advanced recognition by a crowdsourcing worker. Thereby, it is possible to suppress the variation in quality of the learning data including the plurality of images.

Here, for example, in the determining step, a second image in which the two or more first regions are present in the traveling route of the vehicle and the distance between the first regions is equal to or less than the threshold value is further specified. However, the area between the two or more first areas in the plurality of images that are consecutive in time series and included in the specified first image to the second image may be determined as the second area.

Thereby, the second annotation indicating the second region can be mechanically attached to one or more images.

Further, for example, the image processing method is further associated with information indicating the brake strength or the acceleration of the vehicle, which is all the images that are taken in time series and are captured by the vehicle-mounted camera mounted on the vehicle. A first narrowing step of selecting a first narrowed image which is a plurality of images from a time point at which the brake strength or acceleration of the vehicle is greater than a threshold value to a time point before a certain period, among all the existing images; and the first narrowing step. The second narrowing down step of selecting the plurality of images from the first narrowed down images selected in (3).

Thereby, among the plurality of images captured by the vehicle-mounted camera, a time-series image that may be annotated with the second annotation indicating the second region and includes an image annotated with the first annotation indicating the first region The second annotation indicating the second region can be mechanically added after narrowing down to the time-series image.

Here, for example, the image processing method may further include, before the first narrowing step, providing a crowdsourcing worker with a first annotation indicating, for all the images, the first region existing in the images. May be included in the worker step.

Further, for example, in the image processing method, before the second narrowing step, the crowd narrowing worker is further provided with the first narrowed image with respect to the first narrowed image selected in the first narrowing step. A worker step of adding a first annotation indicating the first region existing therein may be included.

By these, the worker of crowdsourcing can be made to give the 1st annotation which shows the 1st field which exists in an image.

Here, for example, each of the two or more first areas may be a person area indicating a person.

As a result, the second annotation indicating the second area can be mechanically added as a dangerous area in which the vehicle may collide with a person while traveling in the second area.

Further, for example, the two or more first areas may include a person area indicating a person and an automobile area indicating a parked vehicle.

As a result, the second annotation indicating the second area can be mechanically attached as the dangerous area in which the vehicle may collide while traveling in the second area.

Further, for example, the second area is a dangerous area in which the objects indicated by the two or more first areas cross each other in front of the vehicle when approaching each other, and there is a risk of collision with the vehicle. The processing method may further include a risk degree giving step in which the second annotation given in the giving step further includes a risk degree that becomes higher as the area of the second region becomes smaller.

As a result, the degree of risk can be further included in the second annotation indicating the second area, which is a dangerous area in which the vehicle travels.

Further, for example, the second area is a dangerous area in which the objects indicated by the two or more first areas cross each other in front of the vehicle when approaching each other, and there is a risk of collision with the vehicle. The processing method is such that, with respect to the second annotation added in the adding step, the one side area and the other side area forming the second area have different risk levels, and the second area is sandwiched between the second annotation and the second area. It is also possible to include a risk degree giving step of including a risk degree that the one side area or the other side area on the side where the first area having a large movement size among the two first areas has a higher value. ..

As a result, the degree of risk can be further included in the second annotation indicating the second area, which is a dangerous area in which the vehicle travels.

Further, for example, in the determination step, in the plurality of images acquired in the acquisition step, the first image to which the first annotation is not added is determined while sequentially tracing the time from the image at the last time on the time series. And a first determination step for performing the first region in the image at the next time on the time series of the third time of the first image determined in the first determination step from the image at the third time. A second determination step of determining whether or not the first region is present at a position in each of the images displaced in the direction toward the moving direction of the vehicle while tracing back in chronological order by image processing. May be included.

As a result, even if the first annotation indicating the first region that should be attached to some images is not attached, it is possible to determine whether or not the image has the first region by image processing. it can. As a result, the second annotation indicating the second region that requires a high degree of recognition can be further added, so that the variation in the quality of the learning data including the plurality of images can be suppressed.

Further, the image processing device according to one aspect of the present invention, when at least one of which is a person region, is attached with a first annotation indicating two or more first regions, and is imaged by an in-vehicle camera mounted on the vehicle. A plurality of images that are consecutive in a sequence, and that include at least one image in which the two or more first regions are present in the travel route of the vehicle and the distance between the first regions is less than or equal to a threshold value. Determination of determining the position of each of the two or more first regions in the acquisition unit that acquires an image and the plurality of images acquired by the acquisition unit while sequentially tracing back the time from the last time image in the time series Part and the first image at the first first time when the determination unit determines that the position of each of the two or more first regions is not in the travel route among the plurality of images, and the identified A determining unit that determines a region between the two or more first regions in the first image as a second region, and a assigning unit that assigns a second annotation indicating the second region determined by the determining unit, Equipped with.

It should be noted that these general or specific aspects may be realized by a recording medium such as a system, a method, an integrated circuit, a computer program or a computer-readable CD-ROM, and the system, the method, the integrated circuit, the computer. It may be realized by any combination of programs or recording media.

Hereinafter, an image processing method and the like according to an aspect of the present invention will be specifically described with reference to the drawings. Each of the embodiments described below shows one specific example of the present invention. Numerical values, shapes, materials, constituent elements, arrangement positions of constituent elements, and the like shown in the following embodiments are examples and are not intended to limit the present invention. Further, among the constituent elements in the following embodiments, constituent elements that are not described in the independent claim indicating the highest concept are described as arbitrary constituent elements. Further, the contents of each of the embodiments can be combined.

(Embodiment 1)
[Configuration of Image Processing Device 10]
FIG. 1 is a diagram illustrating an example of a functional configuration of the image processing apparatus 10 according to the first embodiment.

The image processing apparatus 10 performs image processing on the annotation data stored in the storage unit 20 by mechanically adding an annotation that requires advanced recognition by a worker, and stores the data as learning data. It is output to the unit 30. In the present embodiment, the annotation-added data is a plurality of images to which an annotation (first annotation) indicating a person (personal region) explicitly present in the image is attached by a worker in crowdsourcing, and the in-vehicle camera It is a plurality of images taken by. It should be noted that adding an annotation when a person is explicitly present in an image does not require the worker to have a high level of recognition, so that it is unlikely that individual differences among workers will occur and there will be no variation in quality.

In the present embodiment, the image processing device 10 includes an annotation unit 11, a narrowing unit 12, and a storage unit 13, as illustrated in FIG. 1. Hereinafter, each component will be described in detail.

[Annotation part 11]
FIG. 2 is a diagram showing an example of a plurality of images acquired by the annotation unit 11 according to the first embodiment. 3 and 4 are explanatory diagrams of an example of image processing performed by the annotation unit 11 according to the first embodiment on the plurality of images shown in FIG.

As shown in FIG. 1, the annotation unit 11 includes an acquisition unit 111, a determination unit 112, a determination unit 113, and a provision unit 114.

(Acquisition unit 111)
The acquisition unit 111 is a plurality of time-sequential images to which first annotations indicating two or more first areas, at least one of which is a person area, are added, and which are taken in time series by an in-vehicle camera mounted on the vehicle. Then, a plurality of images including at least one image in which the two or more first regions are present in the travel route of the vehicle and the distance between the first regions is equal to or less than the threshold value is acquired. Hereinafter, each of the two or more first areas will be described as a person area indicating a person.

In the present embodiment, the acquisition unit 111 acquires, from the storage unit 13, data with a first annotation indicating a first region, such as a plurality of images that are continuous in time series shown in FIG.

Here, a plurality of images that are continuous in time series will be described with reference to FIG.

The plurality of images shown in FIG. 2 are a part of the plurality of images forming the learning data, and are, for example, a plurality of time-series continuous images forming a part of an image captured by an in-vehicle camera mounted on the vehicle. It is an image. More specifically, the plurality of images shown in FIG. 2 includes a frame 101a, a frame 101b, a frame 101c, a frame 101d, and the like. Each of the plurality of images includes (represents) the road 1011 and the person 60 and the person 61. In general, in an image forming a video image captured by a vehicle-mounted camera, the movement of a vehicle such as a car is larger (faster) than the movements of the person 60 and the person 61. The person 61 is moving away (or approaching).

Further, a first region (first annotation) is added to the plurality of images (frame 101a to frame 101d). Here, the first area (first annotation) is a person area indicating the person 60 and the person 61 that explicitly exist. Then, for example, in the frame 101d or the frame 101c (one or more images of the plurality of images), two first regions indicating the person 60 and the person 61 exist in the road 1011 which is the traveling route of the vehicle, and The distance between the two first areas is less than or equal to the threshold value. Here, the threshold may be equal to or smaller than the width of one person, or may be zero distance.

(Determination unit 112)
The determination unit 112 determines the position of each of the two or more first areas in the plurality of images acquired by the acquisition unit 111 while sequentially tracing back the time from the image at the last time on the time series.

In the present embodiment, for example, in the plurality of images shown in FIG. 2, the determination unit 112 assigns the frame 101d, the frame 101c, the frame 101b, and the frame 101a in this order to the first annotation attached to each image (frame). Based on this, the positions of the two first areas are determined. For example, the determination unit 112 determines that there are two first areas in the frame 101d because the two first annotations are added to the frame 101d, and determines the two first areas existing in the frame 101d. The position and size of the frame to be displayed are determined. The determination unit 112 also makes similar determinations in this order for the frames 101c to 101a, but since it is as described above, description thereof will be omitted.

(Determination unit 113)
The determination unit 113 specifies the first image at the first first time when the determination unit 112 determines that the positions of the two or more first regions are not on the travel route among the plurality of images. Then, the determining unit 113 determines the area between the two or more first areas in the specified first image as the second area. Note that the determination unit 113 may further specify the second image in which the two or more first areas are present in the traveling route of the vehicle and the distance between the first areas is equal to or less than the threshold value. In this case, the determining unit 113 may determine the area between the two or more first areas in the plurality of time-sequential images included in the identified first image to second image as the second area. Good.

In the present embodiment, as shown in FIG. 3, determination unit 113 determines the positions of the first region showing person 60 and the first region showing person 61 by the determination unit 112 in the plurality of images shown in FIG. The frame 101b (first image) at the first time t1 when it is determined that is not on the road 1011 is specified. Then, the determining unit 113 determines the area between the first area showing the person 60 and the first area showing the person 61 in the specified frame 101b as the second area. Here, the second area means a dangerous area in which the vehicle may collide with an object such as a person indicated by the first area when the vehicle travels.

Note that the determining unit 113 further includes a first area indicating the person 60 and a first area indicating the person 61 on the road 1011 that is the traveling route of the vehicle, and the distance between the first areas is the threshold value. The frame 101d (or the frame 101c) is specified as the second image which will be described below. In this case, the determination unit 113 indicates the first area indicating the person 60 and the person 61 in the frames 101b to 101d, which are multiple images included in the frame 101b that is the first image to the frame 101d that is the second image. A region between the first region and the first region is determined as the second region.

In this way, the determination unit 113 can mechanically determine the second region for one or more images.

(Granting unit 114)
The attaching unit 114 attaches the second annotation indicating the second area determined by the determining unit 113.

In the present embodiment, the adding unit 114 adds the second annotation indicating the second area determined by the determining unit 113 to the image shown in FIG. 4, for example. Further, the adding unit 114 outputs a plurality of images with the second annotation (images with the second annotation further added to the annotation data) to the storage unit 30 as learning data.

The annotation unit 11 may not output a plurality of images. In this case, the adding unit 114 may output information about a plurality of images to be annotated, such as the coordinate values of the first area and the coordinate values of the second area indicating the persons 60 and 61, for example.

[Structure of narrowing unit 12]
FIG. 5 is a diagram showing an example of a detailed functional configuration of the narrowing unit 12 shown in FIG. FIG. 6 is an explanatory diagram of a first narrowing-down method of the narrowing portion 12 in the first embodiment. FIG. 7 is an explanatory diagram of the second narrowing-down method of the narrowing portion 12 in the first embodiment.

As shown in FIG. 5, the narrowing unit 12 includes a first narrowing unit 121 and a second narrowing unit 122.

The narrowing-down unit 12 narrows down the annotation-attached data acquired from the storage unit 20 into a predetermined time-series image, and stores it in the storage unit 13. Here, the predetermined time-series image is a dangerous area in which there is a possibility of collision with the vehicle when people approach each other while the vehicle is traveling, and if the worker is a worker, the altitude is high. It is a time-series image that may have a dangerous area that requires specific recognition.

In the present embodiment, the storage unit 20 is configured by a HDD (Hard Disk Drive), a memory, or the like, and stores data annotated by a crowdsourcing worker (annotation data).

More specifically, the annotation data is all the images that are taken in time series by the vehicle-mounted camera mounted on the vehicle and are linked to the information indicating the brake strength or acceleration of the vehicle. It is an image. In addition, the annotation-added data is given a first annotation indicating a first area, which is a person area existing in the image, in all the images by a crowdsourcing worker.

The first narrowing unit 121 links a plurality of images, which are the annotation data stored in the storage unit 20, with the brake information or the like, for example, into a plurality of images (first narrowed image) in the first period shown in FIG. Narrow down to. More specifically, the first narrowing unit 121 is all images that are taken in time series and are captured by an in-vehicle camera mounted on the vehicle, and are associated with information indicating the brake strength or acceleration of the vehicle. From all the images, the first narrowed-down image, which is a plurality of images from the time when the brake strength or the acceleration of the vehicle is larger than the threshold value to the time before a certain period, is selected.

Then, the second narrowing down unit 122 narrows down the first narrowed down images selected by the first narrowing down unit 121 to the above-described plurality of images. In the present embodiment, the second narrowing unit 122 further narrows down the plurality of images (first narrowed images) narrowed down by the first narrowing unit 121 by image processing or the like. More specifically, the second narrowing unit 122 is provided with a first annotation indicating two first regions indicating the persons 60 and 61, and is mounted on the vehicle, for example, as shown in a frame 101d in FIG. A plurality of images taken in time series by the vehicle-mounted camera, the two first regions being present in the road 1011 and the distance between the first regions being less than or equal to a threshold value. Narrow down to multiple images that include at least.

Then, the second narrowing unit 122 stores the plurality of narrowed images in the storage unit 13.

[Storage 13]
The storage unit 13 is composed of an HDD (Hard Disk Drive), a memory, and the like. The storage unit 13 stores a plurality of images narrowed down by the narrowing unit 12.

[Operation of Image Processing Device 10]
Next, the operation of the image processing apparatus 10 configured as above will be described with reference to FIGS. 8 and 9.

FIG. 8 is a flowchart showing the operation of the narrowing unit 12 of the image processing apparatus 10 according to the first embodiment.

In FIG. 8, first, the narrowing-down unit 12 of the image processing apparatus 10 acquires the annotation data from the storage unit 20.

Next, the narrowing-down unit 12 performs a first narrowing-down process of narrowing down the acquired annotation-added data with brake information or the like (S90). Specifically, as described above, the narrowing-down unit 12 is all images taken in time series by the vehicle-mounted camera mounted on the vehicle and associated with information indicating the brake strength or acceleration of the vehicle. The first narrowed-down image, which is a plurality of images from a time point when the brake strength or acceleration of the vehicle is larger than the threshold value to a time point before a certain period, is selected from all the displayed images.

Next, the narrowing unit 12 performs a second narrowing process that further narrows down the first narrowed image narrowed down in S90 by image processing or the like (S91). Specifically, as described above, the narrowing-down unit 12 is provided with the first annotation indicating the two first regions indicating the person in the first narrowed-down image narrowed down by the first narrowing-down process, and the vehicle A plurality of time-sequential images captured by the vehicle-mounted camera mounted on the vehicle, the two first areas are present in the travel route, and the distance between the first areas is less than or equal to a threshold value. A plurality of images including at least the above images are selected. Then, the plurality of images selected (thinned) by the second narrowing-down process are stored in the storage unit 13.

FIG. 9 is a flowchart showing the operation of the annotation unit 11 of the image processing apparatus 10 according to the first embodiment.

In FIG. 9, first, the annotation unit 11 of the image processing apparatus 10 performs an acquisition process of acquiring a plurality of images narrowed down by the second narrowing down process from the storage unit 13 (S101).

Next, the annotation unit 11 performs a determination process of determining the position of each of the two or more first regions while sequentially tracing back the time from the image at the last time on the time series in each of the plurality of images acquired in S101. (S102).

Next, in S102, the annotation unit 11 identifies the first image at the first first time when it is determined that the positions of the two or more first regions in the plurality of images are not on the travel route, and the identified first image is determined. A determination process is performed to determine the area between the two or more first areas in one image as the second area (S103).

Next, the annotation unit 11 performs the adding process of adding the second annotation indicating the second area determined in S103 (S104). Then, the annotation unit 11 outputs the plurality of images with the second annotation to the storage unit 30 as learning data.

In this way, the image processing apparatus 10 performs image processing on the annotation data stored in the storage unit 20 by mechanically adding an annotation that requires advanced recognition by a worker, and performs learning. It can be output to the storage unit 30 as use data.

In the above description, the image processing apparatus 10 has been described as performing the first narrowing-down process (S90) and the second narrowing-down process (S91) using the annotation-added data stored in the storage unit 20. Not exclusively. That is, before performing the first narrowing-down process (S90), the image processing device 10 causes the worker of the crowdsourcing to provide the first annotation indicating the first region, which is the human region existing in the images, to all the images. Although it has been described that the annotation-attached data is generated by being attached, the present invention is not limited to this.

In other words, the image processing apparatus 10 acquires all the images in time series that are captured by the vehicle-mounted camera to which the annotation is not added, and performs the first narrowing process (S90) on all the acquired images. Good. In this case, if the crowd sourcing worker is given an annotation indicating a region that is a moving object existing in the images to the plurality of images (first narrowed-down images) that have been subjected to the first narrowing-down process. Good. That is, before the second narrowing process (S91), the image processing apparatus 10 asks the worker of the crowdsourcing to perform the first narrowed image with respect to the first narrowed image selected in the first narrowing process (S90). The annotation indicating the first area, which is the person area existing in the above, may be added.

[Effects, etc.]
As described above, according to the present embodiment, it is possible to realize an image processing method and the like that can suppress variations in quality of learning data including the plurality of images.

Here, the effects of the image processing method according to the present embodiment will be described with reference to FIG. FIG. 10 is an explanatory diagram of the effect of the first embodiment. The plurality of images shown in (a) of FIG. 10 is an example of a plurality of images that are continuous in time series in which the front (traveling direction) is captured by the vehicle-mounted camera mounted on the vehicle. FIG. 10B shows the brake strength or the acceleration associated with each of the plurality of images shown in FIG. Note that elements similar to those in FIG. 2 and the like are designated by the same reference numerals, and detailed description thereof will be omitted.

From (a) and (b) of FIG. 10, the driver of the vehicle equipped with the in-vehicle camera begins to see that the person 60 and the person 61 approach each other at the frame 101c at time tp and cross the front of the vehicle. It can be seen that the vehicle is suddenly braked so as not to hit the person 60 and the person 61 and the acceleration changes.

Therefore, in order to notify the driver of the vehicle of the area between the person 60 and the person 61, which is a dangerous area for the vehicle to travel when the person 60 and the person 61 approach each other, the vehicle crosses in front of the vehicle. It is necessary to perform the learning process using the image in which the dangerous area is attached as the annotation (correct answer information).

However, when people approach each other, they will cross the front of the vehicle and it will be dangerous for the vehicle to travel.Therefore, the area between these people is a dangerous area where vehicles cannot pass, and the crowdsourcing worker It is easy for workers to recognize the differences among workers. Therefore, the work of annotating such a dangerous area is likely to cause individual differences among workers. For example, looking at the frame 101b at time t1 shown in (a) of FIG. 10, the area between the person 60 and the person 61 crosses the front of the vehicle when the person 60 and the person 61 approach each other, and the vehicle runs. This is because recognizing the above as a dangerous area requires advanced recognition such as experience and comparison with an image at the next time.

On the other hand, in all the images (frames 101a to 101d) shown in FIG. 10A, the annotation work with the first region showing the person 60 and the person 61 is unlikely to cause individual differences in crowdsourcing workers. This is because the worker can add annotations (first area) indicating the person 60 and the person 61 as they can be seen in the image, and therefore does not require high-level recognition.

As described above, in the image processing method according to the present embodiment, the crowdsourcing worker may be made to perform the annotation indicating the object such as a person who is visible in the image in the time-series image captured by the vehicle-mounted camera.

Then, when the people approach each other, they cross the front of the vehicle, and the area (second area) between the people, which is a dangerous area when the vehicle travels, is set to the image processing apparatus 10 or a computer that executes the image processing method. It may be performed by a machine such as. Specifically, first, a plurality of consecutive images in a time series, to which a first annotation indicating two first areas indicating a person is added, and which are captured by an in-vehicle camera mounted on the vehicle, Two first areas are present in the travel route of the vehicle, and the plurality of images including at least one image in which the distance between the first areas is equal to or less than the threshold value is narrowed down. Then, in the plurality of images, the first image at the first time when it is determined that the position of each of the two first regions is not in the travel route is specified while tracing back the time on the time series, and the specified first image is specified. The area between the two or more first areas in one image may be determined as the second area, and the second annotation indicating the dangerous area (second area) may be added.

As described above, according to the image processing method and the like of the present embodiment, the crowd sourcing worker can be made to add the first annotation indicating the first region which is the human region existing in the image. In addition, the image processing method and the like of the present embodiment uses a second annotation that indicates a second region that requires advanced recognition for a plurality of images captured by an in-vehicle camera if a crowdsourcing worker uses a machine. Can be attached to the target. Thereby, it is possible to suppress the variation in quality of the learning data including the plurality of images.

Note that the image processing method and the like of the present embodiment specifies the first image at the first time when it is determined that the positions of the two first regions are not on the travel route, Although the area between the two or more first areas is determined as the second area, it is not limited thereto. You may specify the 2nd image in which the two 1st area|regions which each show a person exist in the driving|running route of a vehicle, and the distance between 2 1st area|regions is below a threshold value. In this case, a region including the second image and between the two or more first regions from the second image to the image (for example, the first image) a predetermined time before may be determined as the second region.

(Modification 1)
FIG. 11A is a diagram showing an example of a second area determined by the annotation unit 11 according to the first embodiment. FIG. 11B is a diagram showing an example of the second region determined by the annotation unit 11 in the first modification.

In the first embodiment, as shown in FIG. 11A, the second area has been described as a two-dimensional area between two first areas, but the present invention is not limited to this. When distance information exists for each of the persons indicated by the two first areas included in the plurality of images acquired by the annotation unit 11, the annotation unit 11 determines that the two persons (person areas) have the distance information as illustrated in FIG. 11B. The space connecting the two may be determined as the second region.

(Modification 2)
In the first embodiment, as the object indicated by the two or more first areas, the two person areas indicated by the two first areas are described as an example, but the present invention is not limited to this. The two or more first areas may indicate three or more person areas. In this modification, a case where the four first areas indicate four person areas will be described.

FIG. 12 is a diagram showing an example of a plurality of images acquired by the annotation unit 11 in the second modification. FIG. 13 is a diagram showing an example of the second area determined by the annotation unit 11 in the second modification.

The annotation unit 11 in the second modification acquires a plurality of images including the frame 103i and the frame 103n as shown in FIG. Each of the plurality of images shown in FIG. 12 includes a road 1031 and a person 62, a person 63, a person 64, and a person 65. Furthermore, four first areas (first annotations) indicating the person 62, the person 63, the person 64, and the person 65 are added to the plurality of images shown in FIG.

Here, the annotation unit 11 in the modified example 2 makes the first at the first time when it is determined that the positions of the four first areas indicating the persons 62 to 65 are not on the road 1031 in the plurality of images illustrated in FIG. 12. A frame 101a (not shown) is specified as an image. In addition, in the annotation unit 11 in the second modification, the positions of the four first areas indicating the persons 62 to 65 are present in the road 1031 that is the traveling route of the vehicle, and the distance between these first areas is equal to or less than the threshold value. The frame 103n at time t2 is specified as the second image.

Then, as shown in FIG. 13, the annotation unit 11 in the modified example 2 includes, for example, the person 62 to 62 in the frame 103i which is a plurality of images included in the frame 103a which is the first image to the frame 103n which is the second image. The area between the four first areas indicating 65 may be determined as the second area.

In this way, the image processing method and the like of the present modified example similarly, even when the three or more first areas indicate three or more person areas, the first is a dangerous area when the vehicle travels. Two regions can be mechanically determined, and a second annotation indicating the second region can be mechanically added.

(Modification 3)
In the first embodiment and the first and second modifications, the first area is described as a person, but the present invention is not limited to this. The object indicated by the first area may be an automobile parked or stopped. In the present modification, one of the two first areas is a person area, and the other is an automobile area indicating a parked vehicle, which will be described with reference to FIGS. 14 and 15.

FIG. 14 is a diagram showing an example of a plurality of images acquired by the annotation unit 11 in the modified example 3. FIG. 15 is a diagram showing an example of the second region determined by the annotation unit 11 in Modification 3.

The annotation unit 11 in Modification 3 acquires a plurality of images including the frames 104a,..., The frame 104i,. Each of the plurality of images shown in FIG. 14 includes the road 1041, the automobile 66, and the person 67. Furthermore, two first regions (first annotation) showing the automobile 66 and the person 67 are added to the plurality of images shown in FIG.

The annotation unit 11 in the modified example 3 uses the frame 104a at time t1 as the first image at the first time when it is determined that the position of the first region indicating the person 67 is not on the road 1041 in the plurality of images shown in FIG. Specify. In addition, in the annotation unit 11 in the modified example 4, the positions of the two first regions indicating the automobile 66 and the person 67 are present in the road 1041 that is the traveling route of the vehicle, and the distance between the two first regions is the same. The frame 104n at the time t2 is specified as the second image in which is less than or equal to the threshold.

Then, the annotation unit 11 in the modified example 3, as shown in the frame 104i in FIG. 15, includes, for example, a plurality of images (frames 104a to 104n) included in the frame 104a that is the first image to the frame 104n that is the second image. The area between the two first areas indicating the automobile 66 and the person 67 in 1 may be determined as the second area.

Thus, even if one of the objects indicated by the two first areas is a parked vehicle, the image processing method according to the present modified example is similar to the above in that a person is traveling in front of the vehicle. It is possible to mechanically determine a second area that is a dangerous area that will be crossed and will collide when a vehicle passes between them, and to mechanically attach a second annotation indicating the second area. You can

(Modification 4)
Although the annotation unit 11 determines the second region and attaches the second annotation indicating the determined second region in the first embodiment and the first to third variations described above, the present invention is not limited to this. The annotation unit 11 may further determine the degree of risk of the second area in addition to determining the second area that is a dangerous area for the running vehicle. In this case, the annotation unit 11 may add the second annotation indicating the degree of risk in addition to indicating the second area. Hereinafter, a method of determining the degree of danger of the second area and the like will be specifically described.

<First example: risk determination method>
FIG. 16 is a diagram illustrating an example of the second annotation added by the annotation unit 11 in the first example of the modified example 4.

The annotation unit 11 in the first example of the modified example 4 acquires a plurality of images shown in FIG. 14, and as shown in FIG. 15, when the objects indicated by the first region approach each other, the annotation unit 11 crosses the front of the vehicle. It is assumed that the second area, which is a dangerous area in which the vehicle travels, is determined because the vehicle may collide with the vehicle. Note that the operation of the annotation unit 11 to determine the second area has been described in Modification 3, and thus the description here will be described.

In this modification, the annotation unit 11 further determines the risk level according to the determined area of the second region. More specifically, the annotation unit 11 determines the degree of risk that becomes higher as the size of the second area is smaller. This is because the smaller the area of the second region, the higher the possibility of collision with the vehicle 66 and the person 67 when the vehicle travels between the vehicle 66 and the person 67 shown in the first region, and therefore the vehicle cannot pass through. If the area of the second region is equal to or smaller than the predetermined area, the risk level may be determined to be 1.0 (100% risk level).

Then, the annotation unit 11 adds a second annotation indicating the determined second area and the degree of risk of the second area. More specifically, in the first example of the modified example 4, the annotation unit 11 further includes, in the second annotation indicating the second region, a risk degree indicating a higher value as the area of the second region is smaller. For example, in the example illustrated in FIG. 16, the annotation unit 11 adds the second annotation indicating the degree of danger of 0.7 to the second area of the frame 104a at time t1, and sets the second area of the frame 104i at time ti to the second area. , A second annotation indicating a risk level of 1.0 is added. When the vehicle passes through the second area of the frame 104i, the vehicle 67 will definitely collide with the person 67, and thus the second annotation indicating the risk of 1.0 is added.

<Second example: risk determination method>
FIG. 17 is a diagram illustrating an example of the second annotation added by the annotation unit 11 in the second example of the modified example 4.

The annotation unit 11 in the second example of the modified example 4 also acquires the plurality of images shown in FIG. 14, and as shown in FIG. 15, when the objects indicated by the first region approach each other, the object crosses the front of the vehicle, Since there is a possibility of collision with the vehicle, it is assumed that the second area, which is a dangerous area when the vehicle travels, is determined. Note that the operation of the annotation unit 11 to determine the second region has been described in Modification 3, and thus the description here will also be described.

In the present modification, the annotation unit 11 further determines the degree of risk of the determined second area by weighting it within the second area. More specifically, the annotation unit 11 divides the second area into two areas, and the area on the side of one first area indicating a person or the like who is moving more is higher than the area on the side of the other first area. The risk is weighted so that the value becomes a value. This is because it can be said that the vehicle has a high possibility of colliding with a person who moves significantly when traveling along the traveling route.

Then, the annotation unit 11 adds a second annotation indicating the determined second area and the degree of risk of the second area. More specifically, in the second example of the modified example 4, the annotation unit 11 uses the second annotation indicating the second region with different risk levels for the one side region and the other side region forming the second region. Therefore, of the two first areas sandwiching the second area, the degree of risk that the one side area or the other side area where the first area having a large movement is located has a higher value is included. For example, in the example illustrated in FIG. 17, the annotation unit 11 adds the second annotation indicating the risk level of 1.0 to the area near the person 67 in the second area of the frame 104a at time t1, and the second area A second annotation indicating a risk level of 0.7 is added to the area near the automobile 66 in the area.

Here, the annotation unit 11 adds the second annotation indicating the risk level of 1.0 to the entire second area of the frame 104i at the time ti. This is because the area of the area in the vicinity of the person 67 has become equal to or smaller than the area of the second area. When the area of the second region is equal to or smaller than the predetermined area, the weighted risk may not be given, and a uniform risk may be given.

As described above, according to the image processing method and the like of the present modification, the second annotation indicating the second area, which is a dangerous area when the vehicle travels, is added to the second annotation when the vehicle travels further. Area risk can be included.

(Embodiment 2)
In the first embodiment, it has been described that the crowdsourcing worker is made to perform the annotation indicating the object such as a person who is visible in the image in the time-series image captured by the vehicle-mounted camera. However, since the work quality of the worker is not constant, in some images among the time-series images taken by the vehicle-mounted camera, even if an object such as a person is visible in the image, the first It is also possible that an annotation indicating the area is not added.

Hereinafter, this case will be described as a second embodiment focusing on the points different from the first embodiment.

[Configuration of Image Processing Device 10A]
The image processing apparatus 10A according to the second embodiment is different from the image processing apparatus 10 according to the first embodiment in the configuration of the determination unit 112A of the annotation unit 11A. The rest of the configuration is the same as that of the image processing apparatus 10 according to the first embodiment, and therefore its description is omitted.

[Determination unit 112A]
FIG. 18 is a diagram illustrating an example of a detailed configuration of the determination unit 112 according to the second embodiment.

The determination unit 112A determines the first image, to which the first annotation is not added, in the plurality of images acquired by the acquisition unit 111 by sequentially tracing back the time from the image at the last time in the time series.

In the present embodiment, the determination unit 112A, in the plurality of images acquired by the acquisition unit 111, sequentially traces the time from the image at the last time on the time series, and determines the first image to which the first annotation is not added. judge. The determination unit 112A traces the first area in the image at the next time on the time series of the third time of the determined first image, tracing the time from the image at the third time in chronological order while moving in the moving direction of the vehicle. It is determined by image processing whether or not the first region exists at the position in each of the images shifted in the vertical direction.

[Operation of Image Processing Device 10A]
Next, the operation of the image processing apparatus 10A configured as described above will be described with reference to FIGS.

FIG. 19 is a flowchart showing the operation of the determination unit 112A of the image processing apparatus 10A according to the second embodiment. FIG. 20 is a diagram showing an example of a plurality of images acquired by the acquisition unit 111 according to the second embodiment. FIG. 21 is an explanatory diagram of image processing performed by the determination unit 112A in the second embodiment on the plurality of images shown in FIG. The same elements as those in FIGS. 2 to 4 are designated by the same reference numerals, and detailed description thereof will be omitted.

First, the acquisition unit 111 of the image processing apparatus 10A acquires a plurality of images, which are annotation data, from the storage unit 20. In the present embodiment, in a part of the plurality of images acquired by the acquisition unit 111, the first area (first area) indicating that the person 60 or the person 61 is present even if the person 60 or the person 61 is visible in the image (first 1 Annotation) is not added. In the example shown in FIG. 20, the first region is not added even if the person 60 or the person 61 is visible in the image (frame) in some images (frame 101a, frame 101b).

Next, the determination unit 112A determines the first image to which the first annotation is not added while sequentially tracing back the time from the image at the last time on the time series in the plurality of images acquired by the acquisition unit 111. 1 determination processing is performed (S1021). For example, in the plurality of images (frames 101a to 101d) illustrated in FIG. 20, the determination unit 112A traces the time sequentially from the frame 101d, which is the last time image in the time series, and the first annotation, that is, the first region. The frame 101b which is the first image to which is not added is determined.

Next, the determination unit 112A traces the first area in the image at the next time on the time series of the third time of the determined first image, tracing back the time from the image at the third time in chronological order, and A second determination process is performed by image processing to determine whether or not the first region is present at a position in each of the images shifted in the direction perpendicular to the moving direction (S1022). For example, as illustrated in FIG. 21, the determination unit 112A sets the first area in the frame 101c at the next time t4 on the time series of the time t3 (third time) of the frame 101b from the frame 101b at the time t3 to the time. While going back in chronological order, it is determined by image processing whether or not the first region exists at the position in each of the images (frames 101b to 101a) shifted in the direction perpendicular to the moving direction of the vehicle. In the example illustrated in FIG. 21, the determination unit 112A determines that the first area exists in the frames 101a to 101b by image processing.

In this way, the determination unit 112A further determines the presence/absence of the first region indicating a person or the like by image processing with respect to the image without the first annotation among the plurality of images acquired by the acquisition unit 111.

[Effects, etc.]
As described above, according to the present embodiment, when a part of the plurality of images captured by the vehicle-mounted camera is not annotated with the first annotation indicating the first region that does not require advanced recognition However, by tracking the first area while rewinding a plurality of images (videos), the presence or absence of the first area can be mechanically determined by image recognition. That is, even if the first annotation indicating the first region that should be attached to some images is not attached, it is possible to determine whether or not the first region is present in some images by image processing. .. As a result, it is possible to mechanically attach a second annotation indicating a second region that requires a high degree of recognition to a plurality of images captured by the vehicle-mounted camera, and thus learning data including the plurality of images. It is possible to realize an image processing method and the like that can suppress the variation in the quality of the image.

(Embodiment 3)
In the first embodiment, it has been described that the crowdsourcing worker is made to perform the annotation indicating the object such as a person who can be seen in the image in the time-series image captured by the vehicle-mounted camera, but the invention is not limited to this. The image processing apparatus may attach the first area indicating a person or the like and the first annotation indicating the first area to the time-series image, instead of the worker.

Hereinafter, in this case, the third embodiment will be described focusing on the differences from the first embodiment.

[Configuration of Image Processing Device 10B]
FIG. 22 is a diagram showing an example of the functional configuration of the image processing apparatus 10B according to the third embodiment. The same elements as those in FIG. 1 and the like are designated by the same reference numerals, and detailed description thereof will be omitted.

The image processing apparatus 10B shown in FIG. 22 is different in configuration from the image processing apparatus 10 according to the first embodiment in that an annotation adding unit 14B and a storage unit 20B are added. The rest of the configuration is the same as that of the image processing apparatus 10 according to the first embodiment, and therefore its description is omitted.

The storage unit 40 includes an HDD (Hard Disk Drive), a memory, and the like. The storage unit 40 stores video data (time-series images) taken by the vehicle-mounted camera.

The annotation adding unit 14B acquires the video data (time-series images) captured by the vehicle-mounted camera stored in the storage unit 40. The annotation giving unit 14B attaches an annotation indicating the first region indicating an object such as a person who is seen in the image and an annotation indicating the first region to the acquired video data (time-series image) by performing image processing. .. The annotation giving unit 14B outputs the video data (time-series image) with the first annotation to the storage unit 20B as annotation giving data.

The storage unit 20B includes an HDD (Hard Disk Drive), a memory, and the like. The storage unit 40 stores the data (annotation data) to which the first annotation is added by the annotation addition unit 14B.

[Effects, etc.]
As described above, according to the present embodiment, in the video data (time-series image) captured by the vehicle-mounted camera, the first area indicating a person or the like that does not require advanced recognition and the annotation indicating the first area. Can be attached mechanically (by the image processing apparatus 10B) instead of a crowdsourcing worker. Then, the plurality of images captured by the vehicle-mounted camera can be further mechanically attached with the second annotation indicating the second region that requires a high degree of recognition.

In this way, according to the image processing method and the like of the present embodiment, it is possible to realize an image processing method and the like that can suppress variation in quality of learning data including the plurality of images.

Although the image processing method and the like according to one or more aspects of the present invention have been described above based on the embodiment, the present invention is not limited to this embodiment. As long as it does not depart from the gist of the present invention, various modifications made by those skilled in the art may be applied to the present embodiment, or a configuration constructed by combining components in different embodiments may be one or more of the present invention. It may be included in the range of the aspect. For example, the following cases are also included in the present invention.

(1) Each of the above devices is specifically a computer system including a microprocessor, a ROM, a RAM, a hard disk unit, a display unit, a keyboard, a mouse, and the like. A computer program is stored in the RAM or the hard disk unit. Each device achieves its function by the microprocessor operating according to the computer program. Here, the computer program is configured by combining a plurality of instruction codes indicating instructions to the computer in order to achieve a predetermined function.

(2) Some or all of the constituent elements of each of the above devices may be configured by one system LSI (Large Scale Integration). The system LSI is a super-multifunctional LSI manufactured by integrating a plurality of constituent parts on one chip, and specifically, is a computer system including a microprocessor, ROM, RAM and the like. .. A computer program is stored in the RAM. The system LSI achieves its function by the microprocessor operating according to the computer program.

(3) Some or all of the constituent elements of each of the above devices may be configured with an IC card that can be attached to and detached from each device or a single module. The IC card or the module is a computer system including a microprocessor, ROM, RAM and the like. The IC card or the module may include the above super-multifunctional LSI. The IC card or the module achieves its function by the microprocessor operating according to the computer program. This IC card or this module may be tamper resistant.

(4) The present disclosure may be the methods described above. Further, it may be a computer program that realizes these methods by a computer, or may be a digital signal including the computer program.

(5) Further, according to the present disclosure, a computer-readable recording medium for reading the computer program or the digital signal, for example, a flexible disk, a hard disk, a CD-ROM, a MO, a DVD, a DVD-ROM, a DVD-RAM, a BD ( It may be recorded on a Blu-ray (registered trademark) Disc), a semiconductor memory, or the like. Further, the digital signal recorded on these recording media may be used.

(6) Further, the present disclosure may transmit the computer program or the digital signal via an electric communication line, a wireless or wired communication line, a network represented by the Internet, a data broadcast, or the like.

(7) Further, the present disclosure may be a computer system including a microprocessor and a memory, wherein the memory stores the computer program, and the microprocessor operates according to the computer program. ..

(8) Another computer independent by recording the program or the digital signal in the recording medium and transferring the program or by transferring the program or the digital signal via the network or the like. It may be implemented by a system.

The present invention can be used for an image processing method, an image processing device, and a program thereof. In particular, the quality of the learning data used when performing machine learning on the dangerous area where a vehicle may collide with each other will cross the front of the vehicle when the vehicles approach each other, and there is no variation in quality. The image processing method, the image processing apparatus, and the program therefor can be used.

10, 10A, 10B Image processing device 11, 11A Annotation part 12 Narrowing down part 13, 20, 20B, 30, 40 Storage part 14B Annotation giving part 60, 61, 62, 63, 64, 65, 67 Person 66 Car 101a, 101b , 101c, 101d, 102, 102A, 103i, 103n, 104a, 104i, 104n Frame 111 Acquisition unit 112, 112A Judgment unit 113 Determining unit 114 Assigning unit 121 First narrowing unit 122 Second narrowing unit 1011, 1021, 1031, 1041 road

Claims (12)

A plurality of consecutive images taken in-series by a vehicle-mounted camera provided with a first annotation indicating at least one first area, at least one of which is a person area. An acquisition step of acquiring a plurality of images including at least one image in which the first region exists in the travel route of the vehicle and the distance between the first regions is equal to or less than a threshold value;
In the plurality of images acquired in the acquisition step, a determination step of determining the position of each of the two or more first regions while sequentially going back from the image of the last time on the time series,
Of the plurality of images, the first image at the first first time when the position of each of the two or more first regions is determined not to be on the travel route in the determination step is identified and specified. A determination step of determining a region between the two or more first regions in the above as a second region,
An assigning step of assigning a second annotation indicating the second area determined in the determining step,
Image processing method. In the determining step,
A second image in which the two or more first regions are present in the travel route of the vehicle and the distance between the first regions is equal to or less than the threshold value is specified.
An area between the two or more first areas in a plurality of time-series consecutive images included in the specified first image to the second image is determined as the second area,
The image processing method according to claim 1. The image processing method further comprises
Of all the images taken in chronological order taken by the vehicle-mounted camera mounted on the vehicle and associated with the information indicating the brake strength or acceleration of the vehicle, the brake strength of the vehicle Or a first narrowing step of selecting a first narrowed image which is a plurality of images from a time point when the acceleration is larger than a threshold value to a time point before a certain period,
A second narrowing step of selecting the plurality of images from the first narrowed images selected in the first narrowing step.
The image processing method according to claim 1. The image processing method further comprises
Before the first narrowing step, a worker step of causing a worker of crowdsourcing to attach a first annotation indicating the first region existing in the image to all the images,
The image processing method according to claim 3. The image processing method further comprises
Before the second narrowing step, a crowdsourcing worker first indicates the first region existing in the first narrowed image with respect to the first narrowed image selected in the first narrowing step. Including a worker step to add annotations,
The image processing method according to claim 4. Each of the two or more first areas is a person area indicating a person,
The image processing method according to claim 1. The two or more first areas include a person area indicating a person and an automobile area indicating a parked vehicle.
The image processing method according to claim 1. The second area is a dangerous area that may cross the front of the vehicle when objects indicated by the two or more first areas approach each other, and may collide with the vehicle.
The image processing method further comprises
The second annotation added in the assigning step further includes a risk assigning step that includes a risk that becomes higher as the area of the second region becomes smaller.
The image processing method according to claim 1. The second area is a dangerous area that may cross the front of the vehicle when objects indicated by the two or more first areas approach each other, and may collide with the vehicle.
The image processing method further comprises
In addition to the second annotation added in the adding step, there is a different degree of risk in the one side area and the other side area forming the second area, and the two first areas sandwiching the second area. A step of adding a risk degree including a risk degree that the one side area or the other side area on the side where the first area where the movement amount is large is higher.
The image processing method according to claim 1. The determination step is
In a plurality of images acquired in the acquisition step, a first determination step of determining the first image to which the first annotation is not added while sequentially tracing the time from the image at the last time on the time series,
While tracing back the first region in the image at the next time on the time series of the third time of the first image determined in the first determination step from the image of the third time in time sequence order, A second determination step of determining, by image processing, whether or not the first area is present at a position in each of the images that are displaced in a direction toward the direction perpendicular to the moving direction of the vehicle,
The image processing method according to claim 1. A plurality of consecutive images taken in-series by a vehicle-mounted camera provided with a first annotation indicating at least one first area, at least one of which is a person area. An acquisition unit that acquires a plurality of images including at least one image in which the first region exists in the travel route of the vehicle and the distance between the first regions is equal to or less than a threshold value;
In the plurality of images acquired by the acquisition unit, a determination unit that determines the position of each of the two or more first regions while sequentially going back from the image of the last time on the time series,
Of the plurality of images, the determination unit determines the first image at the first first time when the positions of the two or more first regions are determined not to be in the travel route, and identifies the first image. A determining unit that determines a region between the two or more first regions in the above as a second region,
An assigning unit that assigns a second annotation indicating the second region determined by the determining unit,
Image processing device. A plurality of images, which are consecutively time-sequentially taken by an in-vehicle camera mounted on a vehicle, to which at least one first region is attached, the first annotation indicating at least one person region, An acquisition step of acquiring a plurality of images including at least one image in which the first region exists in the travel route of the vehicle and the distance between the first regions is equal to or less than a threshold value;
In the plurality of images acquired in the acquisition step, a determination step of determining the position of each of the two or more first regions while sequentially going back from the image of the last time on the time series,
Among the plurality of images, the first image at the first first time when it is determined that the positions of the two or more first regions are not in the traveling route in the determination step is identified and specified. A determination step of determining a region between the two or more first regions in the above as a second region,
An attaching step of attaching a second annotation indicating the second area determined in the determining step,
A program that causes a computer to execute.

Images