JP2022044156A - Image recognition device, image recognition method, computer program and image recognition system - Google Patents

Abstract

To provide an image recognition device, an image recognition method, a computer program and an image recognition system, capable of accurately recognizing an object in an image without delay.SOLUTION: In an operation support system, an image recognition device 35 provided in a control system of a mobile body includes: an attention area acquisition part 54 configured to acquire, from a storage part storing as attention area information 41 associating attention area in an image and a photographed location of the image, an attention area in an image of a periphery of the mobile body photographed by a camera installed in the mobile body, according to a location of the mobile body; a segmentation part 57 configured to cut out a picture of the attention area from the image of the periphery of the mobile body; and a recognition processing part 59 configured to execute prescribed recognition processing according to the picture of the attention area.SELECTED DRAWING: Figure 3

Description

The present disclosure relates to an image recognition device, an image recognition method, a computer program, and an image recognition system.

Conventionally, a system has been proposed in which a road sign or the like is recognized from an image of the surroundings of a moving body taken by a camera mounted on a moving body such as an automobile, and the traveling of the moving body is controlled based on the recognition result (. For example, see Patent Document 1).

Japanese Unexamined Patent Publication No. 2019-22017

In order to control the traveling of a moving object, it is desired to accurately recognize an object in an image and to process the image without delay.

In order to process the image without delay, it is necessary to extract an image of the region of interest including a predetermined object from the image and intensively execute the recognition process of the object on the extracted image of the region of interest.

However, the image taken by the camera may be blurred due to the influence of nighttime, rainy weather, fog, evening, backlight, shadow of a temporarily installed object, power failure, and the like. In such a case, the image of the area of interest cannot be extracted because the object such as a road sign or a traffic signal appears unclear, or the image of the wrong position can be extracted. It may fail. As a result, it may not be possible to accurately recognize the object. Further, when the image of the region of interest cannot be extracted from the image, the recognition process is executed with the entire region of the image as the processing target, which makes it difficult to process the image without delay.
It should be noted that such a problem occurs not only in automobiles but also in traveling control of other moving objects such as mobile robots.

The present disclosure has been made in view of such circumstances, and provides an image recognition device, an image recognition method, a computer program, and an image recognition system capable of accurately recognizing an object in an image without delay. The purpose is.

The image recognition device according to one aspect of the present disclosure is a camera mounted on the moving body based on the position of the moving body from a storage unit that stores the region of interest in the image and the shooting position of the image in association with each other. Based on the acquisition unit that acquires the region of interest in the image around the moving body taken by the camera, the cutout portion that cuts out the image of the region of interest from the image around the moving object, and the image of the region of interest. It also includes a recognition processing unit that performs predetermined recognition processing.

The image recognition method according to another aspect of the present disclosure is mounted on the moving body based on the position of the moving body from the storage unit that stores the region of interest in the image and the shooting position of the image in association with each other. Based on the step of acquiring the region of interest in the image around the moving object taken by the camera, the step of cutting out the image of the region of interest from the image around the moving object, and the image of the region of interest. It includes a step of performing a predetermined recognition process.

A computer program according to another aspect of the present disclosure is to move a computer from a storage unit that stores a region of interest in an image and a shooting position of the image in association with each other to the moving body based on the position of the moving body. An acquisition unit that acquires an area of interest in an image around the moving object taken by a mounted camera, a cutting unit that cuts out an image of the area of interest from an image around the moving object, and the area of interest. It functions as a recognition processing unit that performs a predetermined recognition process based on the image of.

The image recognition system according to another aspect of the present disclosure is a position of the moving body from a storage unit that stores the camera mounted on the moving body in association with the region of interest in the image and the shooting position of the image. An acquisition unit that acquires an area of interest in an image around the moving object taken by the camera, a cutting unit that cuts out an image of the area of interest from an image around the moving object, and the above. A recognition processing unit that performs a predetermined recognition process based on an image of a region of interest is provided.

Needless to say, the computer program can be distributed via a computer-readable non-temporary recording medium such as a CD-ROM (Compact Disc-Read Only Memory) or a communication network such as the Internet. Further, the present disclosure can also be realized as a semiconductor integrated circuit that realizes a part or all of the image recognition device.

According to the present disclosure, an object in an image can be accurately recognized without delay.

FIG. 1 is a diagram showing an overall configuration of a driving support system according to the first embodiment of the present disclosure. FIG. 2 is a block diagram showing an example of the configuration of the control system according to the first embodiment of the present disclosure. FIG. 3 is a block diagram showing a functional configuration of the processor. FIG. 4 is a diagram showing an example of the area of interest information stored in the server. FIG. 5 is a diagram showing an example of a region of interest. FIG. 6 is a flowchart of processing executed by the control system according to the first embodiment of the present disclosure. FIG. 7 is a flowchart showing the details of the extraction success / failure determination process (step S3 in FIG. 6) according to the first embodiment of the present disclosure. FIG. 8 is a flowchart showing the details of the extraction success / failure determination process (step S3 in FIG. 6) according to the second embodiment of the present disclosure. FIG. 9 is a flowchart showing the details of the extraction success / failure determination process (step S3 in FIG. 6) according to the third embodiment of the present disclosure.

[Summary of Embodiments of the present disclosure]
First, an outline of the embodiments of the present disclosure will be listed and described.
(1) The image recognition device according to the embodiment of the present disclosure is from a storage unit that stores an area of interest in an image and a shooting position of the image in association with each other to the moving body based on the position of the moving body. An acquisition unit that acquires an image of interest in an image around the moving object taken by a mounted camera, a cutting unit that cuts out an image of the area of interest from an image around the moving object, and the area of interest. It is provided with a recognition processing unit that performs a predetermined recognition process based on the image of.

According to this configuration, information on the region of interest corresponding to the position of the moving body is acquired from the storage unit, and an image of the region of interest indicated by the acquired information is cut out from the image. Further, a predetermined recognition process is executed based on the image of the region of interest cut out. For example, when the moving body is a mobile robot that autonomously travels in the factory, the positions of the display device, the road sign, the factory passage, and the like in the factory do not change. Therefore, the areas of interest in the images taken at the same position are the same. Therefore, by acquiring the information of the region of interest from the storage unit, the image of the region of interest can be accurately cut out. This makes it possible to accurately recognize the object in the image without delay.

(2) Preferably, the extraction unit that extracts a predetermined region of interest from the image around the moving body, the determination unit that determines the success or failure of the extraction of the region of interest by the extraction unit, and the determination result of the determination unit. Based on this, the extraction unit further includes a selection unit that selects at least one of the attention region extracted by the extraction unit and the attention region acquired by the acquisition unit, and the cutting unit is the attention region selected by the selection unit. Cut out the image.

According to this configuration, when the extraction of the attention region from the image fails, the information of the attention region corresponding to the position of the moving body is acquired from the storage unit, and the image of the attention region indicated by the acquired information is cut out from the image. Is done. Further, a predetermined recognition process is executed based on the image of the region of interest cut out. For example, when the moving body is a mobile robot that autonomously travels in the factory, the positions of the display device, the road sign, the factory passage for pedestrian passage, etc. in the factory do not change. Therefore, the areas of interest in the images taken at the same position are the same. Therefore, by acquiring the information of the region of interest from the storage unit, the image of the region of interest can be accurately cut out even when the extraction of the region of interest fails due to reasons such as unclearness of the image. This makes it possible to accurately recognize the object in the image without delay.

(3) More preferably, the determination unit determines the success or failure of the extraction of the attention region based on the extraction result of the attention region.

According to this configuration, for example, when the region of interest cannot be extracted from the image, it can be determined that the extraction of the region of interest has failed.

(4) Further, the determination unit may determine the success or failure of the extraction of the attention region based on the certainty degree indicating the certainty of the extraction of the attention region.

According to this configuration, for example, when the certainty of the region of interest is lower than a predetermined criterion, it can be determined that the extraction of the region of interest has failed. As a result, when the certainty of the extracted region of interest is equal to or higher than a predetermined reference, the recognition process is executed based on the image of the region of interest. When the certainty of the extracted region of interest is less than a predetermined reference, the recognition process is executed based on the image of the region of interest corresponding to the position of the moving body. This makes it possible to accurately recognize the object in the image.

(5) Further, the determination unit may determine the success or failure of the extraction of the region of interest based on the abnormal state information at the time of taking the image.

According to this configuration, for example, when only an unclear image can be obtained outdoors due to abnormal weather conditions such as rain or snow, or when there is a power outage or disaster (for example, smoke generation due to a fire) indoors (for example, in a factory). When only an unclear image is obtained due to an abnormal state such as, it can be determined that the extraction of the region of interest has failed. As a result, the recognition process is executed based on the extracted image of the region of interest in a normal state such as when it is sunny or when no abnormality has occurred. Further, in an abnormal state such as rain, snow, or a disaster, the recognition process is executed based on the image of the region of interest corresponding to the position of the moving object. This makes it possible to accurately recognize the object in the image.

(6) Further, the selection unit may further select at least one of the attention area extracted by the extraction unit and the attention area acquired by the acquisition unit based on the selection instruction by the user.

According to this configuration, the region of interest is selected by the user's selection. This makes it possible to switch the region of interest depending on the situation. For example, when the user determines that the recognition accuracy of the recognition process based on the image of the attention area extracted by the extraction unit has deteriorated, the recognition process can be switched to the recognition process based on the image of the attention area corresponding to the position of the moving object. ..

(7) Further, the cutout portion further cuts out an image of a non-attention region including a region other than the attention region from the image, and the image recognition device reduces the amount of data of the image of the non-attention region. Further, the recognition processing unit may perform the predetermined recognition processing based on the image of the attention region and the image of the non-attention region in which the amount of data is reduced.

According to this configuration, the image in the non-attention region is subjected to a data amount reduction process such as a thinning process, and then a predetermined recognition process is executed. The non-attention area is an area other than the object of recognition, such as the sky or a building on the roadside. As a result, the processing load of the recognition processing for the non-attention region can be reduced, and the image can be processed without delay.

(8) Further, the storage unit further stores the image in association with the time at the time of image capture, and the acquisition unit stores the area of interest from the storage unit based on the position of the moving body and the current time. You may get it.

According to this configuration, for example, the region of interest extracted from the image taken at the time closest to the current time can be acquired. As a result, for example, when a moving object is traveling in a reversible lane whose traveling direction changes depending on a time zone, attention is extracted from an image taken by the moving object traveling in the reversible lane in a time zone opposite to the traveling direction. You no longer get the area. Further, for example, when the current time is nighttime, the area of interest extracted from the image taken in the daytime can be preferentially acquired. This makes it possible to obtain an accurate region of interest.

(9) Further, the storage unit further stores the size of the image in association with each other, and the acquisition unit further acquires the size of the image corresponding to the region of interest, and obtains the size and the size of the image. The acquired region of interest may be corrected based on the size of the image captured by the camera.

According to this configuration, even if the size of the image taken by the camera mounted on the moving object is different from the size of the image that is the basis for extracting the region of interest accumulated in the storage unit, it is accurate. An image of the region of interest at the position can be cut out.

(10) Further, the storage unit further stores the image in association with the angle of view of the image, and the acquisition unit further acquires and acquires the angle of view of the image corresponding to the region of interest. The acquired region of interest may be corrected based on the angle and the angle of view of the image taken by the camera.

According to this configuration, even if the angle of view of the image taken by the camera mounted on the moving object is different from the angle of view of the image that is the basis for extracting the region of interest accumulated in the storage unit. It is possible to cut out an image of the region of interest at an accurate position.

(11) In the image recognition method according to another embodiment of the present disclosure, the moving body is based on the position of the moving body from the storage unit that stores the region of interest in the image and the shooting position of the image in association with each other. A step of acquiring an image of interest in an image around the moving body taken by a camera mounted on the moving body, a step of cutting out an image of the region of interest from an image around the moving body, and a step of the area of interest. It includes a step of performing a predetermined recognition process based on the image.

This configuration includes the characteristic processing in the above-mentioned image recognition device as a step. Therefore, according to this configuration, the same operations and effects as those of the above-mentioned image recognition device can be obtained.

(12) The computer program according to another embodiment of the present disclosure is described above based on the position of a moving body from a storage unit that stores a computer in association with a region of interest in an image and a shooting position of the image. An acquisition unit that acquires an area of interest in an image around the moving object taken by a camera mounted on the moving object, and a cutting unit that cuts out an image of the area of interest from an image around the moving object. It functions as a recognition processing unit that performs a predetermined recognition process based on the image of the region of interest.

According to this configuration, the computer can function as the above-mentioned image recognition device. Therefore, the same operation and effect as the above-mentioned image recognition device can be obtained.

(13) The image recognition system according to another embodiment of the present disclosure is moved from a storage unit that stores a camera mounted on a moving body in association with a region of interest in an image and a shooting position of the image. An acquisition unit that acquires an area of interest in an image around the moving body taken by the camera based on the position of the body, and a cutting unit that cuts out an image of the area of interest from an image around the moving body. And a recognition processing unit that performs a predetermined recognition process based on the image of the region of interest.

According to this configuration, information on the region of interest corresponding to the position of the moving body is acquired from the storage unit, and an image of the region of interest indicated by the acquired information is cut out from the image. Further, a predetermined recognition process is executed based on the image of the region of interest cut out. For example, when the moving body is a mobile robot that autonomously travels in the factory, the positions of the display device, the road sign, the factory passage, and the like in the factory do not change. Therefore, the areas of interest in the images taken at the same position are the same. Therefore, by acquiring the information of the region of interest from the storage unit, the image of the region of interest can be accurately cut out. This makes it possible to accurately recognize the object in the image without delay.

[Details of Embodiments of the present disclosure]
Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. In addition, all of the embodiments described below show a specific example of the present disclosure. The numerical values, shapes, materials, components, arrangement positions and connection forms of the components, steps, the order of steps, and the like shown in the following embodiments are examples, and do not limit the present disclosure. Further, among the components in the following embodiments, the components not described in the independent claims are components that can be arbitrarily added. Further, each figure is a schematic view and is not necessarily exactly illustrated.
Further, the same components are designated by the same reference numerals. Since their functions and names are the same, their description will be omitted as appropriate.

<Embodiment 1>
[Overall configuration of driving support system]
FIG. 1 is a diagram showing an overall configuration of a driving support system according to the first embodiment of the present disclosure.

With reference to FIG. 1, the driving support system 1 includes a plurality of mobile bodies 2 traveling on a road capable of wireless communication, one or a plurality of base stations 6 that wirelessly communicate with the mobile body 2, and a base station 6. It includes a server 4 that communicates by wire or wirelessly via a network 5 such as the Internet.

The base station 6 includes a macrocell base station, a microcell base station, a picocell base station, and the like.

The mobile body 2 is, for example, a mobile robot such as a transport robot that transports luggage while autonomously traveling in a factory or a monitoring robot that monitors while autonomously traveling in a factory. In the present embodiment, the mobile body 2 is assumed to be a mobile robot. However, the mobile body 2 is not limited to the mobile robot traveling in the factory. The moving body 2 includes, for example, a normal passenger car traveling on a road and a public vehicle such as a route bus or an emergency vehicle. Further, the moving body 2 may be not only a four-wheeled vehicle but also a two-wheeled vehicle (motorcycle).

Each moving object 2 includes a control system 3 including a camera as described later, and the position of the region of interest in the image obtained by photographing the periphery of the moving object 2 with the camera, and at least the photographing position of the camera and the imaging position of the camera. The shooting direction is transmitted to the server 4 via the network 5. The shooting position of the camera is the same as the position of the moving body 2 when shooting with the camera. In this embodiment, it is assumed that the optical axis of the camera faces the front of the moving body 2. Therefore, the traveling direction of the moving body 2 is set as the shooting direction of the camera.

In the present embodiment, the region of interest is an region containing important information for the automatic operation of the mobile body 2, and is a region including static information or quasi-static information. For example, the area of interest is an area that includes, as static information, a display device, a road sign, a factory passage, etc. in a factory that constantly exists. In addition, the area of interest is an area that includes temporarily existing road construction sites, traffic accident sites, etc. as quasi-static information.

The server 4 receives the position of the region of interest and the set of attached data from the plurality of mobile bodies 2 via the network 5, and stores it in the storage unit of the server 4 as the region of interest information.

The server 4 receives at least the position information of the moving body 2 from the control system 3 via the network 5, and transmits the position information of the region of interest corresponding to the position of the moving body 2 to the control system 3 via the network 5. do.

The control system 3 receives the position information of the region of interest from the server 4 via the network 5. The control system 3 cuts out an image of the region of interest from the image captured by the control system 3 based on the received position information of the region of interest. The control system 3 performs a predetermined recognition process by performing a predetermined image process on the cut out region of interest image. For example, the control system 3 recognizes an object included in the region of interest image and performs processing according to the recognition result. As an example, when the object is a road sign indicating a suspension, the control system 3 performs braking control of the moving body 2 for safely suspending the moving body 2.

[Structure of mobile body 2]
FIG. 2 is a block diagram showing an example of the configuration of the control system 3 according to the first embodiment of the present disclosure.

As shown in FIG. 2, the control system 3 of the mobile body 2 includes a camera 31, a communication unit 32, a clock 33, a control unit (ECU: Electronic Control Unit) 34, and a GPS (Global Positioning System) receiver 37. And a gyro sensor 38 and a speed sensor 39.

The camera 31 comprises an image sensor that captures an image of the surroundings of the moving body 2 (particularly, in front of the moving body 2). The camera 31 is monocular. However, the camera 31 may have compound eyes. The video is composed of a plurality of images in time series.

The communication unit 32 includes, for example, a wireless communication device capable of communication processing compatible with 5G (fifth generation mobile communication system). The communication unit 32 may be an existing wireless communication device in the mobile body 2 or a mobile terminal brought into the mobile body 2 by the passenger.

The passenger's mobile terminal temporarily becomes an in-vehicle wireless communication device by being connected to the in-vehicle LAN (Local Area Network) of the mobile body 2.

The clock 33 clocks the current time.

The control unit 34 includes a computer device that controls each of the devices 31 to 33 and 37 to 39 of the mobile body 2. The control unit 34 obtains the position of the moving body of the self-moving body by the GPS signal periodically acquired by the GPS receiver 37. The control unit 34 complements the GPS signal or is a moving body of a self-moving object by using the GPS complement signal or the GPS augmentation signal received by the receiver of the signal transmitted from the quasi-zenith satellite (not shown) together. The position may be corrected. It is preferable that the position of the moving body has a position accuracy enough to distinguish the lanes.

The control unit 34 complements the position and direction of the moving body 2 based on the input signals of the gyro sensor 38 and the speed sensor 39, and grasps the accurate current position and direction of the moving body 2. Here, the current position of the moving body 2 is indicated by, for example, latitude and longitude. Further, the direction (traveling direction) of the moving body 2 is indicated by, for example, an angle in the range of 0 degrees to 360 degrees clockwise with the north as 0 degrees.

The GPS receiver 37, the gyro sensor 38, and the speed sensor 39 are sensors that measure the current position, direction, and speed of the moving body 2.

The control unit 34 includes a processor 35 and a memory 36.
The processor 35 is an arithmetic processing unit such as a microcomputer that executes a computer program stored in the memory 36.

The memory 36 is a volatile memory element such as a SRAM (Static RAM) or a DRAM (Dynamic RAM), a flash memory or a non-volatile memory element such as an EEPROM (Electrically Erasable Specramable Read Only Memory), or a magnetic storage such as a hard disk. It is composed of devices and the like. The memory 36 stores a computer program executed by the control unit 34, data generated when the computer program is executed by the control unit 34, and the like.

[Functional configuration of processor 35]
FIG. 3 is a block diagram showing a functional configuration of the processor 35.

With reference to FIG. 3, the processor 35 includes an image acquisition unit 51, an attention area extraction unit 52, and an extraction success / failure determination as functional processing units realized by executing a computer program stored in the memory 36. A unit 53, an attention area acquisition unit 54, a moving body position determination unit 55, an attention area selection unit 56, a cutting unit 57, a data amount reduction unit 58, and a recognition processing unit 59 are included.

The image acquisition unit 51 sequentially acquires images in front of the moving body 2 taken by the camera 31 in chronological order. The image acquisition unit 51 sequentially outputs the acquired images to the region of interest extraction unit 52 and the cutout unit 57.

The attention area extraction unit 52 acquires an image from the image acquisition unit 51, and extracts the attention area from the acquired image. For example, the attention region extraction unit 52 obtains the attention region by inputting the acquired image into the learning model using the image and the learning model machine-learned using the attention region in the image as learning data. Further, the attention region extraction unit 52 obtains a certainty degree indicating the certainty of the attention region from the learning model. The learning model is, for example, CNN (Convolution Natural Network), RNN (Recurrent Neural Network), AutoEncoder, etc., and it is assumed that each parameter of the learning model is determined by a machine learning method such as deep learning.

In the present embodiment, the region of interest is a rectangle. Further, the region of interest is defined by the coordinates of the upper left corner of the rectangle and the widths of the rectangle in the X and Y directions. However, the position of the region of interest is not limited to the above. For example, the region of interest may be defined by the coordinates of the upper left corner and the coordinates of the lower right corner of the rectangle. Further, the region of interest may have a shape other than a rectangle such as an ellipse.

The attention area extraction unit 52 outputs the extracted attention area and certainty to the extraction success / failure determination unit 53 and the attention area selection unit 56.

The extraction success / failure determination unit 53 determines the success / failure of the extraction of the region of interest by the region of interest 52. Specifically, the extraction success / failure determination unit 53 determines the success / failure of the extraction of the region of interest based on the certainty of the extraction of the region of interest 52. For example, the extraction success / failure determination unit 53 determines that the extraction of the attention region is successful if the certainty is equal to or higher than the predetermined threshold TH, and fails to extract the attention region if the certainty is less than the predetermined threshold TH. judge.

When a plurality of areas of interest are included in the image, the success or failure of extraction of the areas of interest may be determined based on a plurality of certainty levels. For example, the extraction success / failure determination unit 53 determines that the extraction of the region of interest has been successful if the minimum certainty of the plurality of certainty is equal to or higher than the predetermined threshold TH, and the minimum certainty is less than the predetermined threshold TH. If so, it may be determined that the extraction of the region of interest has failed. Also, instead of the minimum certainty, the average of a plurality of certainty may be used. Further, the extraction success / failure determination unit 53 may determine the success / failure of the extraction for each region of interest.

The extraction success / failure determination unit 53 outputs the determination result to the attention area extraction unit 52, the attention area acquisition unit 54, the moving body position determination unit 55, and the attention area selection unit 56.

When it is determined that the region of interest has been successfully extracted, the region of interest 52 transmits the position of the region of interest and at least the shooting position and shooting direction of the camera 31 to the server 4 via the communication unit 32. do. The server 4 stores the information received from the mobile body 2 as the area of interest information 41.

The attention area acquisition unit 54 acquires an attention area from the attention area information 41 stored in the server 4 based on the position of the mobile body 2 determined by the mobile body position determination unit 55 described later.

FIG. 4 is a diagram showing an example of the attention area information 41 stored in the server 4.
The attention area information 41 includes a moving body ID, a shooting time, a shooting position, a attention area, a moving direction, an angle of view, and an image size. Each mobile unit 2 uploads this information to the server 4.

The mobile ID is the identification information of the uploaded mobile 2. The mobile ID may be an ID unique to the mobile 2 or an ID unique to the user of the mobile 2.
The shooting time indicates the shooting time of the camera 31 mounted on the moving body 2.
The shooting position is the shooting position of the camera 31, and is equal to the position of the moving body 2 at the time of image shooting. The shooting position is indicated by, for example, north latitude N and east longitude E.

The region of interest indicates the position (X, Y, W, H) of the region of interest extracted from the image in the image. X and Y indicate the X and Y coordinates of the upper left corner coordinate of the region of interest, and W and H indicate the width of the region of interest in the X and Y directions, respectively.
The moving direction indicates the traveling direction of the moving body 2.
The angle of view indicates the angle of view of the camera 31.

The image size indicates the size of the image captured by the camera 31. For example, 4K indicates 3840 × 2160 (pixel) and 8K indicates 7680 × 4320 (pixel).

For example, the five areas of interest extracted from the image taken by the moving body 2 with the moving body ID “C001” at the shooting position (N1, E1) at the shooting time “9:00:03” are (X1, Y1). , W1, H1), (X2, Y2, W2, H2), (X3, Y3, W3, H3), (X4, Y4, W4, H4) and (X5, Y5, W5, H5). The direction of the moving body 2 is 45 °, the angle of view of the image is 60 °, and the image size is 4K.

Further, the region of interest extracted from the image taken by the moving body 2 having the moving body ID “C001” at the shooting position (N2, E2) at the shooting time “9:01:45” is (X6, Y6, W6). , H6). The direction of the moving body 2 is 48 °, the angle of view of the image is 60 °, and the image size is 4K.

Further, the region of interest extracted from the image taken by the moving body 2 of the moving body ID “C002” at the shooting position (N1, E1) at the shooting time “20:02:55” is (X7, Y7, W7). , H7). The direction of the moving body 2 is 45 °, the angle of view of the image is 120 °, and the image size is 8K.

FIG. 5 is a diagram showing an example of a region of interest. FIG. 5 shows a rectangular area of interest on an image taken by the camera 31. For example, the area of interest (X1, Y1, W1, H1) is set to include a road sign, and the area of interest (X2, Y2, W2, H2), (X3, Y3, W3, H3) includes a factory passage. , (X4, Y4, W4, H4) and (X5, Y5, W5, H5) are set.

With reference to FIG. 3 again, for example, when the position of the moving body 2 determined by the moving body position determination unit 55 is (N2, E2), the attention area acquisition unit 54 takes a picture from the attention area information 41. The region of interest (X6, Y6, W6, H6) corresponding to the position (N2, E2) is acquired.

However, the attention area acquisition unit 54 may acquire the attention area from the attention area information 41 based on the position of the moving body 2 and the current time. For example, when the position of the moving body 2 determined by the moving body position determination unit 55 is (N1, E1), the attention area information 41 is photographed as an attention area corresponding to the shooting position (N1, E1). Areas of interest (X1, Y1, W1, H1), (X2, Y2, W2, H2), (X3, Y3, W3, H3), (X4, Y4, W4, H4) at time "9:00:03" And (X5, Y5, W5, H5) and the region of interest (X7, Y7, W7, H7) at the shooting time "20:02:55" are included. Therefore, the attention area acquisition unit 54 acquires the attention area corresponding to the time closest to the current time from the attention area information 41. For example, when the current time is "9:30:00", the attention area acquisition unit 54 may use the attention area (X1, Y1, W1, H1), (X2) corresponding to the shooting time "9:00:03". , Y2, W2, H2), (X3, Y3, W3, H3), (X4, Y4, W4, H4) and (X5, Y5, W5, H5). Further, when the current time is "19:00:00", the attention area acquisition unit 54 acquires the attention area (X7, Y7, W7, H7) corresponding to the shooting time "20:02:55". ..

It is desirable that the attention area acquisition unit 54 acquires the attention area in consideration of the direction of the moving body 2 in addition to the position of the moving body 2. As a result, even at the same position, the attention area information obtained from the moving body 2 traveling in the same direction is acquired without acquiring the attention area information obtained from the moving body 2 traveling in the opposite direction. be able to. However, if the moving body 2 is traveling on a one-way road, or if the traveling lane of the moving body 2 can be specified from the position of the moving body 2, the direction of the moving body 2 does not need to be considered. ..

The moving body position determination unit 55 determines the position of the moving body 2. Specifically, the mobile body position determination unit 55 obtains the position of the mobile body 2 by the GPS signal periodically acquired by the GPS receiver 37. The moving body position determination unit 55 complements the position and direction of the moving body 2 based on the input signals of the gyro sensor 38 and the speed sensor 39, and grasps the accurate current position and direction of the moving body 2. Further, the moving body position determination unit 55 refers to the map information 61 stored in advance in the memory 36, and performs a map matching process of associating the obtained position of the moving body 2 with the position on the road, thereby performing the moving body position determination unit 55. Determine the exact position of 2.

The attention area selection unit 56 selects at least one of the attention area extracted by the attention area extraction unit 52 and the attention area acquired by the attention area acquisition unit 54 based on the determination result of the extraction success / failure determination unit 53.

For example, when it is determined that the attention area selection unit 56 has succeeded in extracting the attention area, the attention area extraction unit 52 selects the attention area extracted by the attention area extraction unit 52, and it is determined that the extraction of the attention area has failed. Selects the area of interest acquired by the area of interest acquisition unit 54. When it is determined that the attention area extraction is successful, the attention area selection unit 56 selects both the attention area extracted by the attention area extraction unit 52 and the attention area acquired by the attention area acquisition unit 54. You may.

The cutout unit 57 cuts out an image of the region of interest (image of the region of interest) and an image other than the region of interest (image of the region of interest) from the image acquired by the image acquisition unit 51. The cutout unit 57 outputs the attention region image to the recognition processing unit 59, and outputs the non-attention region image to the data amount reduction unit 58.

The data amount reduction unit 58 performs a data amount reduction process of the non-attention area image. For example, the data amount reduction unit 58 may perform downscaling processing to reduce the non-attention region image, or reduce the number of bits indicating the luminance value of each pixel of the non-attention region image to reduce the degree of gradation (color depth). ) May be reduced. Further, the data amount reduction unit 58 may perform a temporal thinning process of the non-attention region image (for example, a process of deleting the non-attention region obtained from the even-numbered image among the images in time series). The data amount reduction unit 58 outputs the non-attention area image after the data amount reduction processing to the recognition processing unit 59.

The recognition processing unit 59 executes a predetermined recognition process based on the attention area image received from the cutting unit 57 and the non-attention area image received from the data amount reduction unit 58. For example, the recognition processing unit 59 performs predetermined image processing on the image of the region of interest, and recognizes a display device in the factory, a road sign for a stop sign, the presence or absence of a pedestrian on the factory passage, and the like. The recognition result of the recognition processing unit 59 is used, for example, for controlling the automatic operation of the moving body 2. The recognition processing unit 59 obtains the recognition result of the region of interest image by inputting the image of the region of interest into the learning model machine-learned using, for example, an image and a label indicating the recognition result as training data. The learning model is, for example, CNN, RNN, AutoEncoder, or the like, and it is assumed that each parameter of the learning model is determined by a method such as deep learning.
The recognition processing unit 59 may also perform the same processing on the non-attention region image.

[Process flow of control system 3]
FIG. 6 is a flowchart of the process executed by the control system 3 according to the first embodiment of the present disclosure.
The image acquisition unit 51 acquires an image from the camera 31 (step S1).
The attention area extraction unit 52 extracts the certainty of the attention area and the attention area based on the image acquired by the image acquisition unit 51 (step S2).
The extraction success / failure determination unit 53 determines the success / failure of the extraction of the region of interest (step S3).

FIG. 7 is a flowchart showing the details of the extraction success / failure determination process (step S3 in FIG. 6) according to the first embodiment of the present disclosure.
The extraction success / failure determination unit 53 determines whether or not the certainty of the attention region extracted by the attention region extraction unit 52 is equal to or higher than a predetermined threshold value TH (step S21).
If the certainty level is equal to or higher than the threshold value TH (YES in step S21), the extraction success / failure determination unit 53 determines that the extraction of the region of interest has been successful (step S22).
If the certainty is less than the threshold value TH (NO in step S21), the extraction success / failure determination unit 53 determines that the extraction of the region of interest has failed (step S23).

When a plurality of areas of interest are extracted in step S2, for example, the processes of steps S21 to S23 are executed based on the minimum certainty. If the region of interest is not extracted, it is determined that the region of interest has failed to be extracted.

When it is determined that the extraction of the attention region is successful (YES in step S4) with reference to FIG. 6 again, the attention region extraction unit 52 determines the position of the attention region and at least the shooting position and shooting of the camera 31. The region of interest information including the direction is transmitted to the server 4 via the communication unit 32 (step S5).
Further, the attention area selection unit 56 selects the attention area extracted in step S2 and outputs it to the cutting unit 57 (step S6).

If it is determined that the extraction of the region of interest has failed (NO in step S4), the moving body position determination unit 55 determines the position and direction of the moving body 2 (step S7).
The attention area acquisition unit 54 reads out the attention area information corresponding to the position and direction from the server 4 based on the determined position and direction of the moving body 2 (step S8).

The attention area acquisition unit 54 corrects the attention area based on the read out attention area information (step S9). For example, when the image size of the camera 31 mounted on the target moving body 2 is 4K and the image size indicated by the read attention area information is 8K, the value of the attention area indicated by the attention area information is halved. By setting, the value is corrected to the value in the region of interest of the 4K image. Similarly, in the attention area acquisition unit 54, when the angle of view of the camera 31 mounted on the target moving body 2 is 45 ° and the angle of view indicated by the attention area information is 60 °, the attention area information is obtained. The value of the region of interest shown is converted into a value on the image having an angle of view of 45 °. The conversion process is performed using, for example, a conversion table showing the correspondence between positions between different angle of view.

The attention area selection unit 56 selects the attention area acquired in step S8 and corrected in step S9, and outputs the output to the cutting unit 57 (step S10).

The cutting unit 57 cuts out an image of the region of interest from the image acquired in step S1 (step S11). Further, the cutting unit 57 cuts out a non-attention region image from the image acquired in step S1 (step S12).

The data amount reduction unit 58 performs a data amount reduction process on the non-attention region image cut out in step S12 (step S13).

The recognition processing unit 59 recognizes the attention region image and the non-attention region image based on the attention region image cut out in step S11 and the non-attention region image subjected to the data amount reduction processing in step S13. This is executed, and the traveling control process of the moving body 2 is performed based on the recognition result (step S14).

[Effects of Embodiment 1]
As described above, according to the first embodiment of the present disclosure, the information of the attention area corresponding to the position of the moving body 2 is acquired from the server 4, and the image of the attention area indicated by the acquired information is cut out from the image. .. Further, a predetermined recognition process is executed based on the image of the region of interest cut out. For example, the location of display devices, road signs or factory walkways within a factory does not change. Therefore, the areas of interest in the images taken at the same position are the same. Therefore, by acquiring the information of the region of interest from the server 4, the image of the region of interest can be accurately cut out. This makes it possible to accurately recognize the object in the image without delay.

Further, when the extraction of the attention region from the image fails, the information of the attention region corresponding to the position of the moving body 2 is acquired from the server 4, and the acquired image of the attention region is cut out from the image. Thereby, for example, even if the extraction of the region of interest fails due to the image being unclear or the like, the image of the region of interest can be accurately cut out.

The extraction success / failure determination unit 53 determines that the extraction of the region of interest has failed when the certainty of the region of interest is less than the threshold value TH, and determines that the extraction of the region of interest has succeeded when the certainty of the region of interest is greater than or equal to the threshold value TH. ing. As a result, when the certainty of the extracted attention region is equal to or higher than the threshold value TH, the recognition process is executed based on the image of the attention region. When the certainty of the extracted region of interest is less than the threshold value TH, the recognition process is executed based on the image of the region of interest corresponding to the position of the moving body 2. This makes it possible to accurately recognize the object in the image.

Further, the non-attention area image is subjected to a data amount reduction process such as a thinning process, and then a predetermined recognition process is executed. The non-attention area is an area other than the object of recognition, such as the sky or a building on the roadside. As a result, the processing load of the recognition processing for the non-attention region can be reduced, and the image can be processed without delay.

Further, since the attention area acquisition unit 54 acquires the attention area based on the position of the moving body 2 and the current time, for example, the attention area acquired from the image taken at the time closest to the current time is acquired. can do. As a result, for example, when the moving body 2 is traveling in the reversible lane whose traveling direction changes depending on the time zone, from the image taken by another moving body 2 traveling in the reversible lane in the time zone opposite to the traveling direction. It is no longer necessary to acquire the extracted attention area. Further, for example, when the current time is nighttime, the region of interest extracted from the image taken in the daytime may be preferentially acquired. The image is clearer in the daytime than in the nighttime. This makes it possible to obtain an accurate region of interest.

Further, the attention area acquisition unit 54 corrects the attention area acquired from the server 4 based on the image size. Therefore, even if the size of the image taken by the camera 31 mounted on the moving body 2 is different from the size of the image that is the basis for extracting the region of interest stored in the server 4, it is accurate. An image of the region of interest at the position can be cut out.

Further, the attention area acquisition unit 54 corrects the attention area acquired from the server 4 based on the angle of view. Therefore, even if the angle of view of the image taken by the camera 31 mounted on the moving body 2 is different from the angle of view of the image that is the basis for extracting the region of interest stored in the server 4. It is possible to cut out an image of the region of interest at an accurate position.

<Embodiment 2>
In the second embodiment of the present disclosure, another example of the success / failure determination method for extracting the region of interest will be described.
The configuration of the driving support system 1 is the same as that of the first embodiment. Hereinafter, the points different from those of the first embodiment will be mainly described.

FIG. 8 is a flowchart showing the details of the extraction success / failure determination process (step S3 in FIG. 6) according to the second embodiment of the present disclosure.
The extraction success / failure determination unit 53 determines whether or not there is an extraction region extracted by the region of interest extraction unit 52 (step S31).

If there is an extracted region of interest (YES in step S31), the extraction success / failure determination unit 53 determines that the region of interest has been successfully extracted (step S32).
If there is no extracted region of interest (NO in step S31), the extraction success / failure determination unit 53 determines that the extraction of the region of interest has failed (step S33).

According to the second embodiment of the present disclosure, for example, when the region of interest cannot be extracted from the image, it can be determined that the extraction of the region of interest has failed.

<Embodiment 3>
In the third embodiment of the present disclosure, another example of the success / failure determination method for extracting the region of interest will be described.
It is assumed that the control system 3 is connected to a meteorological server (not shown) that provides meteorological information on the position of the mobile body 2 via a network 5.
Therefore, the extraction success / failure determination unit 53 determines the success / failure of extraction of the region of interest based on the weather information of the position of the moving body 2 at the time of image capture by the camera 31.

FIG. 9 is a flowchart showing the details of the extraction success / failure determination process (step S3 in FIG. 6) according to the third embodiment of the present disclosure.
The extraction success / failure determination unit 53 acquires the weather information of the position of the moving body 2 at the time of image capture by the camera 31 from the weather server via the communication unit 32 (step S41).

When the acquired weather information indicates rain or snow (YES in step S41), the extraction success / failure determination unit 53 determines that the extraction of the region of interest has failed (step S43).

When the acquired meteorological information indicates other than rain or snow (NO in step S42), the extraction success / failure determination unit 53 determines that the extraction of the region of interest has been successful (step S44).

According to the third embodiment of the present disclosure, for example, when only an unclear image can be obtained outdoors due to an abnormal state of weather such as rain or snow, it can be determined that the extraction of the region of interest has failed. As a result, in the case of sunny weather, the recognition process is executed based on the extracted image of the region of interest. Further, in the case of rain or snow, the recognition process is executed based on the image of the region of interest corresponding to the position of the moving body 2. This makes it possible to accurately recognize the object in the image.

[Additional Notes]
Although the driving support system 1 according to the embodiment of the present disclosure has been described above, the present disclosure is not limited to this embodiment.

For example, the attention area selection unit 56 may select at least one of the attention area extracted by the attention area extraction unit 52 and the attention area acquired by the attention area acquisition unit 54 based on the selection instruction by the user. This makes it possible to switch the region of interest depending on the situation. For example, when the user determines that the recognition accuracy of the recognition process based on the image of the attention area extracted by the attention area extraction unit 52 has deteriorated, the recognition process based on the image of the attention area corresponding to the position of the moving body 2 is performed. You can switch.

Further, the extraction success / failure determination unit 53 may determine the success / failure of the extraction of the region of interest based on the shooting time of the image by the camera 31. For example, the extraction success / failure determination unit 53 may determine that the extraction of the attention region has failed when the shooting time is nighttime, and may determine that the extraction of the attention region has succeeded when the shooting time is daytime.

Further, the extraction success / failure determination unit 53 may determine the success / failure of the extraction of the region of interest based on the image taken by the camera 31 acquired by the image acquisition unit 51. For example, the extraction success / failure determination unit 53 detects an abnormal state such as a power failure or a disaster (for example, generation of smoke due to a fire) indoors (for example, in a factory) by recognizing an image, and the abnormal state occurs. If so, it may be determined that the extraction of the region of interest has failed. Further, the extraction success / failure determination unit 53 may determine that the extraction of the region of interest has been successful when no abnormal state has occurred.

Further, the target on which the control system 3 is mounted is not limited to a moving body such as a mobile robot. For example, the control system 3 may be mounted on another moving body such as an automobile or a drone.

Further, the purpose of the recognition processing performed by the recognition processing unit 59 is not limited to automatic operation. For example, the control system 3 may be mounted on a mobile robot moving in a factory and used for monitoring to prevent dangerous acts of workers.
Further, a part or all of the components constituting the control system 3 may be composed of one or a plurality of semiconductor devices such as system LSIs.

The above-mentioned computer program may be recorded and distributed on a computer-readable non-temporary recording medium such as an HDD, a CD-ROM, or a semiconductor memory. Further, the computer program may be transmitted and distributed via a telecommunication line, a wireless or wired communication line, a network typified by the Internet, data broadcasting, or the like.
Further, at least a part of the above embodiment and the above modification may be arbitrarily combined.

The embodiments disclosed this time should be considered to be exemplary and not restrictive in all respects. The scope of the present disclosure is expressed by the scope of claims, not the above-mentioned meaning, and is intended to include all modifications within the meaning and scope equivalent to the scope of claims.

1 Driving support system 2 Mobile 3 Control system (image recognition system)
4 Server (storage unit)
5 Network 6 Base station 31 Camera 32 Communication unit 33 Clock 34 Control unit 35 Processor (image recognition device)
36 Memory 37 GPS receiver 38 Gyro sensor 39 Speed sensor 41 Focus area information 51 Image acquisition section 52 Focus area extraction section (extraction section)
53 Extraction success / failure judgment unit (judgment unit)
54 Area of interest acquisition unit (acquisition unit)
55 Moving body position determination unit 56 Focus area selection unit (selection unit)
57 Cutout part 58 Data amount reduction part (reduction part)
59 Recognition processing unit 61 Map information

Claims (13)

In the image around the moving body taken by the camera mounted on the moving body based on the position of the moving body from the storage unit that accumulates the area of interest in the image in association with the shooting position of the image. The acquisition unit that acquires the area of interest of
A cutout portion that cuts out an image of the region of interest from an image around the moving body, and
An image recognition device including a recognition processing unit that performs a predetermined recognition process based on an image of the region of interest. An extraction unit that extracts a predetermined region of interest from an image around the moving object, and an extraction unit.
A determination unit that determines the success or failure of extraction of the region of interest by the extraction unit,
A selection unit for selecting at least one of the attention region extracted by the extraction unit and the attention region acquired by the acquisition unit based on the determination result of the determination unit is further provided.
The image recognition device according to claim 1, wherein the cutout unit cuts out an image of the area of interest selected by the selection unit. The image recognition device according to claim 2, wherein the determination unit determines the success or failure of the extraction of the attention region based on the extraction result of the attention region. The image recognition device according to claim 2 or 3, wherein the determination unit determines the success or failure of the extraction of the attention region based on the certainty indicating the certainty of the extraction of the attention region. The image recognition device according to any one of claims 2 to 4, wherein the determination unit determines the success or failure of extraction of the region of interest based on the abnormal state information at the time of image capture. Any of claims 2 to 5, wherein the selection unit further selects at least one of the attention area extracted by the extraction unit and the attention area acquired by the acquisition unit based on a selection instruction by the user. The image recognition device according to claim 1. The cutout portion further cuts out an image of a non-attention region including a region other than the attention region from the image.
The image recognition device further includes a reduction unit that reduces the amount of data of the image in the non-attention region.
The aspect according to any one of claims 1 to 6, wherein the recognition processing unit performs the predetermined recognition process based on the image of the attention region and the image of the non-attention region in which the amount of data is reduced. Image recognition device. The storage unit further stores the time in association with the time at which the image was taken.
The image recognition device according to any one of claims 1 to 7, wherein the acquisition unit acquires the area of interest from the storage unit based on the position of the moving body and the current time. The storage unit further stores the image in association with the size of the image.
The acquisition unit further acquires the size of the image corresponding to the region of interest, and corrects the acquired region of interest based on the acquired size and the size of the image captured by the camera. , The image recognition device according to any one of claims 1 to 8. The storage unit further stores the image in association with the angle of view of the image.
The acquisition unit further acquires the angle of view of the image corresponding to the region of interest, and the acquired region of interest is obtained based on the acquired angle of view and the angle of view of the image taken by the camera. The image recognition device according to any one of claims 1 to 9, wherein the image recognition device is corrected. In the image around the moving body taken by the camera mounted on the moving body based on the position of the moving body from the storage unit that accumulates the area of interest in the image in association with the shooting position of the image. Steps to get the area of interest and
A step of cutting out an image of the region of interest from an image around the moving object,
An image recognition method including a step of performing a predetermined recognition process based on an image of the region of interest. Computer,
In the image around the moving body taken by the camera mounted on the moving body based on the position of the moving body from the storage unit that accumulates the area of interest in the image in association with the shooting position of the image. The acquisition unit that acquires the area of interest of
A cutout portion that cuts out an image of the region of interest from an image around the moving body,
A computer program for functioning as a recognition processing unit that performs a predetermined recognition process based on the image of the region of interest. The camera mounted on the moving body and
Based on the position of the moving body, the area of interest in the image around the moving body taken by the camera is acquired from the storage unit that stores the area of interest in the image in association with the shooting position of the image. Acquisition department and
A cutout portion that cuts out an image of the region of interest from an image around the moving body,
An image recognition system including a recognition processing unit that performs a predetermined recognition process based on an image of the region of interest.

Images