JP2021192224A - Method and device, electronic device, computer-readable storage medium, and computer program for detecting pedestrian - Google Patents

Abstract

To provide a method and a device, an electronic device, a computer-readable storage medium, and a computer program for detecting a pedestrian, which relate to a field of image processing for pedestrian detection.SOLUTION: A specific embodiment of a method comprises the steps of: acquiring a target image; obtaining a first trimming image by trimming the target image; obtaining a feature map by extracting a feature of the first trimmed image; obtaining a second trimming image by trimming the feature map; and obtaining a detection result of a pedestrian by identifying the pedestrian in the second trimmed image. The embodiment can reduce an amount of calculation of the image, reduce an amount of a calculation resource use, and can be more conveniently applied to a mobile terminal having a low hardware level.SELECTED DRAWING: Figure 2

Description

Examples of the present invention relate to the field of computer technology, particularly to the field of image processing, specifically to methods and devices for detecting pedestrians, electronic devices, computer readable storage media and computer programs.

With continued advances in the accuracy of pedestrian detection algorithms and the urgent need for low-complexity pedestrian detection on the trip computer side, more and more pedestrian detection algorithms are being placed on mobile terminals. These mobile devices have low hardware levels and low computing power. Currently, effective pedestrian detection algorithms use deep learning techniques, but deep learning algorithms require a lot of computational resources. These deep learning algorithms often do not work on mobile devices with low hardware levels.

In the embodiments of the present invention, there are proposed methods and devices for detecting pedestrians, electronic devices, computer-readable storage media and computer programs.

In the first aspect, a feature map is obtained by acquiring a target image, obtaining a first trimmed image by trimming the target image, and extracting features of the first trimmed image. It includes obtaining, obtaining a second trimmed image by trimming the feature map, and obtaining a pedestrian detection result by identifying a pedestrian in the second trimmed image. , Concerning methods for detecting pedestrians.

In the second aspect, an image acquisition unit configured to acquire a target image, a first trimming unit configured to obtain a first trimmed image by trimming the target image, and the above. A feature extraction unit configured to obtain a feature map by extracting the features of the first trimmed image, and a first configured to obtain a second trimmed image by trimming the feature map. 2. A device for detecting a pedestrian, including a trimming unit 2 and a pedestrian detection unit configured to obtain a pedestrian detection result by identifying the pedestrian in the second trimmed image. ..

In a third aspect, an embodiment of the present invention includes one or more processors and a storage device in which one or more programs are stored, and the one or more programs are executed by the one or more processors. The present invention relates to an electronic device that causes one or more processors to perform the method described in any embodiment of the first aspect.

In a fourth aspect, the embodiment of the present invention is a computer-readable medium in which a computer program is stored, and the method described in any of the first embodiments when the computer program is executed by a processor. Regarding computer-readable media that realizes.

In a fifth aspect, an embodiment of the invention relates to a computer program that, when the computer program is executed by a processor, realizes the method described in any of the first embodiments.

According to the technique of the present invention, the problem that the calculation amount of the existing pedestrian detection algorithm is large is solved, and the calculation amount of the image is reduced and the usage amount of the calculation resource is reduced by trimming the target image. , Can be applied more conveniently to mobile terminals with low hardware level.

It should be understood that the content described herein is not intended to identify the key or important features of the embodiments of the present disclosure and is not intended to limit the scope of the present disclosure. Other features of the present disclosure are readily understood by the following description.

In order to further clarify the other features, purposes and advantages of the present invention, reference is made to the detailed description of the non-limiting examples made with reference to the following drawings.

It is an exemplary system architecture diagram to which one embodiment of the present invention can be applied. It is a flowchart of one Embodiment of the method for detecting a pedestrian by this invention. It is a schematic diagram which shows the application scene by 1st Example shown in FIG. It is a flowchart of one Embodiment of the method for detecting a pedestrian by this invention. It is a schematic block diagram of one Example of the apparatus for detecting a pedestrian by this invention. It is a schematic block diagram of the computer system of the electronic device applied to realize the embodiment of this invention.

Hereinafter, the present invention will be described in more detail with reference to the drawings and examples. It is understood that the specific examples described herein are merely for explaining the related invention and not limiting the invention. For ease of explanation, the drawings show only the parts relating to the related invention.

If there is no contradiction, the embodiments of the present invention and the features of the embodiments can be combined with each other. Hereinafter, the present invention will be described in detail according to examples with reference to the drawings.

FIG. 1 shows an exemplary system architecture 100 of an embodiment of a method for detecting a pedestrian or a device for detecting a pedestrian to which the present invention may be applied.

As shown in FIG. 1, the system architecture 100 can include terminal devices 101, 102, 103 and an image acquisition device 104. The image acquisition device 104 and the terminal devices 101, 102, 104 are connected by network communication, and the network can include various connection types such as, for example, wired, wireless communication links, or fiber optic cables.

The terminal devices 101, 102, and 103 can receive the images collected by the image collecting device 104 via the network and process the images. The image collecting device 104 can be attached to the terminal devices 101, 102, 103.

The terminal devices 101, 102, and 103 may be hardware or software. When the terminal devices 101, 102, 103 are hardware, they may be various electronic devices that have a display and support image processing, including smartphones, tablet computers, laptop computers, desktop computers, and the like. , Not limited to these. When the terminal devices 101, 102, 103 are software, they can be attached to the above-mentioned electronic device. It may be implemented as multiple software or software modules (eg, providing distributed services) or as a single software or software module. Here, it is not specifically limited.

The image collecting device 104 may be various devices for collecting images such as a camera and a video camera. The image collecting device 104 can transmit the collected images to the terminal devices 101, 102, 103 in real time.

The method for detecting a pedestrian according to the embodiment of the present invention may be executed by the terminal devices 101, 102, 103, or may be executed by the image collecting device 104. Correspondingly, the device for detecting a pedestrian may be generally arranged in the terminal devices 101, 102, 103, or may be arranged in the image collecting device 104.

It should be understood that the number of terminal devices and image acquisition devices in FIG. 1 is merely exemplary. If necessary, any number of terminal devices and image acquisition devices can be provided.

Subsequently, with reference to FIG. 2, a flow 200 of an embodiment of the method for detecting a pedestrian according to the present invention is shown. The method for detecting a pedestrian in this embodiment includes the following steps.

In step 201, the target image is acquired.

In this embodiment, the executing subject of the method for detecting a pedestrian (for example, the terminal device 101, 102, 103 or the image collecting device 104 shown in FIG. 1) acquires a target image by a wired connection or a wireless connection. be able to. Here, the target image can be any image including a pedestrian.

In step 202, the target image is trimmed to obtain a first cropped image.

The execution subject can obtain the first trimmed image by trimming the image after obtaining the target image. Specifically, the executing subject can obtain the first cropped image by trimming a part of the upper region. Alternatively, the execution subject can also obtain the first cropped image by trimming the target image to a preset size. It can be understood that the portion trimmed by the performer does not have to contain valid pedestrian information, that is, the trimmed portion includes pedestrians walking on the pedestrian bridge. The first cropped image contains valid pedestrian information.

In step 203, a feature map is obtained by extracting the features of the first cropped image.

The execution subject can obtain a feature map by extracting the features of the first trimmed image after obtaining the first trimmed image. Specifically, the execution subject can obtain a feature map by performing feature extraction on the first trimmed image using various feature extraction algorithms. The feature extraction algorithm may include a convolutional neural network or the like. The feature map may include valid pedestrian information such as the contour, position, and center of the pedestrian.

In step 204, the feature map is trimmed to obtain a second cropped image.

The execution subject can obtain a second trimmed image by trimming the feature map after obtaining the feature map. After the feature extraction in step 203, it can be understood that the feature map contains some information of the pedestrian, that is, the information necessary for pedestrian detection. In addition to this, a second cropped image can be obtained by further trimming the feature map in order to further reduce the amount of calculation. Here, the execution subject can trim a part of the area above the feature map. Alternatively, the execution subject can trim the feature map to a preset size.

In step 205, the pedestrian detection result is obtained by identifying the pedestrian in the second cropped image.

After obtaining the second trimmed image, the executing subject obtains a pedestrian detection result by performing pedestrian detection on the second trimmed image. Specifically, the executing subject can obtain a pedestrian detection result by processing the second trimmed image using various pedestrian detection algorithms. The pedestrian detection algorithm can include various neural networks.

In the method for detecting a pedestrian according to the above embodiment of the present invention, the characteristics of the pedestrian can be extracted while trimming the target image multiple times, reducing the amount of calculation and ensuring the accuracy of pedestrian detection. can.

Subsequently referring to FIG. 3, a flow 300 of another embodiment of the method for detecting a pedestrian according to the present invention is shown. As shown in FIG. 3, the method of this embodiment includes the following steps.

In step 301, the image collected by the drive recorder attached to the vehicle is set as the target image.

The executing subject can acquire an image collected by a drive recorder attached to the vehicle and use the above image as a target image. The above image may include information on the traveling environment of the vehicle. It is understandable that the mounting position of the drive recorder is relatively low, the collected images can contain pedestrian bridge information, and there may be pedestrians on the pedestrian bridge. Since the pedestrian does not affect the running of the vehicle, it is not necessary to detect the pedestrian.

In step 302, a first cropped image is obtained by trimming the area of the preset ratio above the target image.

In this embodiment, the execution subject can obtain the first cropped image by trimming the region of the preset ratio above the target image. The preset ratio can be determined by the content contained in the image. Specifically, the execution subject can identify each object in the target image and crop the object included in the area other than the preset distance from the vehicle. For example, the executing subject can trim an area 50 meters ahead of the vehicle. The area is above the target image. In some specific applications, the preset ratio may be 1/4.

In step 303, the feature of the first trimmed image is extracted, and the feature map is obtained by trimming the first trimmed image in the extraction process.

In this embodiment, the execution subject obtains the first trimmed image, then extracts the first trimmed image, performs feature extraction, and trims the first trimmed image in the extraction process to obtain a feature map. obtain. Specifically, the execution subject can obtain an intermediate feature map after performing feature extraction on the first trimmed image by using the feature extraction algorithm. Then, the executing subject can trim the obtained intermediate feature map. Then, the execution subject obtains the intermediate feature map again by performing feature extraction again on the trimmed intermediate feature map using the feature extraction algorithm. The execution subject can obtain the feature map by trimming the obtained intermediate feature map again. It can be understood that the execution subject can obtain the feature map by extracting the features of the first trimmed image multiple times, and can also obtain the feature map by trimming the obtained feature map multiple times. .. This ensures that the effective features of the pedestrian are not trimmed.

In some selectable embodiments of this embodiment, the performer can obtain a feature map by the following steps not shown in FIG. 3, i.e., at least two convolutions with respect to the first cropped image. A feature map is obtained by performing an operation, performing at least one convolution operation, and then performing a trimming process at least once on the obtained feature map.

In this embodiment, the execution subject can perform feature extraction on the first cropped image using at least two convolution layers. The execution subject can perform at least two convolution operations on the first cropped image using the at least two convolution layers. It can be understood that an intermediate feature map is obtained each time the convolution operation is performed. After at least one convolution operation, the obtained intermediate feature map is subjected to at least one trimming process. Specifically, the execution subject can perform one trimming process on the obtained intermediate feature map after performing two convolution operations. For example, trim the upper 1/9 area of the intermediate feature map. Alternatively, the execution subject trims the upper 1/18 area of the intermediate feature map after performing one convolution operation. After that, the convolution operation is performed again on the trimmed intermediate feature map to obtain the intermediate feature map again. The intermediate feature map is trimmed again, that is, the area above 1/18 of the intermediate feature map is trimmed again.

Since the center of the pedestrian is generally located in the lower middle of the target image and the features of the pedestrian are extracted, the upper area of the obtained intermediate feature map can be trimmed. This ensures that the effective features of the pedestrian are not trimmed.

In step 304, the feature map is trimmed to obtain a second cropped image.

After obtaining the feature map, the execution subject can trim the obtained feature map. The feature map contains pedestrian center position information. Therefore, in this case, even if the feature map is trimmed again, the detection result of the pedestrian is not affected. Some areas above the feature map can be trimmed to reduce the amount of computation. In some specific applications, some of the above regions may be 1/4 regions.

In some selectable embodiments of this embodiment, the size of the second cropped image is greater than or equal to 1/2 the size of the target image.

In the present embodiment, the size of the second trimmed image is set to ½ or more of the size of the target image in order to ensure the accuracy of pedestrian detection while reducing the calculation amount of pedestrian detection.

In step 305, the detection result of the pedestrian is obtained by identifying the pedestrian in the second cropped image.

With reference to FIG. 4, FIG. 4 is a schematic diagram showing an application scene of the method for detecting a pedestrian according to the present embodiment. In the application scene of FIG. 4, a drive recorder 401 is attached to the self-driving car, and the drive recorder 401 can transmit the collected images to the in-vehicle computer 402 of the self-driving car. The in-vehicle computer 402 can obtain a pedestrian detection result by performing pedestrian detection according to the above embodiment.

In the method for detecting a pedestrian according to the above embodiment of the present invention, the image collected by the drive recorder can be trimmed a plurality of times, and features including valid pedestrian information can be extracted as much as possible. It is possible to ensure the accuracy of pedestrian detection and reduce the amount of calculation for pedestrian detection.

Further, with reference to FIG. 5, as an embodiment of the method shown in each of the above figures, the present invention provides an embodiment of an apparatus for detecting a pedestrian, and an embodiment of the apparatus is shown in FIG. Corresponding to the method embodiment shown, the apparatus can be specifically applied to various electronic devices.

As shown in FIG. 5, the apparatus 500 for detecting a pedestrian according to the present embodiment includes an image acquisition unit 501, a first trimming unit 502, a feature extraction unit 503, and a second trimming unit 504. , The pedestrian detection unit 505 and the like.

The image acquisition unit 501 is configured to acquire a target image.

The first trimming unit 502 is configured to obtain the first trimmed image by trimming the target image.

The feature extraction unit 503 is configured to obtain a feature map by extracting the features of the first trimmed image.

The second trimming unit 504 is configured to obtain a second trimmed image by trimming the feature map.

The pedestrian detection unit 505 obtains a pedestrian detection result by identifying the pedestrian in the second trimmed image.

In some selectable embodiments of this embodiment, the feature extraction unit 503 further extracts the features of the first cropped image and trims the first cropped image in the extraction process to obtain a feature map. Is configured to obtain.

In some selectable embodiments of this embodiment, the feature extraction unit 503 further performs at least two convolution operations on the first cropped image and is obtained after at least one convolution operation. The feature map is configured to be obtained by performing the trimming process at least once on the feature map.

In some selectable embodiments of this embodiment, the image acquisition unit 501 is further configured to target an image collected by a drive recorder mounted on the vehicle.

In some selectable embodiments of this embodiment, the first cropping unit 502 is further configured to obtain a first cropped image by trimming a region of preset ratio above the target image. To.

In some selectable embodiments of this embodiment, the size of the second cropped image is greater than or equal to 1/2 the size of the target image.

It should be understood that the units 501 to 505 described in the device 500 for detecting a pedestrian correspond to each step in the method described with reference to FIG. 2, respectively. Therefore, the above-mentioned operations and features described for the method for detecting a pedestrian are similarly applied to the device 500 and the units contained therein, and the description thereof is omitted here.

According to the embodiments of the present invention, the present invention further provides electronic devices and readable storage media.

As shown in FIG. 6, it is a block diagram of an electronic device by the method for detecting a pedestrian of the embodiment of this invention. Electronic devices are intended to represent various forms of digital computers such as laptop computers, desktop computers, workbench, personal digital assistants, servers, blade servers, large computers, and other suitable computers. Electronic devices can also represent various forms of mobile devices such as personal digital processing, mobile phones, smartphones, wearable devices, and other similar computing devices. The parts, their connections and relationships, and their functions set forth herein are merely examples and are intended to limit the realization of the invention as described and / or required herein. do not do.

As shown in FIG. 6, the electronic device includes at least one processor 601 and a memory 602, and an interface for connecting each component including a high-speed interface and a low-speed interface. The components may be connected to each other using different buses 603 and mounted on a common motherboard, or may be mounted in other ways as needed. The processor was executed in an electronic device, including in-memory or in-memory instructions for displaying graphical information in the GUI on an external input / output device (eg, a display device coupled to an interface). Can process commands. In other embodiments, a plurality of processors and / or a plurality of buses 603 can be used with the plurality of memories and the plurality of memories, if necessary. Similarly, multiple electronic devices may be connected and each device provides partially required operation (eg, as a server array, a set of blade servers, or a multiprocessor system). In FIG. 6, one processor 601 is taken as an example.

The memory 602 is a non-temporary computer-readable storage medium according to the present invention. Here, the memory 602 causes at least one processor to execute the method for detecting a pedestrian according to the present invention by storing a command that can be executed by at least one processor. The non-temporary computer-readable storage medium of the present invention stores computer commands, which are used to cause a computer to perform a method for detecting a pedestrian according to the present invention.

The memory 602, as a non-temporary computer-readable storage medium, is a program command / unit corresponding to the method for detecting a pedestrian in the embodiment of the present invention (for example, the image acquisition unit 501, 1st shown in FIG. 5). Used to store non-temporary software programs, non-temporary computer executable programs, and modules, such as the trimming unit 502, feature extraction unit 503, second trimming unit 504, and pedestrian detection unit 505). Can be done. Processor 601 performs various functional applications and data processing of the server by executing non-temporary software programs, instructions and modules stored in memory 602, i.e., the pedestrian in the embodiment of the method described above. A method for detection is realized.

The memory 602 can include a program storage area and a data storage area, where the program storage area can store the operating system, an application program required for at least one function, and the data storage area is a walk. It can store data created by the use of electronic devices that perform methods for detecting a person. The memory 602 can also include high speed random access memory and can also include non-temporary memory such as, for example, at least one magnetic disk storage device, flash memory device, or other non-temporary solid-state storage device. In some embodiments, as one of the options, memory 602 may include memory remotely located with respect to processor 601 and these remote memories detect pedestrians over the network. Can be connected to an electronic device that performs the method for. Examples of networks above include, but are not limited to, the Internet, corporate networks, local area networks, mobile communication networks and combinations thereof.

Electronic devices that perform the method for detecting pedestrians can further include input device 604 and output device 605. The processor 601 and the memory 602, the input device 604 and the output device 605 can be connected by the bus 603 or other methods, and in FIG. 6, the connection by the bus 603 is taken as an example.

The input device 604 can generate a key signal input relating to user settings and functional control of an electronic device that receives an input number or letter message and performs a method for detecting a pedestrian, such as a touch screen. Input devices such as keypads, mice, trackboards, touchpads, indicator bars, one or more mouse buttons, trackballs, joysticks, etc. The output device 605 can include a display device, an auxiliary lighting device (eg, LED), a tactile feedback device (eg, a vibration motor), and the like. The display device can include, but is not limited to, a liquid crystal display (LCD), a light emitting diode (LED) display, and a plasma display. In some embodiments, the display device may be a touch screen.

Various embodiments of the systems and techniques described herein are realized in digital electronic circuit systems, integrated circuit systems, dedicated ASICs (dedicated integrated circuits), computer hardware, firmware, software, and / or combinations thereof. be able to. These various embodiments may include the following content, i.e., implemented in one or more computer programs, the one or more computer programs running on a programmable system including at least one programmable processor. And / or can be interpreted, the programmable processor may be a dedicated or general purpose programmable processor, receiving data and commands from a storage system, at least one input device, and at least one output device, and. Data and commands can be transmitted to the storage system, the at least one input device, and the at least one output device.

These computational programs (also called programs, software, software applications, or codes) include machine instructions for programmable processors, utilize programming languages for advanced processes and / or objects, and / or assembly / machine languages. You can run these calculators. As used herein, the terms "machine readable medium" and "computer readable medium" are any computer program product, device, and / or device for providing machine instructions and / or data to a programmable processor. (For example, a magnetic disk, an optical disk, a memory, a programmable logic device (PLD)), and includes a machine-readable medium that receives a machine-readable signal, which is a machine-readable signal. The term "machine readable signal" refers to any signal for providing machine commands and / or data to a programmable processor.

In order to provide interaction with the user, the systems and techniques described herein can be implemented on a computer, which is a display device for displaying information to the user (eg, a CRT). Alternatively, it comprises an LCD (Liquid Crystal Display) monitor), a keyboard and a pointing device (eg, a mouse or trackball), and the user can provide input to the computer via the keyboard and the pointing device. Other types of devices can also be used to provide interactivity with the user. For example, the feedback provided to the user may be any form of sensing feedback (eg, visual feedback, auditory feedback, or tactile feedback), including any form (voice input, voice input, or tactile input). ) May receive input from the user.

The systems and techniques described herein can be a computing system that includes background components (eg, a data server), or a computing system that includes middleware components (eg, an application server), or a computing system that includes front-end components (eg,). A user computer having a graphical user interface or network browser may be mentioned, and the user may interact with embodiments of the systems and techniques described herein via the graphical user interface or network browser), or. It can be implemented on a computing system that includes any combination of such background components, middleware components, or front-end components. The components of the system may be interconnected by any form or media digital data communication (eg, a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

A computer system can include a client and a server. Clients and servers are generally separated from each other and usually operate interactively over a communication network. The client-server relationship is generated by a computer program that runs on the corresponding computer and has a client-server relationship with each other.

It should be understood that steps can be sorted, added, or deleted using the various forms of flow shown above. For example, each step described in the present invention may be performed in parallel, sequentially, or in a different order, as desired by the proposed technology disclosed in the present invention. As long as the results are achieved, no limitation is made herein.

The specific embodiment does not constitute a limitation on the scope of protection of the present invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and alternatives can be made, depending on design requirements and other factors. Any modifications, equal substitutions, improvements, etc. made within the spirit and principles of the invention should be included in the scope of protection of the invention.

Claims (15)

To get the target image and
By trimming the target image, a first trimmed image can be obtained.
By extracting the features of the first trimmed image, a feature map can be obtained.
By trimming the feature map, a second trimmed image can be obtained.
A method for detecting a pedestrian, which comprises obtaining a pedestrian detection result by identifying the pedestrian in the second cropped image. Obtaining a feature map by extracting the features of the first trimmed image is performed by extracting the features of the first trimmed image and trimming the first trimmed image in the extraction process. The method of claim 1, comprising obtaining the feature map. Obtaining the feature map by extracting the features of the first trimmed image and trimming the first trimmed image in the extraction process is possible.
Performing the convolution operation at least twice for the first cropped image,
The method according to claim 2, wherein the feature map is obtained by performing a trimming process at least once on the obtained feature map after at least one convolution operation. The method according to claim 1, wherein acquiring the target image includes using an image collected by a drive recorder attached to the vehicle as the target image. Obtaining the first trimmed image by trimming the target image includes obtaining the first trimmed image by trimming the region of the preset ratio above the target image. Item 1. The method according to Item 1. The method according to claim 1, wherein the size of the second cropped image is ½ or more of the size of the target image. An image acquisition unit configured to acquire the target image, and
A first trimming unit configured to obtain a first trimmed image by trimming the target image, and a first trimming unit.
A feature extraction unit configured to obtain a feature map by extracting features of the first trimmed image, and a feature extraction unit.
A second trimming unit configured to obtain a second trimmed image by trimming the feature map, and a second trimming unit.
A device for detecting a pedestrian, including a pedestrian detection unit configured to obtain a pedestrian detection result by identifying the pedestrian in the second cropped image. The feature extraction unit is further configured to obtain the feature map by extracting the feature of the first trimmed image and trimming the first trimmed image in the extraction process. The device described. The feature extraction unit is
The first cropped image is subjected to at least two convolution operations.
The apparatus according to claim 8, further configured to obtain the feature map by performing at least one trimming process on the obtained feature map after at least one convolution operation. The image acquisition unit further
The device according to claim 7, wherein an image collected by a drive recorder attached to a vehicle is used as a target image. The device according to claim 7, wherein the first trimming unit is further configured to obtain the first trimmed image by trimming a region of a preset ratio above the target image. The apparatus according to claim 7, wherein the size of the second cropped image is ½ or more of the size of the target image. With one or more processors
Including a storage device in which one or more programs are stored.
An electronic device that, when the one or more programs are executed by the one or more processors, realizes the method according to any one of claims 1 to 6 on the one or more processors. A computer-readable medium in which a computer program is stored, wherein the method according to any one of claims 1 to 6 is realized when the computer program is executed by a processor. A computer program that realizes the method according to any one of claims 1 to 6, when the computer program is executed by a processor.

Images