CN112837341B - Self-adaptive time-space domain pedestrian appearance restoration method - Google Patents

Self-adaptive time-space domain pedestrian appearance restoration method Download PDF

Info

Publication number
CN112837341B
CN112837341B CN202110106572.8A CN202110106572A CN112837341B CN 112837341 B CN112837341 B CN 112837341B CN 202110106572 A CN202110106572 A CN 202110106572A CN 112837341 B CN112837341 B CN 112837341B
Authority
CN
China
Prior art keywords
pedestrian
visual rhythm
time
rhythm
space
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN202110106572.8A
Other languages
Chinese (zh)
Other versions
CN112837341A (en
Inventor
张云佐
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Shijiazhuang Tiedao University
Original Assignee
Shijiazhuang Tiedao University
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Shijiazhuang Tiedao University filed Critical Shijiazhuang Tiedao University
Priority to CN202110106572.8A priority Critical patent/CN112837341B/en
Publication of CN112837341A publication Critical patent/CN112837341A/en
Application granted granted Critical
Publication of CN112837341B publication Critical patent/CN112837341B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/20—Analysis of motion
    • G06T7/207—Analysis of motion for motion estimation over a hierarchy of resolutions
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10016—Video; Image sequence

Abstract

The invention discloses a self-adaptive time-space domain pedestrian appearance restoration method, which comprises the following steps of: completing horizontal visual rhythm splicing by taking a single pixel row as a sampling unit; researching the relation between the pedestrian track slope and the pedestrian movement speed in the horizontal visual rhythm; and the pedestrian track slope is used as a vertical vision rhythm sampling width determination basis to finish self-adaptive time-space domain pedestrian appearance restoration. The experimental result shows that compared with the pedestrian appearance formed by the traditional visual rhythm, the pedestrian appearance restored by the method has better effects on contour fluency and visual perception, and the difference proportion of the pedestrian appearance restored by the method and the pedestrian image in the original spatial domain is smaller.

Description

Self-adaptive time-space domain pedestrian appearance restoration method
Technical Field
The invention relates to the technical field of image processing methods, in particular to a self-adaptive time-space domain pedestrian appearance restoration method.
Background
The number of surveillance videos is increasing at an incredible rate, and browsing and watching a large amount of surveillance videos becomes a time-consuming and labor-consuming task while the life safety of people is guaranteed. The video abstraction technology can generate a high-compression-rate video abstraction on the premise of keeping target activity information in an original video, can quickly obtain interested contents, and greatly reduces the time for a user to browse the video.
Saeid Bagheri proposes a method for mapping a surveillance video to a time profile for indexing, which effectively visualizes video content in a two-dimensional time image, extracts the time profile from the video to deliver accurate time information, and simultaneously retains certain spatial features for identification. The method of Saeid Bagheri enables the video abstraction to achieve good effects on both computational efficiency and visual quality. However, in many cases the temporal width of the target is compressed due to the limited frame rate on close objects with fast speed, resulting in a deformation of the contour map in the spatial-temporal domain, as shown in fig. 1.
Disclosure of Invention
The invention aims to provide a human appearance restoration method which has a better effect on the contour fluency and visual perception of the restored pedestrian appearance and has a smaller difference proportion with the original spatial domain pedestrian image.
In order to solve the technical problems, the technical scheme adopted by the invention is as follows: a self-adaptive time-space domain pedestrian appearance restoration method is characterized by comprising the following steps:
completing horizontal visual rhythm splicing by taking a single pixel row as a sampling unit;
researching the relation between the pedestrian track slope and the pedestrian movement speed in the horizontal visual rhythm;
and the pedestrian track slope is used as a vertical vision rhythm sampling width determination basis to finish self-adaptive time-space domain pedestrian appearance restoration.
The further technical scheme is as follows: the slope of the target track in the horizontal visual rhythm is defined as k; a (x)1,y1) And B (x)2,y2) Are two points on the trajectory, the expressions of k and v are:
Figure BDA0002917622870000021
the above formula shows the numerical relationship between the slope k of the space-time motion trajectory of the target and the motion velocity v of the target in the horizontal direction; as long as the slope k of the spatiotemporal motion trajectory of the pedestrian in the horizontal visual rhythm can be obtained, the motion velocity v of the pedestrian relative to the hardware device can be obtained.
The further technical scheme is as follows: the sampling width of the vertical visual rhythm in the single frame image is defined as delta x, and the relation between the delta x and the target movement speed v is as follows:
Δx＝|v·f|
wherein f represents a unit frame, and as the value of f is 1, the sampling width delta x of the vertical visual rhythm is only required to be ensured to be equal to the value of the pedestrian moving speed v to ensure that the pedestrian appearance in the space-time domain keeps higher resolution;
the relationship between the vertical visual rhythm sampling width Deltax and the slope k of the pedestrian space-time motion trajectory is as follows:
Figure BDA0002917622870000022
sampling region S of vertical visual rhythm in ith frame of videoiyExpressed as the following equation:
Figure BDA0002917622870000023
wherein y isi,l jRepresents the position (j, l), l ∈ [1, M ]]The pixel is determined as long as the sampling width value delta x of the vertical visual rhythm and the space-time motion of the target are ensuredThe inverse of the trajectory slope k is consistent, a complete and smooth contour of the target appearance image can be obtained in the 2D spatio-temporal image.
Adopt the produced beneficial effect of above-mentioned technical scheme to lie in: the method comprises the steps of firstly, completing horizontal visual rhythm splicing by taking a single pixel row as a sampling unit; secondly, researching the relation between the pedestrian track slope and the pedestrian movement speed in the horizontal visual rhythm; and finally, the pedestrian track slope is used as a vertical visual rhythm sampling width determination basis to finish self-adaptive time-space domain pedestrian appearance restoration. The experimental result shows that compared with the pedestrian appearance formed by the traditional visual rhythm, the pedestrian appearance restored by the method has better effects on contour fluency and visual perception, and the difference proportion of the pedestrian appearance restored by the method and the pedestrian image in the original spatial domain is smaller.
Drawings
The present invention will be described in further detail with reference to the accompanying drawings and specific embodiments.
FIG. 1 is a raw resolution of a time slice, where the time resolution of the object is too low to be identified;
FIG. 2 is an overall flow chart of the method according to the embodiment of the invention
FIG. 3 is a schematic diagram of horizontal visual cadence formation in an embodiment of the invention;
FIG. 4 is a composite schematic of vertical visual cadence sample widths according to an embodiment of the invention;
FIG. 5a is a pedestrian image reconstructed in the time-space domain using a conventional visual rhythm method;
FIG. 5b is a pedestrian image reconstructed in the time-space domain using the method proposed in the present application;
FIG. 5c is a reconstructed pedestrian image in the spatial domain in the starting video;
FIG. 6a is a pedestrian image reconstructed in the time-space domain using a conventional visual rhythm method;
FIG. 6b is a pedestrian image reconstructed in the time-space domain using the method described in the present application;
FIG. 7a is a pedestrian image reconstructed in the time-space domain using a conventional visual rhythm method;
FIG. 7b is a pedestrian image reconstructed in the time-space domain using the method described in the present application;
fig. 8 is a graph of experimental results for each test video.
Detailed Description
The technical solutions in the embodiments of the present invention are clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present invention, but the present invention may be practiced in other ways than those specifically described and will be readily apparent to those of ordinary skill in the art without departing from the spirit of the present invention, and therefore the present invention is not limited to the specific embodiments disclosed below.
Improved visual rhythm
The visual rhythm is also called space-time slice, which is a high-efficiency video space-time analysis method and has wide application in video processing. Video is defined as V ═ F1,F2,L,FtIn which FiAnd i is more than or equal to 1 and less than or equal to t represents a video frame with the size of h multiplied by w. Let T (F)i)＝SiIs a general formula FiOperation of the column vector mapped to n × 1. Defining the visual cadence as an nxt image:
VR(VΥ)＝[T(F1) T(F2) L T(Ft)]＝[S1 S2 L St] (1)
vertical visual rhythm S obtained from ith frame imageiyIs defined as:
Siy＝(yi,l j)T＝(yi,l 1,yi,l 2,yi,l 3,L,yi,l h)T (2)
wherein, yi,l jDenotes the position (j, l), l ∈ [1, w ∈]OfA pixel.
Conventional visual rhythms typically accomplish the reorganization of two-dimensional spatio-temporal images in a basic unit of a single pixel column (row). The single and simple splicing method enables a space target with continuous information to be distorted in a time-space domain, and the shape of the space target is different from that of the target which can be perceived by human vision. Two-dimensional space-time pedestrians exist in a form which is not easy to recognize, so that understanding of the video abstract is more difficult, and the showing form is also poor. The application provides a new visual rhythm reorganization method, which breaks through the limitation of the traditional single-column (row) pixel reorganization, takes the number of the row (row) pixels capable of forming the complete appearance of a space-time target as a basic unit for forming the visual rhythm, and completes the self-adaptive reorganization and splicing of the two-dimensional space-time target. Assuming m columns of pixels as a reorganization unit of the vertical visual rhythm, equation (2) can be expressed as:
Figure BDA0002917622870000041
equation (3) shows an improved representation of vertical visual cadence, providing the possibility of restoring pedestrian appearance in the time-space domain using visual cadence.
Studying pedestrian deformation influence factors:
experiments prove that the factors influencing the deformation of the pedestrian in the space-time domain comprise two aspects, namely the frame rate of a hardware device for shooting video and the walking speed of the pedestrian relative to the ground. Assuming that a pedestrian walks at a speed v, the frame rate of a hardware device that captures a video is f1，f2The sampling width of the vertical visual rhythm is w1，w2. If f is1＝μ×f2Then, then
Figure BDA0002917622870000051
Conversely, if the frame rate f of the hardware device remains unchanged, then the pedestrian is present at v respectively1And v2Is traveling at a speed of v and v1＝μ×v2At this time, the flow rate of the gas is increased,
w1＝μ×w2 (5)
the pedestrian walking speed displayed in the video refers to the horizontal moving speed v of the pedestrian relative to the hardware equipmentp(pixel/frame), the expression of which is shown in equation (6).
vp＝α·v+β·f (6)
Where v denotes a horizontal moving speed of the pedestrian with respect to the ground, f denotes a frame rate of the hardware device, and α, β denote weight coefficients.
Through the above analysis, the present application provides a self-adaptive time-space domain pedestrian appearance restoration method, as shown in fig. 2, the method includes the following steps:
completing horizontal visual rhythm splicing by taking a single pixel row as a sampling unit;
researching the relation between the pedestrian track slope and the pedestrian movement speed in the horizontal visual rhythm;
and the pedestrian track slope is used as a vertical vision rhythm sampling width determination basis to finish self-adaptive time-space domain pedestrian appearance restoration.
The method makes the video image mapped on the two-dimensional time section more vivid, and makes the pedestrian on the two-dimensional time section exist in the visual field in an easier-to-recognize mode. The method breaks through the limitation of the traditional visual rhythm, and utilizes the pedestrian track information in the horizontal visual rhythm to assist in completing the splicing of the vertical visual rhythm, thereby completing the self-adaptive restoration of the pedestrian image in the space-time domain and enabling the pedestrian appearances in the space-time domain and the space domain to be infinitely close.
Splicing horizontal visual rhythm:
the space-time motion track reflects the state change of all moving objects, and the horizontal visual rhythm is an effective tool for exploring the space-time motion track. Assume that the object moves straight from left to right at a constant speed in the horizontal direction at a speed of v pixels/frame (p/f). The single pixel line area where the target center position is located is sampled through experiments, one side of the spliced horizontal visual rhythm is the same as the horizontal length of an original video frame, and the other side of the spliced horizontal visual rhythm is
Figure BDA0002917622870000061
A pixel. The formation process of the horizontal visual rhythm and the target motion trail is shown in fig. 3.
Explore the relationship of k to v:
the slope of the target trajectory in the horizontal visual rhythm is defined as k. A (x)1,y1) And B (x)2,y2) Are two points on the trajectory, and the expressions of k and v are shown in formula (7).
Figure BDA0002917622870000062
Equation (7) shows the numerical relationship between the slope k of the target spatiotemporal motion trajectory and its motion velocity v. As long as the slope k of the spatiotemporal motion trajectory of the pedestrian in the horizontal visual rhythm can be obtained, the motion velocity v of the pedestrian relative to the hardware device can be obtained.
Adaptive time-space domain pedestrian stitching
The width of vertical visual rhythm sampling in a single frame image directly affects the appearance of the target in the spatio-temporal domain. If the sampling width is too narrow, the target shape is compressed and is difficult to recognize; however, if the sampling width is too wide, severe splicing traces can be generated, and the target contour is not smooth. The sampling width of the vertical visual rhythm in the single frame image is defined as Δ x, and the relationship between Δ x and the target movement velocity v can be obtained from fig. 4, as shown in equation (8).
Δx＝|v·f| (8)
Where f denotes a unit frame. Since the value of f is 1, to maintain a higher resolution of the appearance of the pedestrian in the spatiotemporal domain, it is only necessary to ensure that the sampling width Δ x of the vertical visual rhythm is equal to the value of the pedestrian moving speed v.
The relationship between the vertical visual rhythm sampling width Deltax and the slope k of the pedestrian space-time motion trajectory is as follows:
Figure BDA0002917622870000071
sampling region S of vertical visual rhythm in ith frame of videoiyExpressed as the following equation:
Figure BDA0002917622870000072
wherein y isi,l jRepresents the position (j, l), l ∈ [1, M ]]And the pixel can obtain a complete and smooth contour of the target appearance image in the 2D space-time image as long as the sampling width value delta x of the vertical visual rhythm is ensured to be consistent with the reciprocal of the target space-time motion track slope k.
Experimental verification
All experiments were performed on Windows10 system, Intel (R) core (TM) i7-4790cpu octal processor, AMD radeonr7200 series graphics card, 16GB memory. The video data involved in the experiment are all self-collected monitoring video data, and 10 sections of videos and 6 different scenes are tested. The method and the device use the reciprocal of the slope of the empty motion track of the pedestrian descending at the horizontal visual rhythm as the basic unit of vertical visual rhythm sampling to reconstruct a complete pedestrian appearance image in the space-time domain. FIGS. 5a-5c, 6a-6b, and 7a-7b show the results for single and multiple targets, respectively, in the spatio-temporal domain. As can be seen from fig. 5a-5c, fig. 6a-6b and fig. 7a-7b, although the pedestrian in the time-space domain recovered by the method has a slightly different appearance from that of the pedestrian in the original video in the space domain, the method has a great improvement compared with the traditional visual rhythm reorganization image.
The performance of the method provided by the application and the performance of the traditional visual rhythm method are compared by taking the difference proportion as an evaluation index. The value range of the difference ratio is [0,1], and the smaller the difference ratio is, the better the difference ratio is. The formula of the difference ratio is shown in formula (11). The experimental results show that the average difference ratio of the traditional visual rhythm is 0.8401, and the average difference ratio of the method is 0.0857. Fig. 8 shows a comparison of the difference ratios on each test video.
Figure BDA0002917622870000073
Fig. 8 shows that the method provided by the present application has a smaller difference ratio compared with the conventional visual rhythm method, that is, a better pedestrian recovery effect is achieved in the time-space domain.

Claims (1)

1. A self-adaptive time-space domain pedestrian appearance restoration method is characterized by comprising the following steps:
completing horizontal visual rhythm splicing by taking a single pixel row as a sampling unit;
researching the relation between the pedestrian track slope and the pedestrian movement speed in the horizontal visual rhythm;
the pedestrian track slope is used as a vertical vision rhythm sampling width determination basis to finish self-adaptive time-space domain pedestrian appearance restoration;
the sampling width of the vertical visual rhythm in the single frame image is defined as delta x, and the relation between the delta x and the target movement speed v is as follows:
Δx＝|v·f| (1)
wherein f represents a unit frame, and as the value of f is 1, the sampling width delta x of the vertical visual rhythm is only required to be ensured to be equal to the value of the pedestrian moving speed v to ensure that the pedestrian appearance in the space-time domain keeps higher resolution;
the relationship between the vertical visual rhythm sampling width Deltax and the slope k of the pedestrian space-time motion trajectory is as follows:
Figure FDA0003573639740000011
sampling region S of vertical visual rhythm in ith frame of videoiyExpressed as the following equation:
Figure FDA0003573639740000012
wherein y isi,l jRepresents the position (j, l), l ∈ [1, M ]]Pixel of (b), as long as vertical visual rhythm is guaranteedThe sample width value delta x is consistent with the reciprocal of the slope k of the target space-time motion track, so that a complete and smooth contour of a target appearance image can be obtained in a 2D space-time image;
assuming that a pedestrian walks at a speed v, the frame rate of a hardware device that captures a video is f1，f2The sampling width of the vertical visual rhythm is w1，w2If f is1＝μ×f2Then, then
Figure FDA0003573639740000013
Conversely, if the frame rate f of the hardware device remains unchanged, then the pedestrian is present at v respectively1And v2Is traveling at a speed of v and v1＝μ×v2At this time, the process of the present invention,
w1＝μ×w2 (5)
the pedestrian walking speed displayed in the video refers to the horizontal moving speed v of the pedestrian relative to the hardware equipmentp(pixel/frame), the expression of which is shown in equation (6).
vp＝α·v+β·f (6)
Wherein v represents the horizontal moving speed of the pedestrian relative to the ground, f represents the frame rate of the hardware equipment, and alpha and beta represent weight coefficients;
the slope of the target track in the horizontal visual rhythm is defined as k; a (x)1,y1) And B (x)2,y2) Are two points on the trajectory, and the expressions of k and v are:
Figure FDA0003573639740000021
the above formula shows the numerical relationship between the slope k of the space-time motion trajectory of the target and the motion velocity v of the target in the horizontal direction; as long as the slope k of the spatiotemporal motion trajectory of the pedestrian in the horizontal visual rhythm can be obtained, the motion velocity v of the pedestrian relative to the hardware device can be obtained.
CN202110106572.8A 2021-01-26 2021-01-26 Self-adaptive time-space domain pedestrian appearance restoration method Active CN112837341B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202110106572.8A CN112837341B (en) 2021-01-26 2021-01-26 Self-adaptive time-space domain pedestrian appearance restoration method

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202110106572.8A CN112837341B (en) 2021-01-26 2021-01-26 Self-adaptive time-space domain pedestrian appearance restoration method

Publications (2)

Publication Number Publication Date
CN112837341A CN112837341A (en) 2021-05-25
CN112837341B true CN112837341B (en) 2022-05-03

Family

ID=75931717

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202110106572.8A Active CN112837341B (en) 2021-01-26 2021-01-26 Self-adaptive time-space domain pedestrian appearance restoration method

Country Status (1)

Country Link
CN (1) CN112837341B (en)

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8489176B1 (en) * 2000-08-21 2013-07-16 Spectrum Dynamics Llc Radioactive emission detector equipped with a position tracking system and utilization thereof with medical systems and in medical procedures
CN106023076A (en) * 2016-05-11 2016-10-12 北京交通大学 Splicing method for panoramic graph and method for detecting defect state of guard railing of high-speed railway
CN106101487A (en) * 2016-07-04 2016-11-09 石家庄铁道大学 Video spatiotemporal motion track extraction method
CN106127813A (en) * 2016-07-04 2016-11-16 石家庄铁道大学 The monitor video motion segments dividing method of view-based access control model energy sensing
CN110020985A (en) * 2019-04-12 2019-07-16 广西师范大学 A kind of video-splicing system and method for Binocular robot
CN110300977A (en) * 2017-02-17 2019-10-01 考吉森公司 Method for image procossing and video compress

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8489176B1 (en) * 2000-08-21 2013-07-16 Spectrum Dynamics Llc Radioactive emission detector equipped with a position tracking system and utilization thereof with medical systems and in medical procedures
CN106023076A (en) * 2016-05-11 2016-10-12 北京交通大学 Splicing method for panoramic graph and method for detecting defect state of guard railing of high-speed railway
CN106101487A (en) * 2016-07-04 2016-11-09 石家庄铁道大学 Video spatiotemporal motion track extraction method
CN106127813A (en) * 2016-07-04 2016-11-16 石家庄铁道大学 The monitor video motion segments dividing method of view-based access control model energy sensing
CN110300977A (en) * 2017-02-17 2019-10-01 考吉森公司 Method for image procossing and video compress
CN110020985A (en) * 2019-04-12 2019-07-16 广西师范大学 A kind of video-splicing system and method for Binocular robot

Also Published As

Publication number Publication date
CN112837341A (en) 2021-05-25

Similar Documents

Publication Publication Date Title
Lipton Local application of optic flow to analyse rigid versus non-rigid motion
US8073203B2 (en) Generating effects in a webcam application
CN108134937B (en) Compressed domain significance detection method based on HEVC
CN106530328B (en) A method of it is followed based on video image to moving object detection and smoothly
Zhu et al. Head pose estimation for driver monitoring
Berdnikov et al. Real-time depth map occlusion filling and scene background restoration for projected-pattern-based depth cameras
CN103369208A (en) Self-adaptive de-interlacing method and device
CN108289175B (en) Low-delay virtual reality display method and display system
CN112837341B (en) Self-adaptive time-space domain pedestrian appearance restoration method
CN108270945A (en) A kind of motion compensation denoising method and device
CN111353382A (en) Intelligent cutting video redirection method based on relative displacement constraint
CN1196542A (en) Block matching method by moving target window
Lee et al. Statistical and entropy based multi purpose human motion analysis
Lebreton et al. V-BMS360: A video extention to the BMS360 image saliency model
CN112364680A (en) Abnormal behavior detection method based on optical flow algorithm
Hara et al. Predicting appearance of vehicles from blind spots based on pedestrian behaviors at crossroads
US11205405B2 (en) Content arrangements on mirrored displays
CN112446245A (en) Efficient motion characterization method and device based on small displacement of motion boundary
Morais et al. A content-based viewport prediction model
Kim et al. Real-time human segmentation from RGB-D video sequence based on adaptive geodesic distance computation
Xiong et al. Extraction of hand gestures with adaptive skin color models and its applications to meeting analysis
Durucan et al. Moving object detection between multiple and color images
JP2591337B2 (en) Moving image generation method and motion vector calculation method
Verhack et al. Progressive modeling of steered mixture-of-experts for light field video approximation
Li et al. RGBD Scene Flow Estimation with Global Nonrigid and Local Rigid Assumption

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant