本說明書的實施例提出一種新的指定區域的人群流量狀況估算，從安置在指定區域上方的攝影機拍攝的即時視訊串流中提取當前有效框，檢測當前有效框中的行人、產生行人的單框屬性並識別與之前有效框中的行人是否相同，以確定經過指定區域的行人並統計經過指定區域的人群流量的個人屬性統計值。本說明書的實施例以較低的運算負荷，準確的估算出經過指定區域的人群流量狀況，對指定區域附近服務項目的設置和實施能夠起到重要的指導作用。
本說明書的實施例可以運行在任何具有計算和儲存能力的設備上，如手機、平板電腦、PC(Personal Computer，個人電腦)、筆記型電腦、伺服器等設備；還可以由運行在兩個或兩個以上設備的邏輯節點來實現本說明書實施例中的各項功能。
本說明書的實施例採用攝影機拍攝的視訊串流來進行人群流量狀況估算。攝影機安置在指定區域的上方，以斜向下的角度對指定區域內的人群進行拍攝，產生即時視訊串流，攝影機的拍攝範圍能夠完全覆蓋指定區域。以圖1中的廣告螢幕為例，指定區域(實線框內部)位於該廣告螢幕的前方，攝影機可以安置在廣告螢幕的上方，以傾斜向下的角度持續拍攝整個指定區域。
本說明書的實施例中，指定區域的人群流量狀況估算方法的流程如圖2所示。
步驟210，從拍攝指定區域的即時視訊串流中提取當前有效框。
安裝在指定區域上方的攝影機將持續輸出以斜向下視角拍攝的視訊串流，視訊串流由連續的一框框圖像構成。可以基於一定的條件，持續不斷的從視訊串流中將符合該條件的各框圖像提取出來作為有效框，藉由連續的辨識有效框中的行人來進行人群流量狀況的估計。將最後一個從視訊串流中提取的有效框作為當前有效框。
提取有效框的條件可以根據實際應用場景對統計時間精度的要求、運行本實施例的設備的處理能力等因素來設置，例如，可以將與上一個有效框間隔K(K為自然數)框的一框圖像作為下一個有效框，也可以從每M(M為大於1的自然數)個連續框中提取一框作為有效框。
步驟220，檢測當前有效框中的每個行人，產生每個行人的特徵資訊，根據特徵資訊辨識每個行人的N(N為自然數)個單框屬性。
在提取當前有效框後，藉由深度學習的目標檢測演算法判斷當前有效框中是否存在人體，如果存在則定位每個行人的位置、以及該行人所佔據的部分圖像區域。
本說明書實施例中對採用的目標檢測演算法不做限定，如可以採用Faster R-CNN(Faster Regions with Convolutional Neural Network features，採用卷積神經網路特徵的快速目標區域識別)、SSD(Single Shot MultiBox Detector，單次目標多框預測)等。
在既要求較低計算量，又要求檢測準確率的應用場景中，可以採用YOLO(You Only Live Once)目標檢測演算法，從當前有效框中提取每個行人的圖像範圍和位置資訊，往往可以達到更好的效果。
對目標檢測演算法輸出的每個行人的位置、以及該行人所佔據的部分圖像區域，可以由每個行人的位置產生該行人的位置特徵(如該行人佔據的部分區域在圖片坐標系中的坐標)，由該行人所佔據的部分區域的圖像產生該行人的外觀特徵(如衣服顏色、衣服紋理、手提包、背包、帽子等)，將外觀特徵、或者將位置特徵和外觀特徵作為該行人的特徵資訊。
本說明書實施例中，屬性用來描述行人的某個特定方面，例如，屬性可以是性別、年齡段、身體朝向、衣著等等中的一個到多個。可以根據實際應用場景的需求、攝影機分辨率等因素，來確定將哪些屬性作為辨識和統計的對象。根據某一個有效框中某個行人的圖像所辨識出的該行人的屬性，稱為該行人的單框屬性。
依據當前有效框中每個行人的特徵資訊，可以辨識出每個行人的N個單框屬性的具體值，例如性別屬性(是男是女)、年齡段屬性(如處於20歲以下、20到30歲、30到40歲、40歲以上的哪個年齡段)、身體朝向屬性(如正面、側面還是背面朝向攝影機)。
可以採用機器學習分類模型來進行各個單框屬性的辨識。具體而言，以行人的特徵資訊為輸入，以屬性各個具體值的概率為輸出構建機器學習分類模型，採用有標記的樣本進行訓練後，即可使用該分類模型得出對行人的各個單框屬性的估計值。本說明書實施例對機器學習分類模型的種類不做限定，例如可以是二分類模型、多分類模型等；對分類模型所採取的機器學習演算法也不做限定，例如可以是線性回歸、決策樹、隨機森林等。
步驟230，採用行人重識別演算法，根據特徵資訊識別當前有效框中與之前的至少一個有效框中相同的行人。
當某個行人從指定區域經過時，會被拍攝到多個有效框中。在進行人群流量狀況估算時，需要在各個有效框中找出相同的行人，避免對同一個行人多次計數，才能得到準確的資料。
行人重識別(Person ReID，Person Re-identification)能夠利用電腦視覺技術來判斷圖像中是否存在特定行人，可以用來進行同一個攝影機或跨攝影機的人物追蹤。本說明書的實施例中，採用行人重識別演算法來判斷在當前有效框中檢測到的所有行人中，哪些是已經出現在之前有效框中的行人，哪些是在當前有效框中新出現的行人。
可以藉由查找當前有效框中的某個行人是否出現在當前有效框之前的K(K為自然數)個有效框裡，來判斷該行人是否是新出現的行人。由於本說明書實施例中攝影機以傾斜向下的角度拍攝指定區域，行人較為密集時，可能會出現某個行人在某個有效框或某幾個連續的有效框中被他人遮擋而沒有被檢測到的情形，選取較大的K值可以避免在這種情況下錯誤的將該行人重複計數，但較大的K值也會帶來更大的運算負荷。實際應用場景中，可以根據指定區域的行人密集程度、相鄰有效框的間隔時間、運行本實施例的設備的處理能力等因素，來選擇適當的K值。
在一種實現方式中，可以根據行人的外觀特徵和位置特徵，判定當前有效框中的每個行人是否是之前的K個有效框中已存在的行人，如果不是則產生新的人物標識標記該行人，如果是則以已有的人物標識標記該行人。
具體而言，採用當前有效框中某個行人的位置特徵和外觀特徵，在之前的K個有效框中查找是否已經存在該行人，如果不存在，為該行人產生新的人物標識並用產生的人物標識標記該行人，人物標識用來唯一的代表一個行人，可以是索引號、字串等，不做限定。如果已經存在該行人，則該行人已經具有自己的人物標識，沿用之前已有的人物標識標記當前有效框中的該行人即可。對當前有效框中檢測出的所有行人逐個執行上述查找過程，直到檢測出的每個行人都標記有人物標識。
可以根據實際應用場景的需要，來選擇識別不同有效框中的行人是否是同一個人時採用的演算法，不做限定。例如，可以採用匈牙利演算法，來根據行人的外觀特徵和位置特徵進行當前有效框與之前有效框中的行人匹配。
步驟240，基於行人重識別演算法的結果確定經過指定區域的行人，由經過指定區域的行人的單框屬性統計得到經過指定區域的人群流量的N個個人屬性的統計值。
可以根據實際應用的需要，來確定將哪些行人作為經過指定區域的行人，不做限定。以下舉例說明：
第一個例子：可以將在各個有效框中進入指定區域的行人作為經過指定區域的行人。如果在某個有效框中某個行人出現在指定區域內、並且在該有效框之前的K個有效框中未出現在指定區域內，則認為該行人在該有效框進入指定區域。
第二個例子：可以將在各個有效框中離開指定區域的行人作為經過指定區域的行人。如果某個行人在某個有效框之前的K個有效框中曾經出現在指定區域內、並且在該有效框以及該有效框之後的1個到K個有效框中未出現在指定區域內，則認為該行人在該有效框離開指定區域。
在確定經過指定區域的行人後，經過指定區域的人群流量即是這些行人的集合。由這些行人在各個有效框中的N個單框屬性，可以統計出經過指定區域的人群流量的N個個人屬性的統計值。其中，個人屬性的統計值是對某個行人集合中各個行人的個人屬性具體值的統計結果，例如性別屬性(男性總數和女性人數)、年齡段屬性(如20歲以下的人數、20到30歲的人數、30到40歲的人數、40歲以上的人數)、身體朝向屬性(如正面朝向攝影機的人數、側面朝向攝影機的人數、背面朝向攝影機的人數)。
在由經過指定區域的行人的單框屬性產生個人屬性的統計值時，可以根據同一個行人在若干個有效框中的N個單框屬性確定該行人的N個個人屬性，由統計時間段內經過指定區域的所有行人的N個個人屬性，統計得到該統計時間段經過指定區域的人群流量的N個個人屬性的統計值。其中，統計時間段是對個人屬性的統計值進行累計的時間段。
在行人經過指定區域時，通常會出現在多個有效框中，以這些有效框中該行人的圖像為依據所估計的該行人的屬性，稱為該行人的個人屬性。在該行人出現的每個有效框中，該行人都會有N個單框屬性的具體值，可以由這些各個單框屬性的具體值，產生對應的個人屬性的具體值。
舉例說明，在第一種實現方式中，可以將某個單框屬性在該行人出現的所有有效框中具有最大可能性的具體值，作為該行人對應的個人屬性的具體值；在第二種實現方式中，可以將該行人出現的所有有效框中某個單框屬性具體值的可能性進行累加，將累加結果最大的一個具體值作為該行人對應的個人屬性的具體值。如，某行人在3個有效框中出現，其單框屬性中性別屬性的具體值分別是：第一個有效框中男性0.33、女性0.67，第二個有效框中男性0.52、女性0.48，第二個有效框中男性0.42、女性0.58，則在第一種實現方式中，具有最大可能性的具體值為女性0.67，則該行人個人屬性中性別屬性為女性；在第二種實現方式中，男性的可能性總和為1.27，女性的可能性總和為1.73，則該行人個人屬性中性別屬性為女性。
對統計時間段內經過指定區域的所有行人的N個個人屬性分別進行累計，即可得到統計時間段經過指定區域的人群流量的N個個人屬性的統計值。
在以人物標識對每個行人進行標記的實現方式中，可以將每個行人進入指定區域的有效框、與該行人離開指定區域的有效框之間的時間長度，作為該行人在指定區域的停留時間長度。在統計經過指定區域的人群流量的個人屬性的統計值時，還可以統計這些行人的停留時間長度，如停留時間長度總和或平均停留時間長度。
需要說明的是，在統計經過指定區域的人群流量的個人屬性統計值時，累加每個個人屬性每個具體值的統計值，即可得到經過指定區域的人群流量數量，因而可以不必單獨對人群流量做累計。
可見，本說明書的實施例中，從安置在指定區域上方的攝影機拍攝的即時視訊串流中提取當前有效框，藉由檢測當前有效框中的行人，產生每個行人的特徵資訊和單框屬性，採用行人重識別演算法識別出當前有效框中的行人與之前有效框中相同的行人，確定經過指定區域的行人並統計經過指定區域的人群流量的個人屬性的統計值，能夠以較低的運算代價，準確的估算出經過指定區域的人群流量狀況，從而對指定區域附近服務項目的設置和實施起到指導作用。
由於運算負荷較低，本說明書實施例的方法適於運行在嵌入式開發板上，並且對嵌入式開發板的硬體環境沒有特別要求。運行本說明書實施例的嵌入式開發板可以安裝在攝像機附近，將以一定週期統計出的人群流量狀況資料藉由自身的通信模組發送給負責採集人群流量狀況資料的伺服器，而無需上傳攝像機拍攝的影片或者圖像，能夠在不侵犯行人的隱私的條件下得到精確的統計資料。
上述對本說明書特定實施例進行了描述。其它實施例在所附申請專利範圍的範圍內。在一些情況下，在申請專利範圍中記載的動作或步驟可以按照不同於實施例中的順序來執行並且仍然可以實現期望的結果。另外，在圖式中描繪的過程不一定要求示出的特定順序或者連續順序才能實現期望的結果。在某些實施方式中，多任務處理和並行處理也是可以的或者可能是有利的。
在本說明書的一個應用示例中，需要對一塊廣告螢幕所在的廣告點位品質進行評估。將該廣告點位前行人可以停留或通過、並且能夠瞭解該廣告螢幕上播放內容的的區域作為指定區域，在廣告螢幕的頂部中間位置安裝RGB(Red Green Blue，紅綠藍)攝影機，以斜向下的視角對指定區域進行拍攝，指定區域位於拍攝範圍的中央部分，與拍攝範圍的邊界相隔一定距離。
人群流量狀況統計由運行在NVIDIA(英偉達)嵌入式開發板上的程式進行，嵌入式開發板安裝在攝影機附近，其上包括通信單元，可以與攝影機藉由近距離無線方式連接，從攝影機獲取其拍攝的影片資料。嵌入式開發板還可以藉由通信單元將估算得出的人群流量狀況資料上傳給預定的伺服器。
嵌入式開發板上預先保存有採用有標記樣本訓練完畢的機器學習分類模型，該分類模型以行人在一框圖像中的外觀特徵作為輸入，以性別、年齡段、身體朝向、衣著這4個屬性的具體值的概率為輸出。
嵌入式開發板上運行的人群流量狀況統計軟體的結構如圖3所示。
RGB攝影機以25框/秒的速度持續拍攝開放場景的圖像，形成視訊串流。嵌入式開發板從拍攝的視訊串流中每隔固定數量的框提取一框RGB圖像，作為當前有效框。
人群流量狀況統計軟體採用YOLO目標檢測演算法，識別出當前有效框中的每個行人，確定每個行人在圖像坐標系中的位置坐標(一種位置特徵)，以及每個行人所佔據的部分區域的圖像。
針對當前有效框中的每個行人，從該行人佔據區域的圖像中提取該行人的外觀特徵。將外觀特徵輸入到機器學習分類模型，可以得到該行人的4個單框屬性的每個具體值的概率。
分別以當前有效框中的每個行人的外觀特徵和位置坐標為依據，行人重識別演算法採用匈牙利演算法判斷該行人是否與當前有效框之前的3個有效框中的各個行人是否匹配，鑒別出該行人是否在之前的3個有效框中出現過，如果未曾出現過，則為該行人產生新的人物標識來唯一代表該行人，並用新的人物標識標記該行人；如果曾經出現過，則採用該行人已有的人物標識來標記該行人。
基於行人重識別演算法給當前有效框中每個行人標記的人物標識，以及YOLO目標檢測演算法輸出的每個行人的位置坐標，判斷在各個有效框中進入和離開指定區域的行人。如果在某個有效框中某個行人出現在指定區域內、並且在該有效框之前的3個有效框中未出現在指定區域內，則認為該行人在該有效框進入指定區域；如果某個行人在某個有效框之前的3個有效框中曾經出現在指定區域內、並且在該有效框以及該有效框之後的2個有效框中未出現在指定區域內，則認為該行人在該有效框離開指定區域。
以預定的統計時間段為週期，對在該統計時間段的所有有效框離開指定區域的每個行人，累加該行人出現的各個有效框的各個單框屬性的各個具體值，將某個屬性的累加和最高的具體值作為該行人對應的個人屬性的具體值；將該行人離開指定區域的有效框和該行人進入指定區域的有效框之間的時間間隔作為該行人在指定區域的停留時間長度。
分屬性分具體值累計在一個週期內所有有效框中離開指定區域的行人的數量，將各個個人屬性的各個具體值累計結果作為該統計時間段內的人群流量狀況統計結果；累加在一個週期內所有有效框中離開指定區域的行人的停留時間長度，計算這些行人的平均停留時間長度。
在一個週期結束後，嵌入式開發板向預定的伺服器發送該週期的人群流量狀況統計結果和人群流量的平均停留時間長度。
伺服器在收到嵌入式開發板上傳的資訊後，可以根據經過指定區域的人群流量的個人屬性統計值來對廣告點位的品質進行評估。伺服器可以獲得點位附近在各個統計時間段的人群流量數量、人群流量在廣告點位前停留的時間、人群流量是正向、側向還是背向螢幕等的詳細資訊，不僅可以向廣告主提出準確的廣告投放建議，同時也能為廣告運營商提供精細的報價依據。
與上述流程實現對應，本說明書的實施例還提供了一種指定區域的人群流量狀況估算裝置。該裝置可以藉由軟體實現，也可以藉由硬體或者軟硬體結合的方式實現。以軟體實現為例，作為邏輯意義上的裝置，是藉由所在設備的CPU(Central Process Unit，中央處理器)將對應的電腦程式指令讀取到內部儲存器中運行形成的。從硬體層面而言，除了圖4所示的CPU、內部儲存器以及儲存器之外，指定區域的人群流量狀況估算裝置所在的設備通常還包括用於進行無線信號收發的晶片等其他硬體，和/或用於實現網路通信功能的板卡等其他硬體。
圖5所示為本說明書實施例提供的一種指定區域的人群流量狀況估算裝置，包括有效框提取單元、行人檢測及特徵單元、行人重識別單元和屬性統計單元，其中：有效框提取單元用於從拍攝所述指定區域的即時視訊串流中提取當前有效框；所述視訊串流由安置在指定區域上方的攝影機拍攝，其拍攝範圍覆蓋整個指定區域；行人檢測及特徵單元用於檢測當前有效框中的每個行人，產生每個行人的特徵資訊，根據特徵資訊辨識每個行人的N個單框屬性；N為自然數；行人重識別單元用於採用行人重識別演算法，根據特徵資訊識別當前有效框中與之前的至少一個有效框中相同的行人；屬性統計單元用於基於行人重識別演算法的結果確定經過指定區域的行人，由經過指定區域的行人的單框屬性統計得到經過指定區域的人群流量的N個個人屬性的統計值。
可選的，所述行人檢測及特徵單元根據特徵資訊辨識每個行人的單框屬性，包括：採用機器學習分類模型辨識每個行人的若干個單框屬性；所述機器學習分類模型以行人的特徵資訊為輸入，採用有標記的樣本進行訓練。
可選的，所述屬性統計單元由經過指定區域的行人的單框屬性統計得到經過指定區域的人群流量的N個個人屬性的統計值，包括：根據同一個行人在若干個有效框中的N個單框屬性確定所述行人的N個個人屬性，由統計時間段內經過指定區域的所有行人的N個個人屬性，統計得到所述統計時間段經過指定區域的人群流量的N個個人屬性的統計值。
可選的，所述單框屬性和個人屬性分別包括以下一項到多項：性別、年齡段、身體朝向、衣著。
一個例子中，所述特徵資訊包括：外觀特徵和位置特徵；所述行人重識別單元具體用於：根據行人的外觀特徵和位置特徵，判定當前有效框中的每個行人是否是之前的至少一個有效框中已存在的行人，如果不是則產生新的人物標識標記所述行人，如果是則以已有的人物標識標記所述行人。
上述例子中，所述行人重識別單元根據行人的外觀特徵和位置特徵，判定當前有效框中的某個行人是否是之前有效框中已存在的行人，包括：採用匈牙利演算法，根據行人的外觀特徵和位置特徵進行當前有效框與之前有效框中的行人匹配。
一種實現方式中，所述經過指定區域的行人包括：進入指定區域的行人、或離開指定區域的行人；所述進入指定區域的行人包括：在某個有效框中出現在指定區域內、並且在所述有效框之前的至少一個有效框中未出現在指定區域內的行人；所述離開指定區域的行人包括：在某個有效框之前的相鄰有效框中出現在指定區域內、並且在所述有效框及其後的至少一個有效框中未出現在指定區域內的行人。
上述實現方式中，所述裝置還包括：停留時間長度統計單元，用於統計經過指定區域的人群流量在指定區域的停留時間長度，其中經過指定區域的每個行人的停留時間長度由所述行人進入指定區域的有效框與離開指定區域的有效框之間的時間長度確定。
可選的，所述裝置運行在嵌入式開發板上。
可選的，所述指定區域包括：廣告點位前的指定區域；所述裝置還包括：點位評估單元，用於根據經過指定區域的人群流量的個人屬性統計值屬性對廣告點位的品質進行評估。
本說明書的實施例提供了一種電腦設備，該電腦設備包括儲存器和處理器。其中，儲存器上儲存有能夠由處理器運行的電腦程式；處理器在運行儲存的電腦程式時，執行本說明書實施例中指定區域的人群流量狀況估算方法的各個步驟。對指定區域的人群流量狀況估算方法的各個步驟的詳細描述請參見之前的內容，不再重複。
本說明書的實施例提供了一種電腦可讀儲存媒體，該儲存媒體上儲存有電腦程式，這些電腦程式在被處理器運行時，執行本說明書實施例中指定區域的人群流量狀況估算方法的各個步驟。對指定區域的人群流量狀況估算方法的各個步驟的詳細描述請參見之前的內容，不再重複。
以上所述僅為本說明書的較佳實施例而已，並不用以限制本申請，凡在本申請的精神和原則之內，所做的任何修改、等同替換、改進等，均應包含在本申請保護的範圍之內。
在一個典型的配置中，計算設備包括一個或多個處理器(CPU)、輸入/輸出介面、網路介面和內部儲存器。
內部儲存器可能包括電腦可讀媒體中的非永久性儲存器，隨機存取記憶體(RAM)和/或非易失性內部儲存器等形式，如唯讀記憶體(ROM)或快閃記憶體(flash RAM)。內部儲存器是電腦可讀媒體的示例。
電腦可讀媒體包括永久性和非永久性、可移動和非可移動媒體可以由任何方法或技術來實現資訊儲存。資訊可以是電腦可讀指令、資料結構、程式的模組或其他資料。電腦的儲存媒體的例子包括，但不限於相變隨機存取記憶體(PRAM)、靜態隨機存取記憶體(SRAM)、動態隨機存取記憶體(DRAM)、其他類型的隨機存取記憶體(RAM)、唯讀記憶體(ROM)、電可抹除可程式化唯讀記憶體 (EEPROM)、快閃記憶體或其他內部儲存器技術、唯讀光碟(CD-ROM)、數位化多功能光碟(DVD)或其他光學儲存、磁盒式磁帶，磁帶磁磁碟儲存或其他磁性儲存設備或任何其他非傳輸媒體，可用於儲存可以被計算設備存取的資訊。按照本文中的界定，電腦可讀媒體不包括暫存電腦可讀媒體(transitory media)，如調變的資料信號和載波。
還需要說明的是，術語“包括”、“包含”或者其任何其他變體意在涵蓋非排他性的包含，從而使得包括一系列要素的過程、方法、商品或者設備不僅包括那些要素，而且還包括沒有明確列出的其他要素，或者是還包括為這種過程、方法、商品或者設備所固有的要素。在沒有更多限制的情況下，由語句“包括一個……”限定的要素，並不排除在包括所述要素的過程、方法、商品或者設備中還存在另外的相同要素。
本領域技術人員應明白，本說明書的實施例可提供為方法、系統或電腦程式產品。因此，本說明書的實施例可採用完全硬體實施例、完全軟體實施例或結合軟體和硬體方面的實施例的形式。而且，本說明書的實施例可採用在一個或多個其中包含有電腦可用程式碼的電腦可用儲存媒體(包括但不限於磁碟儲存器、CD-ROM、光學儲存器等)上實施的電腦程式產品的形式。The embodiment of the present specification proposes a new method for estimating the flow of people in a designated area, extracting the current valid frame from the real-time video stream captured by a camera placed above the designated area, detecting pedestrians in the current effective frame, and generating a single frame for pedestrians Attribute and identify whether it is the same as the pedestrian in the previous valid box, to determine the pedestrian passing through the specified area and to count the personal attribute statistics of the crowd flow through the specified area. The embodiments of the present specification accurately estimate the flow of people passing through the designated area with a low calculation load, and can play an important role in guiding the setting and implementation of service items near the designated area.
The embodiments of this specification can be run on any device with computing and storage capabilities, such as mobile phones, tablet computers, PCs (Personal Computers), notebook computers, servers, etc.; it can also be run on two or The logical nodes of more than two devices implement various functions in the embodiments of this specification.
The embodiments of the present specification use video streaming captured by a camera to estimate crowd flow conditions. The camera is placed above the designated area, and the crowd in the designated area is photographed at an obliquely downward angle to generate real-time video streaming. The shooting range of the camera can completely cover the designated area. Taking the advertising screen in FIG. 1 as an example, the designated area (inside the solid frame) is located in front of the advertising screen, and the camera can be placed above the advertising screen to continuously shoot the entire designated area at an oblique downward angle.
In the embodiment of the present specification, the flow of the method for estimating the flow of people in a designated area is shown in FIG. 2.
Step 210: Extract the currently valid frame from the real-time video stream of the designated area.
The camera installed above the designated area will continue to output the video stream taken at an obliquely downward viewing angle. The video stream consists of a continuous frame image. Based on a certain condition, the images of each frame that meet this condition can be continuously extracted from the video stream as an effective frame, and the crowd flow status can be estimated by continuously identifying the pedestrians in the effective frame. Use the last valid frame extracted from the video stream as the current valid frame.
The conditions for extracting effective frames can be set according to factors such as the requirements of the actual application scenario for statistical time accuracy, the processing capacity of the device running this embodiment, and so on. For example, the interval between the previous effective frame and the K (K is a natural number) frame One frame image is used as the next effective frame, and one frame can be extracted as an effective frame from every M (M is a natural number greater than 1) consecutive frames.
Step 220: Detect each pedestrian in the current valid frame, generate feature information of each pedestrian, and identify N (N is a natural number) single-frame attributes of each pedestrian according to the feature information.
After extracting the current effective frame, the target detection algorithm of deep learning is used to determine whether there is a human body in the current effective frame, and if so, locate the position of each pedestrian and part of the image area occupied by the pedestrian.
The target detection algorithm used in the embodiments of this specification is not limited. For example, Faster R-CNN (Faster Regions with Convolutional Neural Network features, fast target area recognition using convolutional neural network features), SSD (Single Shot MultiBox Detector, single target multi-box prediction), etc.
In application scenarios that require both low computational complexity and detection accuracy, you can use the YOLO (You Only Live Once) target detection algorithm to extract the image range and position information of each pedestrian from the current effective frame, often Can achieve better results.
For the position of each pedestrian output by the target detection algorithm and the part of the image area occupied by the pedestrian, the position characteristics of the pedestrian can be generated from the position of each pedestrian (such as the partial area occupied by the pedestrian in the picture coordinate system Coordinates), the image of the partial area occupied by the pedestrian generates the appearance characteristics of the pedestrian (such as clothing color, clothing texture, handbag, backpack, hat, etc.), taking the appearance characteristics, or the location characteristics and appearance characteristics as Characteristic information of the pedestrian.
In the embodiment of the present specification, the attribute is used to describe a specific aspect of the pedestrian. For example, the attribute may be one or more of gender, age range, body orientation, clothing, etc. You can determine which attributes are the objects of identification and statistics based on factors such as the actual application scene requirements and camera resolution. The attribute of the pedestrian identified by the image of a pedestrian in a valid frame is called the single-frame attribute of the pedestrian.
According to the characteristic information of each pedestrian in the current effective frame, the specific values of the N single-frame attributes of each pedestrian can be identified, such as gender attributes (male or female), age attributes (such as under 20 years old, 20 to 30-year-old, 30-40 years old, which age group is over 40 years old), body orientation attribute (such as front, side or back facing the camera).
Machine learning classification models can be used to identify the attributes of each single frame. Specifically, using the feature information of pedestrians as input, and the probability of each specific value of the attribute as the output to build a machine learning classification model, after training with labeled samples, you can use the classification model to derive individual boxes for pedestrians The estimated value of the attribute. The embodiments of this specification do not limit the types of machine learning classification models, such as binary classification models, multi-classification models, etc.; the machine learning algorithms adopted by classification models are also not limited, such as linear regression, decision trees , Random forest, etc.
In step 230, a pedestrian re-identification algorithm is used to identify the same pedestrian in the current effective frame and at least one previous effective frame according to the characteristic information.
When a pedestrian passes through the designated area, it will be photographed into multiple valid frames. In the estimation of crowd flow conditions, it is necessary to find the same pedestrian in each valid box to avoid counting the same pedestrian multiple times in order to obtain accurate information.
Person ReID (Person ReID, Person Re-identification) can use computer vision technology to determine whether a specific pedestrian exists in the image, and can be used to track people on the same camera or across cameras. In the embodiment of the present specification, a pedestrian re-identification algorithm is used to determine which of all pedestrians detected in the current valid box are pedestrians who have already appeared in the previous valid box and which are newly appeared pedestrians in the current valid box .
It can be judged whether the pedestrian is a new pedestrian by looking for whether a certain pedestrian in the current effective box appears in K (K is a natural number) effective boxes before the current effective box. Because the camera shoots the designated area at an oblique downward angle in the embodiment of the present specification, when pedestrians are dense, a pedestrian may be blocked by others in a valid frame or several consecutive valid frames without being detected In the situation, choosing a larger K value can avoid incorrectly counting the pedestrian in this case, but a larger K value will also bring greater computing load. In an actual application scenario, an appropriate K value may be selected according to factors such as the pedestrian density of the designated area, the interval between adjacent valid frames, and the processing capability of the device running this embodiment.
In one implementation, according to the appearance characteristics and location characteristics of pedestrians, it can be determined whether each pedestrian in the current valid box is an existing pedestrian in the previous K valid boxes, and if not, a new character identification is generated to mark the pedestrian , If yes, mark the pedestrian with the existing character ID.
Specifically, using the position and appearance characteristics of a pedestrian in the current valid box, find out whether the pedestrian already exists in the previous K valid boxes, and if not, generate a new character logo for the pedestrian and use the generated character The mark marks the pedestrian, and the character mark is used to uniquely represent a pedestrian, which can be an index number, a string, etc., without limitation. If the pedestrian already exists, the pedestrian already has his own personal identification, and the existing personal identification can be used to mark the pedestrian in the currently valid box. The above search process is performed on all the pedestrians detected in the current valid box one by one until each detected pedestrian is marked with a person identification.
The algorithm used when identifying whether the pedestrians in different valid frames are the same person can be selected according to the needs of the actual application scenario, without limitation. For example, a Hungarian algorithm can be used to match the current valid box with the pedestrian in the previous valid box according to the appearance characteristics and location features of the pedestrian.
Step 240: Determine the pedestrians passing through the designated area based on the result of the pedestrian re-identification algorithm, and obtain the statistical values of N personal attributes of the crowd flow through the designated area from the single-frame attribute statistics of the pedestrians passing through the designated area.
It can be determined according to the needs of actual applications which pedestrians pass through the designated area without limitation. The following examples illustrate:
First example: Pedestrians entering the designated area in each valid box can be regarded as pedestrians passing through the designated area. If a pedestrian in a valid box appears in the designated area, and the K valid boxes before the valid box do not appear in the designated area, the pedestrian is considered to enter the designated area in the valid box.
Second example: Pedestrians leaving the designated area in each valid frame can be regarded as pedestrians passing through the designated area. If a pedestrian has appeared in the specified area in the K effective frames before a certain effective frame, and the effective frame and 1 to K effective frames after the effective frame have not appeared in the specified area, then The pedestrian is considered to leave the designated area in the valid box.
After determining the pedestrians passing through the designated area, the flow of people passing through the designated area is a collection of these pedestrians. From the N single-frame attributes of these pedestrians in each effective frame, the statistical values of N personal attributes of the crowd flow through the designated area can be counted. Among them, the statistical value of the personal attribute is the statistical result of the specific value of the personal attribute of each pedestrian in a pedestrian collection, for example, the gender attribute (the total number of men and the number of women), the age attribute (such as the number of people under 20, 20 to 30 The number of people aged 30, 40 to 40 years old, the number of people over 40 years old), body orientation attributes (such as the number of people facing the camera on the front, the number of people facing the camera on the side, and the number of people facing the camera on the back).
When the statistics of personal attributes are generated from the single-frame attributes of pedestrians passing through the designated area, the N personal attributes of the pedestrian can be determined according to the N single-frame attributes of the same pedestrian in several valid frames. N personal attributes of all pedestrians passing through the designated area, and statistically obtaining statistical values of N personal attributes of the crowd flow through the designated area during the statistical period. Among them, the statistical time period is a time period for accumulating the statistical values of personal attributes.
When a pedestrian passes through a designated area, it usually appears in multiple valid frames. The pedestrian's attributes estimated based on the pedestrian's images in these valid frames are called the pedestrian's personal attributes. In each valid box where the pedestrian appears, the pedestrian will have N specific values of the single box attributes, and the specific values of the individual box attributes may be used to generate specific values of the corresponding personal attributes.
For example, in the first implementation, the specific value of a single box attribute that has the highest probability in all valid boxes where the pedestrian appears can be used as the specific value of the personal attribute corresponding to the pedestrian; in the second In an implementation manner, it is possible to accumulate the possibility of a specific value of a single-box attribute in all valid boxes where the pedestrian appears, and take the specific value with the largest accumulation result as the specific value of the personal attribute corresponding to the pedestrian. For example, a pedestrian appears in 3 valid boxes, and the specific values of the gender attributes in the single box attribute are: male 0.33, female 0.67 in the first valid box, male 0.52, female 0.48, the second valid box In the two effective boxes, male 0.42 and female 0.58, in the first implementation, the specific value with the highest probability is female 0.67, then the gender attribute of the pedestrian's personal attribute is female; in the second implementation, The total probability of males is 1.27 and the total probability of females is 1.73, so the gender attribute of the pedestrian's personal attributes is female.
The N personal attributes of all pedestrians passing through the specified area during the statistical period are accumulated separately to obtain the statistical values of the N personal attributes of the crowd flow through the specified area during the statistical period.
In the implementation method of marking each pedestrian with a character identification, the length of time between each pedestrian entering the valid frame of the designated area and the pedestrian leaving the valid frame of the designated area can be used as the pedestrian's stay in the designated area length of time. When counting the statistical values of the personal attributes of the crowd flow through the designated area, the length of stay of these pedestrians can also be counted, such as the total length of stay or the average length of stay.
It should be noted that, when the statistics of the personal attributes of the flow of people passing through the specified area are counted, the statistical value of each specific value of each personal attribute is accumulated to obtain the number of flow of people passing through the specified area. Traffic is accumulated.
It can be seen that in the embodiment of the present specification, the currently valid frame is extracted from the real-time video stream captured by the camera placed above the designated area, and by detecting the pedestrians in the currently valid frame, the feature information and the single-frame attributes of each pedestrian are generated , The pedestrian re-identification algorithm is used to identify the pedestrians in the current effective frame and the same pedestrians in the previous effective frame, determine the pedestrians passing through the specified area and count the personal attribute statistics of the crowd flow through the specified area, which can be lower Calculate the cost and accurately estimate the flow of people passing through the designated area, thus playing a guiding role in the setting and implementation of service projects near the designated area.
Due to the low computational load, the method of the embodiments of the present specification is suitable for running on an embedded development board, and has no special requirements on the hardware environment of the embedded development board. The embedded development board running the embodiment of this specification can be installed near the camera, and the crowd flow status data collected at a certain period can be sent to the server responsible for collecting the crowd flow status data through its own communication module without uploading the camera Films or images taken can obtain accurate statistics without intruding on the privacy of pedestrians.
The foregoing describes specific embodiments of the present specification. Other embodiments are within the scope of the attached patent application. In some cases, the actions or steps described in the scope of the patent application may be performed in a different order than in the embodiment and still achieve the desired result. Additionally, the processes depicted in the drawings do not necessarily require the particular order shown or sequential order to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.
In an application example of this manual, the quality of an advertising spot where an advertising screen is located needs to be evaluated. Use the area where pedestrians in front of the advertising spot can stay or pass and can understand the content of the advertising screen as the designated area. Install an RGB (Red Green Blue) camera in the middle of the top of the advertising screen. The downward viewing angle shoots the designated area, which is located in the central part of the shooting range and is separated from the boundary of the shooting range by a certain distance.
Crowd flow statistics are carried out by a program running on the NVIDIA embedded development board. The embedded development board is installed near the camera, which includes a communication unit, which can be connected to the camera by short-distance wireless connection, and obtain it from the camera Information about the film taken. The embedded development board can also upload the estimated crowd flow status data to a predetermined server through the communication unit.
The embedded development board pre-stores a machine learning classification model that has been trained using labeled samples. The classification model takes the appearance features of pedestrians in a frame of images as input, and takes the gender, age, body orientation, and clothing. The probability of the specific value of the attribute is the output.
The structure of the crowd flow statistics software running on the embedded development board is shown in Figure 3.
The RGB camera continuously captures images of open scenes at a rate of 25 frames per second to form a video stream. The embedded development board extracts a frame of RGB images every fixed number of frames from the captured video stream as the current effective frame.
The crowd flow statistics software uses the YOLO target detection algorithm to identify each pedestrian in the currently valid frame, determine the position coordinates (a type of position feature) of each pedestrian in the image coordinate system, and the portion occupied by each pedestrian The image of the area.
For each pedestrian in the current valid frame, the appearance characteristics of the pedestrian are extracted from the image of the area occupied by the pedestrian. By inputting appearance features into the machine learning classification model, the probability of each specific value of the pedestrian's four single-frame attributes can be obtained.
Based on the appearance characteristics and position coordinates of each pedestrian in the current effective frame respectively, the pedestrian re-identification algorithm uses the Hungarian algorithm to determine whether the pedestrian matches each pedestrian in the three effective frames before the current effective frame. Find out whether the pedestrian has appeared in the previous 3 valid boxes. If it has not appeared before, then generate a new character logo for the pedestrian to uniquely represent the pedestrian, and mark the pedestrian with the new character logo; if it has ever appeared, then Mark the pedestrian with the character identification that the pedestrian already has.
Pedestrians who enter and leave the designated area in each valid frame are determined based on the personal identification of the pedestrian re-identification algorithm to mark each pedestrian in the current valid frame and the position coordinates of each pedestrian output by the YOLO target detection algorithm. If a pedestrian in a valid box appears in the designated area, and the three valid boxes before the valid box do not appear in the designated area, the pedestrian is considered to enter the designated area in the valid box; if a certain If the pedestrian has appeared in the 3 valid boxes before a valid box in the designated area, and the valid box and the 2 valid boxes after the valid box have not appeared in the designated area, the pedestrian is considered to be in the valid area The box leaves the designated area.
Taking a predetermined statistical time period as a cycle, for all pedestrians who leave the specified area for all valid frames in the statistical time period, accumulate the specific values of each single-frame attribute of each valid frame where the pedestrian appears, and combine The cumulative and highest specific value is taken as the specific value of the personal attribute corresponding to the pedestrian; the time interval between the effective frame where the pedestrian leaves the specified area and the effective frame where the pedestrian enters the specified area is taken as the length of time the pedestrian stays in the specified area .
Accumulate the number of pedestrians who leave the designated area in all valid boxes in a period by sub-attributes and specific values, and take the cumulative result of each specific value of each individual attribute as the statistical result of the crowd flow status in the statistical time period; accumulate in one cycle The length of stay of all pedestrians leaving the designated area in the valid box, and calculate the average length of stay of these pedestrians.
After the end of a period, the embedded development board sends the statistical results of the crowd flow status of the period and the average length of stay of the crowd flow to the predetermined server.
After the server receives the information uploaded by the embedded development board, it can evaluate the quality of the advertising spots based on the personal attribute statistics of the crowd flow through the designated area. The server can obtain detailed information about the number of crowd traffic near the point in each statistical period, how long the crowd traffic stays in front of the advertising point, whether the crowd traffic is positive, sideways or back to the screen, etc., not only to the advertiser Accurate ad placement recommendations can also provide fine quote basis for advertising operators.
Corresponding to the implementation of the above process, the embodiments of the present specification also provide a device for estimating the flow of people in a designated area. The device can be realized by software, or by hardware or a combination of hardware and software. Taking software implementation as an example, as a logical device, it is formed by the CPU (Central Process Unit, central processing unit) of the device where it reads the corresponding computer program instructions into the internal storage and runs it. From the hardware level, in addition to the CPU, internal storage, and storage shown in FIG. 4, the device where the crowd flow estimation device in the designated area is located usually includes other hardware such as chips for wireless signal transmission and reception. , And/or other hardware such as boards for network communication.
FIG. 5 shows a device for estimating a crowd flow condition in a designated area provided by an embodiment of the present specification, including an effective frame extraction unit, a pedestrian detection and feature unit, a pedestrian re-identification unit, and an attribute statistical unit, where the effective frame extraction unit is used to Extract the current valid frame from the real-time video stream shooting the specified area; the video stream is shot by a camera placed above the specified area, and its shooting range covers the entire specified area; pedestrian detection and feature unit are used to detect the current effective Each pedestrian in the frame generates feature information of each pedestrian, and identifies N single-frame attributes of each pedestrian based on the feature information; N is a natural number; the pedestrian re-recognition unit is used to adopt a pedestrian re-identification algorithm based on feature information Identify the same pedestrians in the current valid box as at least one valid box before; the attribute statistics unit is used to determine the pedestrians passing through the specified area based on the results of the pedestrian re-identification algorithm. Statistical value of N personal attributes of crowd flow in the specified area.
Optionally, the pedestrian detection and feature unit identifies the single-frame attributes of each pedestrian based on the feature information, including: using a machine learning classification model to identify several single-frame attributes of each pedestrian; the machine learning classification model uses pedestrians’ Feature information is input, and labeled samples are used for training.
Optionally, the attribute statistical unit obtains statistical values of N personal attributes of the crowd flow through the specified area from the single-frame attribute statistics of pedestrians passing through the specified area, including: according to the N of the same pedestrian in several valid frames The single-frame attributes determine the N personal attributes of the pedestrians. The N personal attributes of all pedestrians passing through the specified area during the statistical period are counted to obtain the N personal attributes of the crowd flow through the specified area during the statistical period Statistics.
Optionally, the single-frame attribute and the personal attribute respectively include one or more of the following: gender, age group, body orientation, and clothing.
In one example, the feature information includes: appearance characteristics and position characteristics; the pedestrian re-identification unit is specifically configured to determine whether each pedestrian in the current valid frame is at least one of the previous ones according to the appearance characteristics and position characteristics of the pedestrian If the pedestrian already exists in the valid frame, a new character identification is generated to mark the pedestrian, and if it is, then the pedestrian is marked with an existing character identification.
In the above example, the pedestrian re-identification unit determines whether a pedestrian in the current effective frame is a pedestrian already in the previous effective frame according to the appearance characteristics and location characteristics of the pedestrian, including: using the Hungarian algorithm, based on the appearance of the pedestrian Features and location features are used to match the current valid box with the pedestrian in the previous valid box.
In one implementation, the pedestrians passing through the designated area include: pedestrians entering the designated area, or pedestrians leaving the designated area; the pedestrians entering the designated area include: appearing in the designated area in a valid box, and in Pedestrians that do not appear in the designated area in at least one valid frame before the valid frame; the pedestrians that leave the designated area include: the adjacent valid box that precedes a valid frame appears in the designated area and is in the designated area Pedestrians that do not appear in the designated area in at least one valid frame and the following valid frame.
In the above implementation manner, the device further includes: a residence time length statistical unit for counting the length of stay time of the crowd flow through the designated area in the designated area, wherein the length of residence time of each pedestrian passing through the designated area is determined by the pedestrian The length of time between the valid frame entering the designated area and the valid frame leaving the designated area is determined.
Optionally, the device runs on an embedded development board.
Optionally, the designated area includes: a designated area before an advertising spot; the device further includes: a spot evaluation unit, which is used to determine the quality of the advertising spot according to the personal attribute statistical value of the crowd traffic passing through the designated area to evaluate.
The embodiments of the present specification provide a computer device including a memory and a processor. Among them, a computer program that can be run by the processor is stored on the storage; when the processor runs the stored computer program, each step of the method for estimating the flow of people in the designated area in the embodiment of the present specification is executed. For a detailed description of the various steps of the method for estimating the flow of people in the designated area, please refer to the previous content and will not be repeated.
The embodiments of the present specification provide a computer-readable storage medium on which computer programs are stored. When the computer programs are executed by a processor, each step of the method for estimating the population flow situation in the designated area in the embodiments of the present specification is executed . For a detailed description of the various steps of the method for estimating the flow of people in the designated area, please refer to the previous content and will not be repeated.
The above are only the preferred embodiments of this specification and are not intended to limit this application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of this application should be included in this application Within the scope of protection.
In a typical configuration, the computing device includes one or more processors (CPUs), input/output interfaces, network interfaces, and internal storage.
Internal storage may include non-permanent storage in computer readable media, random access memory (RAM) and/or non-volatile internal storage, such as read only memory (ROM) or flash memory Body (flash RAM). The internal storage is an example of a computer-readable medium.
Computer-readable media, including permanent and non-permanent, removable and non-removable media, can be stored by any method or technology. The information can be computer readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase change random access memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), and other types of random access memory (RAM), read-only memory (ROM), electrically erasable and programmable read-only memory (EEPROM), flash memory or other internal storage technologies, read-only discs (CD-ROM), digital A functional optical disc (DVD) or other optical storage, magnetic cassette tape, magnetic tape magnetic disk storage or other magnetic storage device or any other non-transmission medium can be used to store information that can be accessed by a computing device. According to the definition in this article, computer-readable media does not include temporary computer-readable media (transitory media), such as modulated data signals and carrier waves.
It should also be noted that the terms "include", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device that includes a series of elements includes not only those elements, but also includes Other elements not explicitly listed, or include elements inherent to this process, method, commodity, or equipment. Without more restrictions, the element defined by the sentence "include one..." does not exclude that there are other identical elements in the process, method, commodity, or equipment that includes the element.
Those skilled in the art should understand that the embodiments of this specification can be provided as methods, systems, or computer program products. Therefore, the embodiments of the present specification may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware. Moreover, the embodiments of the present specification may employ computer programs implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code The form of the product.
