Multimodal Deep Learning Information Fusion for Fine-Grained Traffic State Estimation and Intelligent Traffic Control

Abstract

Traffic analysis from real time camera images is the most adopted method for traffic state estimations due to many reasons like easy installation, low cost and portability. Many methods have been proposed to estimate traffic density. The current methods only provide an estimate in terms of number of vehicles on the lane or volume of lane occupied. But this information alone is not sufficient for intelligent traffic management. A more accurate traffic state estimation in terms of density, speed, flow for different category of vehicles in different segments of lane is needed in presence of various clutters and occlusions. This research work addresses this problem and proposes a multimodal deep learning information fusion based framework for traffic state estimation in presence of clusters and occlusions.   The multimodal deep learning information fusion based framework has three important components. The first component is vehicle categorization using hybrid traditional and deep learning features and estimation of traffic for vehicle category. The second component is categorization of different regions of road segments for presence of emergency vehicles using both visual and audio cues. The third component is construction of traffic state map and usage of the traffic state map for intelligent traffic control. In this work, fine-grained traffic state information refereed as a traffic state map is constructed using deep learning models. A traffic state map is dynamic with information on vehicular density, movement, and flow information. Compared to round-robin-based traffic scheduling at traffic signals, can realize more effective traffic scheduling with the traffic information map. Real-time camera-based traffic analysis has become the most widely adopted method for estimating traffic states due to its easy installation, cost-effectiveness, and portability. Although various methods have been proposed for estimating traffic density, the existing approaches typically provide only a basic estimate in terms of the number of vehicles or lane occupancy. However, this limited information falls short for intelligent traffic management. To address this issue, this research introduces a novel approach based on multimodal deep learning information fusion for accurate traffic state estimation, considering density, speed, and flow for different vehicle categories within various lane segments, even in the presence of clutters and occlusions. The proposed framework consists of three crucial components. The first method involves vehicle categorization using a combination of traditional and deep learning features to estimate traffic for each vehicle category. The second method focuses on categorizing different road segments to detect the presence of emergency vehicles, utilizing both visual and audio cues. Finally, the third method entails constructing a dynamic traffic state map using deep learning models. This traffic state map provides fine-grained information on vehicular density, movement, and flow. By leveraging the traffic state map, intelligent traffic control can be achieved, allowing for more effective traffic scheduling compared to conventional round-robin-based traffic signal scheduling. This framework holds promise for enhancing traffic management and optimizing the flow of vehicles on the road