Machine Learning-Based Detection of Malicious Web Requests Using Hybrid Synthetic and Real Web Server Logs

Abstract

Web-server logs contain a detailed trace of how users, automated tools, and attackers interact with an application. This makes them a practical source for detecting web attacks, but it also creates a familiar research problem: models trained or validated only on synthetic or benchmark traffic may appear highly accurate while failing to identify attacks in more realistic operating conditions. This paper compares three supervised machine-learning models for malicious HTTP request detection: Logistic Regression, Random Forest, and Gradient Boosting. The experiment uses two data sources: a synthetic web-attack dataset derived from the CSIC 2010 HTTP corpus and an anonymized set of commercial web-server logs containing real attack patterns. Seven features were selected from HTTP requests and server responses: source IP, URL, User-Agent, URL length, status code, GET method indicator, and POST method indicator. The models were evaluated using confusion matrices, accuracy, precision, recall, F1-score, ROC-AUC, PCA-based decision-boundary visualization, and learning curves. On the synthetic dataset, all models achieved strong results, with Random Forest and Gradient Boosting producing near-perfect ROC-AUC values. On the real-world dataset, the difference between models became more visible. Random Forest achieved high specificity but missed most positive cases, producing only 7 true positives and 136 false negatives. Logistic Regression improved recall but still left 88 attacks undetected. Gradient Boosting provided the best operational balance, with 129 true positives, 14 false negatives, 4 false positives, 98.62% accuracy, 96.99% precision, 90.21% recall, 93.48% F1-score, and 1.00 ROC-AUC. The results suggest that ensemble boosting is a strong candidate for log-based web attack detection when detection performance must remain stable across synthetic and real traffic.