Download
Apache Log Parser and Data Normalization Application
Python handles File Processing & MySQL handles Data Processing
ApacheLogs2MySQL consists of two Python Modules & one MySQL Schema to automate importing Access & Error files and normalizing data into database designed for reports & data analysis.
Runs on Windows, Linux and MacOS & tested with MySQL versions 8.0.39, 8.4.3, 9.0.0 & 9.1.0.
4 LogFormats & 2 ErrorLogFormats can be loaded and 5 MySQL Stored Procedures can be processed in a single Python `ProcessLogs function` execution.
Database system designed to accommodate unlimited domains. Step-by-step guide for easy installation.
Web interface with Drill Down Capability and apache/echarts Log Visualization integration in development.
MySQL apache_logs schema currently has 49 Tables, 853 Columns, 168 Indexes, 66 Views, 7 Stored Procedures and 43 Functions to process Apache Access log in 4 formats & Apache Error log in 2 formats. Database normalization at work!
Features
- Creates MySQL schema - apache_logs
- Includes 2 Python modules
- Imports Apache Access Logs - common, combined and vhost_combined
- Imports Apache Error Logs
- Complete Apache Log Data Normalization
- MySQL does all Data Manipulation & Processing
- Python handles polling of log file folders
- Python executes MySQL Database LOAD DATA statements
- Python executes Stored Procedures & Functions and SQL Statements
- Requires installation of 4 Python modules - requirements.txt included
Project Samples
