System logs are vital to knowing what's going on with your systems. System logs capture a variety of system and application information and can help you ascertain the health and security of your systems. And these days, compliance requirements such as the Payment Card Industry Data Security Standard (PCI, for short) and the Sarbanes-Oxley Act (SOX) make logging and log retention required. Your particular business may also have other legal and regulatory requirements.

The next few postings here will show the way to building a high-performance, reliable, and secure logging infrastructure. The techniques here are not meant for a handful of hosts in a single location. We'll be looking at a multi-tier, world-wide system that can handle hundreds of millions of log entries per day.

We'll also talk about ensuring the integrity of your system logs to make the data therein useful and reliable, even for the legal and compliance world.

None of this is revolutionary or even particularly difficult, but I wanted to collect the techniques into one place, almost like a recipe, after having spent a lot of time assembling such a system over the months.

Scenario Details:

• Heterogeneous (Unix/Linux/Windows) hosts
• Heterogeneous (Cisco/Juniper/other) network devices
• Geographically dispersed data centers and points-of-presence (POP)
• Compartmentalized (firewalls) networks
• Requirements for PCI, SOX, and other monitoring
• Requirements for remote logging to aid in intrusion detection and forensics

Components:

• logging hosts with lots of disk capacity
• Syslog-NG
• Server load-balancing and Virtual IP network gear
• Standard software packaging and installation methods
• Scripts/tools to generate parallel requests
• Indexing tools to index and search logs
• Monitoring tools to read log streams

Index to the postings (links updated as we progress):

• The Scheme
• The Client
• The Transport, Part I
• The Relay
• The Transport, Part II
• Availability
• Storage
• Log Integrity
• Consuming The Logs
• Niceties
• The Recipe (checklists galore)

References:

• Syslog-NG
• Global server load-balancing
• Virtual IP address
• PCI DSS
• Sarbanes-Oxley Act