The client layer is where the logs are generated. I often refer to these as the 'edge hosts', but that doesn't necessarily mean they are all at the network edge. The clients can be on a variety of networks and in various locations around the world.

A word of warning: In a large and heterogeneous environment, the client layer is where you're going to have to make the most compromises.

Many clients, especially older hosts and network gear, can't have their syslog software upgraded or replaced. The saving grace is that almost everything supports network syslog delivery. We'll take advantage of this to get the logs off the host and into our infrastructure.

Some things to consider:

• Do you have space for all those logs? You should profile the remote clients' log usage, over a week and over a month, to determine if you're going to be able to take them all and keep them as long as you'd like.
• Along with that daily volume number, consider the rate at which logs are written. Do you have a fast and furious period or are logs spread evenly throughout the day? The rate of arrival (and projected increases) will affect how you spread your syslog master hosts and what type of storage you'll need on them.
• Do you have admin access to upgrade clients or modify existing syslog client configurations? You'll need the access or the commitment of your Operations peers to get the configurations done.
• Where are your clients? Are they across WAN links, VPNs, or other network hops? Be aware and prepared to get firewall ACLs updated to allow traffic to pass. In our solution here, we use a local collector to capture UDP logs, then take advantage of TCP reliability across the miles.
• Do you have access to the clients to generate log entries (typically via logger or a similar utility)? This is needed as you move through each site to confirm that logs are flowing as they should.
• Have you considered conversion tools for non-syslog hosts, such as Microsoft Windows? There are both pay and free clients out there.

Now, what to send? I'm strongly in favor of sending everything that you can, provided you have the storage (which is cheap!) to hold it. That makes the client configurations straightforward, which speeds up the work and testing. That's a bonus if you have others helping you do the setup.

How to send it? If you can manage it, which means package, distribute, and configure the application in a repeatable way, I strongly suggest using Syslog-ng as a client. Using Syslog-ng gives you rich filtering at the client and TCP connectivity, which improves delivery reliability.

If you can't do this, add a remote logging line to syslog.conf (or similar config, depending on your device). From the syslog.conf man page:

       *.*                          @finlandia
       This rule would redirect all messages to a remote host 
       called finlandia.  This is useful especially in a cluster 
       of machines where all syslog messages will be stored on 
       only one machine.

This remote configuration will send your syslog packets over UDP to the remote host.

One other thing to remember, as you convert clients over to the new syslog infrastructure, you don't want newly deployed clients to be missed or require re-work. Work with your Operations peers to get the updated syslog configs built into your base configurations. That way, new hosts will be deployed and begin logging immediately.

Don't worry, we're only covering concepts now. At the end of this series, I'll provide a recipe with the order of operations needed to cook up your logging system.

Prior articles in this series:

• Series Overview
• Syslog Scheme

Part of my job is helping our system administrators respond to attacks against our web properties. These can include SQL injection, password cracking, and fraudulent account registrations, among others.

One of the fingerprints of these three types of activity is the repetitive nature of the attack. Attackers are greedy, generally wanting to get in, do the work, then get out, so they are often noisy in doing so. This activity stands out, but since it's mixed in with legitimate traffic, it's sometimes difficult to weed out the good traffic from the bad.

So, I use a script to find these top talkers, the noisy abusers who are hitting our sites to do badness.

$ top_talkers.sh access_log
KEY: 63.7.190.134 Count: 3942
KEY: 54.15.88.41 Count: 294
KEY: 227.9.112.155 Count: 155
KEY: 212.49.27.32 Count: 122
KEY: 129.94.7.56 Count: 111

In this example, it's easy to see that 63.7.190.134 is doing a lot more account registrations than other users, so that's where we'll start our investigation. These are, of course, bogus IP addresses.

The script is simple. It collects loglines matching a pattern, then counts, in this example, the IP addresses that hit a certain page, 'registerAcct.jsp'.

The script is small, something I can drop into /var/tmp/ and run as needed.

Modify line 27, GREP_FILTER, to suit your needs. Widen your search by increasing line 20, LOG_LINES. I keep the number low for active attacks so that it runs quickly.

Download the script here.

The first step in building our high-performance syslog delivery infrastructure is a review of how the thing works at its most basic level.

Traditionally, the syslog protocol on Unix-like hosts and network devices uses the UDP transport. UDP lacks any mechanism to ensure a connection is made and that packets are delivered. This alone makes standard syslog prone to quiet delivery failures on large networks, especially where network paths involve multiple router hops or tunneling. UDP may also be blocked at a router or firewall, preventing the syslog traffic from flowing.

Secondarily, the configuration of standard Unix syslogd offers little flexibility beyond facility and priority in directing delivery to log files or remote logging systems.

So, to solve these two problems, I chose (along with thousands of others) Syslog-ng as my logging and relaying application. Syslog-ng accepts traditional UDP syslog traffic, which is useful for a migration or for network devices whose syslogging facility can not be modified or replaced, as well as accepting/generating TCP syslog traffic which vastly improves delivery, and troubleshooting of delivery difficulties.

Syslog-ng also has rich set of listening, filtering, and relaying options that provide one with extreme flexibility in managing large volume, disparate log contents.

The following diagram illustrates a tiered collection system:

Given uptime and reliabilty requirements, this tiered system allows one to manipulate and replace components with minimum impact. Syslog traffic flows down the diagram, from the client (servers and network devices) to a collector layer, which may be geographically or otherwise centralized, to load-balancing, high-availability Layer-4 network gear, perhaps a Foundry switch or Netscaler/BigIP device.

Behind that, we have an aggregation and reporting tier where syslogs ultimately reside and may be archived, indexed, and consumed in other ways. And that's really the point of this whole thing, having the logs we need to troubleshoot and secure systems, as well as meet regulatory requirements.

References:

• OSI Model: Transport Layer
• Citrix Netscaler
• F5 BigIP
• Foundry switch

Prior articles in this series:

• Series Overview

System logs are vital to knowing what's going on with your systems. System logs capture a variety of system and application information and can help you ascertain the health and security of your systems. And these days, compliance requirements such as the Payment Card Industry Data Security Standard (PCI, for short) and the Sarbanes-Oxley Act (SOX) make logging and log retention required. Your particular business may also have other legal and regulatory requirements.

The next few postings here will show the way to building a high-performance, reliable, and secure logging infrastructure. The techniques here are not meant for a handful of hosts in a single location. We'll be looking at a multi-tier, world-wide system that can handle hundreds of millions of log entries per day.

We'll also talk about ensuring the integrity of your system logs to make the data therein useful and reliable, even for the legal and compliance world.

None of this is revolutionary or even particularly difficult, but I wanted to collect the techniques into one place, almost like a recipe, after having spent a lot of time assembling such a system over the months.

Scenario Details:

• Heterogeneous (Unix/Linux/Windows) hosts
• Heterogeneous (Cisco/Juniper/other) network devices
• Geographically dispersed data centers and points-of-presence (POP)
• Compartmentalized (firewalls) networks
• Requirements for PCI, SOX, and other monitoring
• Requirements for remote logging to aid in intrusion detection and forensics

Components:

• logging hosts with lots of disk capacity
• Syslog-NG
• Server load-balancing and Virtual IP network gear
• Standard software packaging and installation methods
• Scripts/tools to generate parallel requests
• Indexing tools to index and search logs
• Monitoring tools to read log streams

Index to the postings (links updated as we progress):

• The Scheme
• The Client
• The Transport, Part I
• The Relay
• The Transport, Part II
• Availability
• Storage
• Log Integrity
• Consuming The Logs
• Niceties
• The Recipe (checklists galore)

References:

• Syslog-NG
• Global server load-balancing
• Virtual IP address
• PCI DSS
• Sarbanes-Oxley Act

Take the survey.

Take part in the 2007 Perl Survey!

The Perl Survey is an attempt to capture a picture of the Perl community in all its diversity. No matter what sort of Perl programmer you are, we'd love to hear from you.

The survey can be found at: http://perlsurvey.org/

It only takes about 5 minutes to complete.

The survey will be open until September 30th, 2007. After that, we'll be reporting on the results and making the data freely available.

Please feel free to forward this email to any other Perl programmers you know.

Thanks for your help!

Yours,

Kirrily "Skud" Robert
The Perl Survey
info@perlsurvey.org

A guy (Scott Lazzari) on the SAGE list asked:

I've been tasked with putting together a risk assessment for the local office where I do nuts-to-bolts IT support. So far, I've identified the key equipment, and assigned a criticality level to this equipment. I'm not sure where I should go from here. My background is much more tech-oriented - fixing and installing equipment, servers, etc. so this level of business analysis is a little new to me.

Does anyone have some good resources, or advice they could drop my way?

Summary of some risk assessment resources, with responders, suggested in response:

• http://www.securityfocus.com/infocus/1591 [John Buys]
• http://www.securitymetrics.org/content/Wiki.jsp [Lynda]
• http://www.iatrp.com/ [Brian Kirouac]
• www.rcmp-grc.gc.ca/tsb//pubs/it_sec/g2-001_e.pdf [Sean MacGuire, he of Big Brother fame

Somewhere pretty far along in your career, you should be thinking strategically. New projects, system improvements, proactive, all the buzzwords.

But then that host croaks.

There goes the day.

Attending technical conferences can provide the system administrator with a number of benefits. A good conference can broaden or deepen your skills, expose you to the state of the art, and provide networking opportunities and some valuable recharging and entertainment away from the office grind. And it can be fun.

At conferences, I tend to choose training sessions on topics or problems that maybe we don't have today, or haven't identified yet as a weakness or opportunity. For example, we don't conduct our own penetration testing or web application reviews (we have a dedicated Security team for that), but therein lies an opportunity for our admins to become trained, or certainly more aware of the practices in this area that can make us more secure. "Many hands make light work" and all... So I signed up for session on penetrating and exploiting web applications.

The tough part (and potentially a cause of expectation mismatch with your boss) might be your ability to return from the conference, head aswim with ideas, get dropped back into the fray, yet still find the time and energy to share what you've learned with your team. "We spent $X,000 and what do we have to show for it?"

What to do?

• Make your course guides and handouts available to your team as soon as you return.
• Send a summary of the classes you took and their potential applicability in your world to your team. Keep it short, they're busy.
• Over the next few weeks or months, as you have opportunities, revisit your course materials and consider them in the light of your current job role or interests.
• Consider teaching a "potpourri" session where you cover the "5 Coolest Things I Learned At Usenix", for example, to give people a jumping-off point for new ideas.
• If you're shy of speaking in public, share that information on a team wiki or internal blog. I'd do that anyway, teaching or not, since your teammates may vary in how they consume new ideas.
• If you decide to "teach", don't attempt to reteach the course. Unless you took copious notes, you'll likely miss something teaching from someone else's powerpoint deck.

While the kids watched Tarzan this afternoon, I spent about an hour re-reading the presentation notes from Dan Geer's Measuring Security session at USENIX in June of last year. In light of new responsibilities and changing emphases in my job since then, I came away with 9 new tasks or ideas that my team can or should do. They're now on the GTD list and in the pipe to make us more aware and hopefully make our infrastructure more secure.

Jacob's has posted the deck for his Web2.0 Expo talk on Geographic Distribution for Global Web Application Performance

My friend Rob Carlson has assembled an Amazon list: Essential Developer/Sysadmin Toolkit

The only thing I'd add would be Database Nation, by Simson Garfinkel, as a solemn reminder of the privacy and trust implications we face as sysadmins when handling our user's business data and email.

I applaud the Getting Things Done as the first item on the list.

It's now less than six days until the mini-y2k of our newly adjusted daylight savings time switchover date.

Are you patched?

# mkdir `ifconfig | grep inet | grep Bc | awk '{print $2}' | awk -F: '{print $2}' | tr . _`

Some days we move the platform/product/team/company forward. Some days we tend the garden. Both are needed. My exercise the past two weeks has been the planning and prioritization of the my group's infrastructure projects for the year. I'm lucky in that my boss and I are sympatico, seeing the opportunities and shortcomings in much the same way. Still, it's hard to say what the landscape will look like in October - that's a whole nine months away. Yet, we try. I'm balancing the garden-tending against the big initiatives, trying to not let the weeds overtake us.

This time of year, starting with a fresh list (although with some carryover) emphasizes the personal satisfaction that I find in my position. I'd never plead indispensability, but it's good to be dedicated and focused and know that we'll get support for most of the good projects and finish them. Looking back at what we did in 2006, project-wise, puts a soft-focus on the year and takes the edge off days of host recoveries, difficult on-call weeks, and the occasional pettiness of daily corporate life.

I've thought a lot about how to make a career. I have friends who are attorneys, engineers, doctors, and accountants. Their professional paths are well-defined. Knowledge and skill are prized among them. Bigger cases, projects, and deals are the hallmarks of growing and progressing in those fields. It's human nature probably to compare jobs, so I weigh my days as a system administrator often, and check the progression. In my field, unless you head into management, the careers progress with projects and innovation. Are the projects technically challenging? Do they move us forward or are you tending garden? Are they bold or simply incremental? These are the things I consider.

So what am I doing? I can't very well list my projects here, but the areas of focus are very buzzword-compliant. To wit:

• Privacy and Security
• Integration and Standardization
• Redundancy (with resulting service availability goodness)
• Reporting and Monitoring

Each of these areas has a bunch of verbs, "improve", "upgrade", "migrate", "decomm" (my favorite!), and objects such as mail and DNS. Some of the projects are technical challenges, while others simply need a long span of attention to finish - no wondering off after that next shiny thing.

The interesting part of this whole exercise, beyond moving us forward, is the balancing of company interests and goals with my professional goals, interests, and skills. Somehow it all works out, maybe I'm good with puzzles, and we now have a set of marching orders.

Google has chosen Lenoir, NC for a new data center. Yahoo Finance reports:

Search engine giant Google Inc. plans to spend $600 million to build a data center in North Carolina, state officials and the company said Friday.

...

The state will give the company $4.8 million as part of a total incentives package that could reach more than $100 million.

Nothing on the Google site yet about it.

Yahoo was a leader in many areas - search, portal, messaging. But as they've aged, their engineering teams are beginning to suffer some of the same problems as the rest of us. Cutting-edge platforms (at one time), often of a proprietary nature, need a hard look and difficult, often expensive, choices need to be made about the continued use of those platforms.

A Yahoo insider comments on their dead-end infrastructure:

And let me tell you this. Yahoo! is now rotten from the inside out. Here's my take of how to fix Yahoo!'s engineering:

... 4) Slowly port all Yahoo! software to linux and phase out FreeBSD. Start supporting and encouraging multi-threading programming. I bet Google is laughing their asses off at us because we are still stuck with FreeBSD, gcc-2.95 and single process model.

...

5) Slowly get rid of all Yahoo-specialized open source software. Why do we have "YApache" (based on Apache 1.3), and why do we have the dreaded yut/ycore++ libraries when we can use STL and boost? And why do we have YPAN when we can just use CPAN??? The platform group is doing the wrong job supporting this dead-end infrastructure.