IBM has an excellent article on virtualization techniques and architectures.

From a business perspective, there are many reasons for using virtualization. Most come down to what's called server consolidation. Simply put, if you can virtualize a number of under-utilized systems on a single server, there are distinct savings in power, space, cooling, and administration due to having fewer servers. Because it can be difficult to determine server utilization, virtualization technologies support what's called live migration. Live migration allows an operating system and its applications to be migrated to a new server to balance the load over the available hardware.

Technorati Tags: system administration

If you're running PHP on production sites, especially in a shared hosting environment, you should probably add the PHP Security Blog to your RSS reader.

If you can get past the mild case of bad attitude, the details are very interesting.

Technorati Tags: system administration

Wired has a fascinating article on the changes at Google and at Ask.com to build out infrastructure to support cloud computing for searches and other tasks.

Although the evergreen mazes, mountain majesties, and always-on skiing surely play a role, two amenities in particular make this the perfect site for a next-gen data center. One is a fiber-optic hub linked to Harbour Pointe, Washington, the coastal landing base of PC-1, a fiber-optic artery built to handle 640 Gbps that connects Asia to the US. A glassy extension cord snakes through all the town's major buildings, tapping into the greater Internet though NoaNet, a node of the experimental Internet2. The other attraction is The Dalles Dam and its 1.8-gigawatt power station. The half-mile-long dam is a crucial source of cheap electrical power- once essential to aluminum smelting, now a strategic resource in the next phase in the digital revolution. Indeed, Google and other Silicon Valley titans are looking to the Columbia River to supply ceaseless cycles of electricity at about a fifth of what they would cost in the San Francisco Bay Area. Why? To feed the ravenous appetite of a new breed of computer.

Find The Dalles, on Google Maps, of course or see pictures of the super secret place.

Technorati Tags: system administration

I sent this to my team recently:

Here are the various security vulnerability sources I am using daily:

• http://secunia.com/
• http://www.securityfocus.com/
• http://www.securityfocus.com/vulnerabilities
• http://packetstormsecurity.org/advisories50.html

I'm also reading RSS feeds for the following:

• bugtraq
• incidents
• full-disclosure

You can get subscription info at http://seclists.org/

The SANS Internet Storm Center RSS is also good,http://isc.sans.org/

There's occasional duplication in some of these.

Computer security professional, Bruce Schneier, makes some good points about the importance of some software to an industry or even the economy. And he says, for the one-millionth time, "practice defense in depth."

It's a situation that snuck up on us. Everyone knew that the software that flies 747s or targets cruise missiles was critical, but who thought of the airlines' weight and balance computers, or the operating system running the databases and spreadsheets that determine which cruise missiles get shipped where?

And over the years, common, off-the-shelf, personal- and business-grade software has been used for more and more critical applications. Today we find ourselves in a situation where a well-positioned flaw in Windows, Cisco routers or Apache could seriously affect the economy.

...

If we were to get serious about critical infrastructure, we'd recognize it's all critical and start building security software to protect it. We'd build our security based on the principles of safe failure; we'd assume security would fail and make sure it's OK when it does. We'd use defense in depth and compartmentalization to minimize the effects of failure. Basically, we'd do everything we're supposed to do now to secure our networks.

I'd add that the ability to quickly respond to an exploit or vulnerability comes from being prepared. You should never have to hand-compile Apache and push it to your web-servers or futz with some arcane dependency problems in the face of an attack or vulnerability.

Take the time now, with no one in your face, to package your software and work out the dependencies. Practice the drill for remediating a serious flaw. As an administrator who cares about security, ask yourself, "how would I react to the announcement of a serious flaw in ________ (choose your most visible, important, or exposed piece of software)?"

Rinse and repeat.

Work out the weaknesses in your packaging, communications, and processes. The processes may not exist or may be broken, better to find out now than during an incident, right?

Now you have a to-do list. Get to work.

Technorati Tags: system administration

The next set of activities in the engineering design process can be called "transformation." That is, the effort to transform the problem at hand into something solvable, moving from the unfamiliar to the familiar. We're adding progressively more structure as we move through the phases.

In this phase, we may begin to see pressures from the customer to produce. Be sensitive to that, while relating the problems to the long-term strategy or architecture framework under which you work. After the problem analysis, the customer may expect a fully-formed solution to just pop out. Manage those expectations early in the design process.

The guiding question in this phase is, "Have we solved this problem or something similar to this before?"

By answering that question, we decide whether we will be able to leverage prior efforts, avoid prior mistakes, or perhaps do both. Both of these allow us to move the problems toward solutions more quickly and hopefully in a more cost-effective way than blazing a new trail, although that too might be required.

To leverage prior efforts, we have to discern patterns in the problem that fit the work we've done before. Some examples of patterns in the SysAdmin world might be batch-processing, sessions, synchronous versus asynchronous communication, or layered security. These patterns are part of the problem domain that we noted in the analysis phase. The domain will have basic classes of objects and formalizes the heirarchy of those objects. As I noted in the first article in this series, the body of knowledge that might document this domain heirarchy is still not fully mature in much of the IT industry.

This problem seems brand-new, something you've never tackled? Working with the customer, manipulate it a bit:

Can the problem be:

• Redefined - Do you need to design a dedicated widget or service to fit the need? Does the customer know that we already offer a similar service, scaled for X (where X is some number of transactions/time, amount of user traffic, amount of records) and ready today? Examples might be authentication, mail transport, indexing/search functions, or publishing.
• Rearranged - Can we "start small?" Do we commit full resources to the solution if the outcome (traffic estimates, sales, customer uptake, etc) are unknown or sketchy in some way? Can some features go into a "version 2.0?"
• Regrouped - Given the answers to the two questions above, can we consolidate or change the groupings of the sub-problems to fit a known pattern? Is batch-processing sufficient or do we need real-time output?

At the end of this phase, you now have all of the definition and analysis necessary to begin idea development. You have eliminated duplications and are able to leverage prior work and existing systems. With this, you've reduced your problem size and hopefully complexity.

Past articles in this series:

1. Analysis - Breaking It Down
2. What Is It We're Doing Here? - Applying Engineering Principles To System Administration
3. Who Is It? - Defining Who Might Benefit

Technorati Tags: system administration

Over at Habiblog, David offers us some of his thoughts on process. I think in the extreme cases, David has a point. Being excessively tethered to process may reduce your competitive advantage or turn your business into Initech ("you got the memo about the cover sheets, right?") or it could be a frantic, back-on-its-heels business where you wear out your people and deliver inconsistently over the long-term.

There's a couple of points with which I take issue:

I've seen places where every step was scripted, every decision was pre-made, and every task was strictly documented. These places smelled of the mania of the leadership and the desperation of the employees. Creativity, innovation, insight- all were starved. Productivity is how you measure it, of course, but these places gave themselves high marks. Those places aren"t doing what you would call "well" right now.

I've not worked at the same places yet I have seen these symptoms, but they are symptoms rather than the disease. IBM, known for its straight-laced and engineer-driven culture, is doing well. Their products, Websphere, AIX, AS400, and the Thinkpad are revered in the industry. Each is well-built, featureful, and engineered to work like a draft horse. It's probably occasionally dry to work there, but we're in the business of enhancing shareholder value and the engineered solution with long-term planning maximizes shareholder value for IBM.

Of course, one success story doesn't prove anything, except that perhaps that orderliness and documentation do not by definition stem from mania or only flourish best in creative starvation.

I'm not suggesting that you can or need to design a website or application from the ground up every time, but instead do the engineering work on the platforms and precompile your decisions [Hogan, Limoncelli, 577] with solid processes at the outset. Every product that comes after that and deployed on that standard platform will benefit from a faster time-to-market and less proverbial reinvention of the wheel.

If you've engineered a platform, you'll know its features and bounds and how the application will perform. You move from a "let's try this and see how it works" model (which is fraught with guesswork and often a sinkhole for waste and opportunity costs) to "we'll deploy here with the right resources, beat the competition, and then we can go on to the next thing". Competitive advantage is not only getting that one product out the door ahead of your competitor, it's doing it time and again that takes you to the top. It takes courage to commit resources to something other than hitting the fastball coming right now across the plate, but it's an investment, with attendant risk and reward.

...standard procedures are put in place as a codification of the organization's known best practices

If the research I call out above is true, then standard procedures are like the neural pathways in our brains. The more often a certain route is exercised, the more ingrained it gets, representing our habits, mannerisms, and behaviors- our character. An organization's standard procedures, the research says, are its memory- and the organizationas character is constructed on top of them.

I don't disagree with this point except that it's not a guarantee that your codified process is the known best practice today or ever was at all. Someone may have developed a procedure or tool, or maybe the practice was great (or maybe just good enough) and it was institutionalized to the organization's benefit.

Here's a cautionary note to the new manager or the visionary in your group though: those procedures and neural pathways didn't get there by themselves. They came from somewhere.

For a dynamic organization to conquer new ground, those procedures have to be updated to grow and scale and be leveraged (buzzword!) You can't build servers the same way as you did in 1999. You can't deploy apps on platforms where there's no development support, year after year, and expect to stay competitive.

Someone has to run slightly ahead, looking for the pitfalls and tweaking the procedures or even looking for the leap ahead in process or engineering your next platform, simply so that it will be ready (remember the goal of being more strategic than tactical) when your organization needs it.

In closing, the people innovating in products that bring you revenue or mindshare need not be the same people innovating in process to build strong foundations, but your organization needs both.

Technorati Tags: system administration

A well-executed and thoughtful analysis of the problem at hand, provided by a knowledgeable set of professionals can be extremely useful to your project. Time spent now analyzing the parts of problem and devising a methodology for solving them will be handsomely repaid later in reduced re-work, altered and slipped schedules, and feature creep. All of those bad things mean one thing: money. Avoid or mitigate them and you'll save money.

How do we do that analysis?

The first step in the this phase should be the decomposition of the problem into sub-problems. You might use a tool like an outliner or Visio to begin to sort the sub-problems or build a hierarchy of items. A visual representation of the sub-problems and how the problems relate to others up the chain can be useful. After a number of these exercises, your company may have a set of analysis procedures which standardize this process. If not, it should; don't reinvent the wheel each time.

How do you know when you've broken the problem down far enough? To decompose a problem into useful sub-problems, someone familiar with the domain of the problem should be engaged.

Suppose my website needs a shopping cart for on-line retail sales. Someone knowledgeable in the field of e-commerce would know that email delivery of order confirmations and shipment status is a de facto requirement in this space. Knowing that, we'd add "email delivery" or perhaps, more broadly "customer communication" as a sub-problem. We'd then engage someone who is knowledgeable in the delivery of mail at our expected volume.

This is a vastly simplified example, but the customer may not have the depth of knowledge to even articulate that portion of the problem beyond "yeah, we need to send receipts to our customers". At this point, if the subproblem lives in one domain and can be solved as a unit, we've probably reached the bottom of this sub-problem and need no further decomposition. This is a somewhat tough thing to teach and comes from the experience in the domain.

As we document sub-problems, it is important to keep focus and to relate every sub-problem back to the main problem at hand. In our shopping cart example, it might be interesting or even technically sexy to explore delivery of customer communications via alternative methods such as SMS, for example. However, someone knowledgeable in industry standards and best practices for the problem domain can tell you this is not a common practice or expectation and is really only peripherally related to the problem at hand: processing the sales of widgets on your website.

Here you may encounter the familiar tension between "engineering" and the business or marketing groups pushing for new gee-whiz features, when the design costs for them may be prohibitively expensive and any real-world utility suspect.

The last pass over the sub-problems should be used to arrive at a rough timetable and sequence for attacking each subproblem. The sum of the time to complete the sub-problems, acknowledging that some may be done in parallel, while others have a serial nature, may be weighed against the deadlines or time-to-market (ah, there's a buzzword) goal.

What are we trying to get out of the exercise?

Remember also, at this point, we're only documenting the problems, not solving them or writing project plans to implement possible solutions.

At the end of this work, you should have the following:

• A coherent statement of the problem
• Forecast of metrics, applicable dates, and financial bounds for solving the problem
• A set of sub-problems, categorized in a useful way
• A set of sub-problems in manageable, solvable chunks
• Techniques that may be applied to solving each sub-problem
• Identification of personnel (or gaps in knowledge) in your organization who can solve each sub-problem
• A timeline and level-of-effort for solving the problem
• A plan for reporting on the progress of solving each sub-problem

Next time, we'll review the sub-problems in the Transformation phase, to help us make use of past solutions and patterns from similar problems, so that we can arrive at cost-effective and standard solutions.

Hit me with some comments, if you've got techniques or suggestions for boiling problems down in their basic components.

Past articles in this series:

1. What Is It We're Doing Here? - Applying Engineering Principles To System Administration
2. Who Is It? - Defining Who Might Benefit

Technorati Tags: system administration

Customize your very own "official seal" over at www.says-it.com

Technorati Tags: official seal

Before proceeding on to the next item in the engineering design process, I'm taking David's (of Habiblog) question about my definition of 'system administrator'.

Again, depending on whom you ask, the term 'system administrator' can apply to many skill-sets or job roles. There's quite a range in skills and responsibility, often loosely related to the size or industry of the installation. The guy working to keep the systems running at a small company of 40 people has to be the jack of all trades, especially if the company is not in IT. The guy supporting a manufacturer of children's clothing may not have the budget to spend deeply on 'enterprise-class' gear or have dedicated and redundant hosts. He'll run hosts longer and harder (because he has to), and in many cases, it will be gear that has names like 'Digital' or 'IBM System xx' stamped on it. He'll twiddle with EDI, keep an eye on the frac-T1, answer questions on Win95 (still!), and wonder if he can scrounge a box to try out Linux and Samba. This is where many of us started.

However, the guy (likely with a team) supporting yahoo.com or aol.com has a larger budget, technology at hand that improves his job, a more specialized focus, and a commitment to some goal, be it traffic numbers, availability, search query response times, etc. Few of us start here.

There's a continuum in between; folks who support desktop users at a university, the girl who runs the Beowulf cluster at Sandia, and the great-UnixHead, who lives only to rack, compile, and tune.

What I am getting at is the fact, and you see this in the dying throes of SAGE as it is extruded from USENIX and the fitful self-consciousness in LOPSA (which, by the way, hasn't yet made me System Administrator of the Week), is that this a profession that is relatively young and is looking to mature. The frantic pace of technological change and industry upheaval (boom and bust cycles), has left precious little time for the profession to arrive at a body of knowledge. But I do see the emergence of leaders in the field, technologists who can contribute their considerable energies and skills to advancing the profession.

So, in answer to the question, "what is your definition of a 'system administrator' [in this context]?", I have to place the most emphasis on the sysadmin's ability to decide. Does the sysadmin have the skills, the management backing, and the resources to make his work strategic and to improve (insert overused term 'architect') his installation by hewing to engineering practices? In summary, I'm thinking of the SAGE Level IV system administrator.

Appropriate Responsibilities

• Designs/implements complex local and wide-area networks of machines.
• Manages a large, complex site or network.
• Works under general direction from senior management.
• Establishes/recommends policies on system use and services.
• Provides technical lead and/or supervises system administrators, system programmers, or others of equivalent seniority.
• Has purchasing authority and responsibility for purchase justification.

Technorati Tags: system administration

Depending on whom you ask, the very nature of system administration varies. Is it art? Black art? A profession advanced through the apprenticeship model? Are we smithies? Many would argue, especially the virus fighter or self-healing system designer, that system administration is a science, with natural rules and reactions.

Others might suggest, and often do, that we're somehow engineers. I'd argue that most of us are closer to locomotive engineers than to the trained and licensed engineering professional, one who uses repeatable processes, measurement, and design; the hallmarks of the engineering profession.

In the next few articles, I'll be walking the border between the province of the engineer and the shire of the common system administrator. I'll be using some textbook definitions of engineering terms and applying them to the things we do in our daily work. I do this not as an indictment of my very own profession, but in hope of peeling back the layers of myth and guesswork so prevalent in designing and running large computer systems. We've made tremendous strides in some areas, so all is not lost.

Many of us know our systems and applications deeply, and like our children, we know when they'll misbehave. To me, though, that's not a repeatable or easily teachable relationship or one that serves us well in the engineering or business worlds, where certainties such as total cost of ownership and project completion dates are expected.

Why do this? I'm interested in advancing our cause and strengthening the differentiation in unskilled vs. skilled, between the carpenter and the guy that built Taipei 101. Rather than simply stoking the fires and twisting the throttle, I'd like to see us closer to being CAPCOM, coolly wearing the vest and shepherding the mission (our system) that was scoped well, designed correctly, and responds in known and predictable ways.

We can improve the use of tools and measurement and make decisions based on hypotheses confirmed with data, rather than gut-feelings. The outcome is bound to be better systems, ones certainly better in cost of ownership and efficiency. If we've designed systems correctly, with a nod to modularity and standards, they'll scale better and our time-to-market will improve. Much of this is self-apparent to many of you, I'm sure. It hasn't reached everyone yet.

By way of definition, I'm using the terms "system" and "computer system" to describe a set of computers, processes, and interconnections, typically in a large installation supporting large corporations or government agencies. These are places where volume and traffic require true efficiency and scalability. I'm not describing the world of the Electrical Engineer, designing the actual boards or integrated circuits, nor are we hoping to apply much thought to setting up a print server for the twenty-five person company. We're not interested in the heat generated by the CPU or the hard disk, except in the most macro way, i.e. "how do we minimize cost, be that in energy consumption or HVAC, with our implementation?"

Let's start off with the engineering design process. I'm using an engineering textbook, Engineering: An Introduction to a Creative Profession [Beakley, Evans, Keats, 1986]. The specific naming of the stages is not so important and perhaps will vary according to the problem or system at hand, but the process itself lends us structure and provides a known path to progress from Need to Project Plan.

Stages of Engineering Design

1. Identification of the problem
2. Analysis
3. Transformation
4. Idea Development
5. Modeling
6. Information Gathering
7. Experimentation
8. Synthesis
9. Evaluation and testing
10. Presentation of the solution

Phew! That's quite a list of things to do before you build that new mail server, right? Well, that mail service is the thing that needs the process, not your lowly server. The ground-up design of the system, following industry and de facto standards is the time and the place for the steps shown above. Let's delve into these a bit, shall we?

Identification

What is it that is needed? Seems simple, right? Requirements gathering and scoping are two of the major gotchas in this business. The marketing and business people come bearing grand, shiny ideas, marveling at their own cleverness, only to be ground down by the reality, the lack of support from legacy backend systems, and the endless questions from the 'engineers', who are now seen as a roadblock to be bypassed.

This is where we must perservere; to dig, to not take things at face value, to ask the same questions in different ways. You must not only ask what is needed, but what is not needed. In helping the customer to fully identify the problem, we should consider some of the following areas and constraints in this first phase.

• Can the purpose of the system be reduced to a single sentence or paragraph? Have we clarified to that level?
• What is the scope of the problem? How many users, customers, data sets, or records are affected?
• What are the interconnections, data flows, and processes that will feed our new system or allow it to feed other systems?
• Can we identify, even in rough terms, the availability of funding and management commitment to the project? Is the system part of our charter or corporate mission?
• What are our legal and ethical constraints for this system, its data, and its users?
• What is the expected outcome of this design exercise? Are we simply studying the feasability of solving the problem or actually moving to do that work?

Write these things down. The use of standardized forms or questions can help guide and document the discussions, but don't rely on them alone. Repeat back your understanding of the problem and constraints to the customer, letting him hear what you understand to be the problem. One other useful tactic in discussions of upgrading or replacing systems can be a review of the limitations of the old system.

With proper problem identification, you'll have formed an agreement with all concerned over what is actually to be fixed, designed, or upgraded, thereby reducing feature-creep or mismatched expectations later in the project, when refactoring or modification will be expensive and disruptive. Don't skimp at this stage.

Next time, we'll jump into the analysis stage for some thoughts on problem decomposition and application of known data to the problem that we've identified.

If you've got suggestions or maybe comments on how I've gotten it all wrong, hit the Comments link below and let's hear it.

Technorati Tags: system administration

This is some foo.