Using the Method of Procedure (MOP) for Effective Network Change Control

Uncontrolled network changes are dangerous. Actions taken on a live network have consequences, and they are not always desirable. The use of a method of procedure (MOP) can greatly reduce risks and improve efficiency in the management of a network. Without proper change control, your enterprise can suffer irreparable losses. Anyone responsible for a network should give full attention to change control. It is at the heart of network management. (For a broader look at IT management, see The Basics of IT Planning.)

The Change Control Process

You may have seen how flow diagrams are used to illustrate the logic, decisions and actions of a process. A visual representation of a process is helpful to show how a change in the IT environment is handled. It is common for large network carriers to use such graphic displays to capture and communicate every detail of potential “moves and changes” in the network.

While the flow chart may be prominently displayed in a network operations center (NOC), usually the change control process is also elaborated in a fully developed change control document. Routine changes are normally defined in vendor equipment manuals. Whatever the form, good documentation is critical to ensuring successful change control.

It is important to have an adequate system for managing specific changes. One basic element of change control is the change request (CR). This is the document used to request the change. It may take the form of an individual email, a formal document or a ticket as part of a ticketing system.

In some projects or operations, managers will assign individuals to a change control team. Its members may include subject matter experts (SMEs) from different departments, brought in to consult on various technical issues. In other organizations, the change control process may be managed by dedicated departments. Every CR should undergo pertinent technical and management review and approval. Getting the green light for a CR depends on a well-defined procedure.

Method of Procedure (MOP)

The following sections describe a useful tool in the execution of the CR. There are, of course, different approaches to defining procedures and managing changes. The concept discussed here is drawn from years of experience in operations and maintenance (O&M) duties while handling data networks in the telecommunications industry (with a special thanks to the excellent work of colleagues in my projects for Nortel Networks). Let me present here a tool for change control called the “method of procedure.”

A “method,” simply defined, is a systematic way of doing things. It is an ordered arrangement of steps to achieve a specific task. In other words, given a specific set of circumstances, take these specific actions. The word “procedure” also refers to an established way of doing things. Using both terms, method and procedure, in one phrase seems almost redundant, as if someone is really trying to get the point across. But the name is fitting. If you are working on a live network that provides services to many thousands of users, including multi-billion-dollar companies, you’d better be careful. Follow the method of procedure exactly as written – or be prepared to suffer the consequences!

Components of a MOP

The design of the MOP will vary according to the needs of the organization, but we can identify components that are commonly used. A web search for MOP templates will also give you ideas. It’s a good idea to put together your own MOP template that can be adapted for individual change requests or standard procedures. It may take considerable time and thought to finalize a workable template of your own.

Now let’s take a closer look at the components. These can be grouped as you please, but I am putting them in sections below as a general way to describe them.

Title, Author, Date, Unique Identifier/Change Request Number, Version Control

Of course, the title and author(s) remain the same in all versions of the document, but the unique identifier and version control may change with subsequent drafts. The unique identifier may be a document number that is part of an organization’s document control system, or it may be a change request number. This designation may change if there is a new version of the MOP. The version control section is often laid out as a table that shows changes to the document, including the date of the revision and a note about what was edited.

Purpose, Description, Risk Assessment/Impact, Cost and Benefits

These parts of the MOP clearly lay out for the reader the justification and potential consequences of the change. What are the circumstances that require the change? Which internal or external customers might be impacted? What are the financial concerns? These questions should be answered before making any changes to a live network.

Maintenance Window, Notifications

For critical network elements, the time of the change is important. Network providers often set regular weekly maintenance windows on the weekend at times when the fewest users are affected. This may be on graveyard shift on Saturday night/Sunday morning from 11:00 p.m. to 7:00 a.m. Or it could be at another time during the week as agreed upon by the customers affected. Maintenance windows are often set for four or eight hours. If the scheduled work cannot be completed during this time, it should be stopped, and the changes should be rolled back (see below).

It is important to notify anyone who may be affected by the change. This could be external customers, or it may be internal departments who might notice the network activity. Information about maintenance windows or network changes is often included in regular emails, in a standing change control database or within the standard ticketing system.

Prerequisites, Special Tools or Parts, Safety Requirements, Resource Requirements

These MOP sections will answer more important questions. What must be done, or what is the necessary situation, before the start of the change? When a technician goes to site, what tools or parts are required to make the change? How do we ensure safety? What personnel, equipment or other resources are needed?

Procedures

Now we are at the heart of the MOP. This is the most important part. And even more important than having written procedures, the technical professionals doing the change must follow them! In my experience, we used an important “technical” acronym: RTFM. This means, loosely, “read the manual.” And the person(s) enacting the change should make sure that they have the most recent version of the MOP.

These essential action points should be clear, logical, sequential and comprehensive. To get there, it helps when procedures are tested in the laboratory. The expected results should be included, perhaps in the form of screen shots, and visual information like pictures and network diagrams are often helpful. (To learn more about networking, see The 4 Most Confusing Concepts in Networking Explained.)

Rollback, Approvals

What happens when a change procedure fails? It should be possible to revert the network to its initial state, or at least to a safe and stable state. The rollback section should include clear step-by-step procedures to make this happen.

The finished MOP should go through all the necessary channels. Technical review and management approval are necessary to make sure that all bases are covered. The MOP is not complete without these required signatures.

Conclusion

No planned procedure is foolproof. The greatest danger is when an overconfident engineer feels that he knows everything. On my teams, we always printed the procedures and followed them step-by-step – even if we had done them many times before. One fellow engineer failed to do a backup – as per the MOP – and cost the company a huge network disruption and hundreds of thousands of British pounds.

Whatever form you choose for planning and documenting network changes, be sure that each change is clearly defined, reviewed, and approved before taking action. The method of procedure can be an effective tool for doing just that.