Network Operation Center (NOC) Best Practices – Part 2: Knowledge & skills

This is the second of our 3 parts blog series discussing Network Operation Centers (NOC’s) best practices. The first post was dedicated to NOC tools. This part is dedicated to knowledge and skills. By ‘knowledge and skills’ we do not mean the obvious technical knowledge, network ‘know-how’ your team members must hold in order to run day-to-day operations, but rather –

How you can ensure your team’s skills are used to their best potential, and how to keep those skills up to date over time.

Clearly define roles

Definition of roles may vary between data centers and will depend on team size, the IT environment and tasks. Still, there should be a clear distinction between the roles and responsibilities of operators vs. shift supervisors in the NOC.

Why does it matter?

Mainly matters because of Decisions making. Without clearly defined roles and responsibilities, a disagreement between operators may lead to late decisions and actions, or to no decisions taken at all. This may affect customers, critical business services, and urgent requests during off hours.
It should be clearly defined, therefore, that a shift manager makes the final decisions.

Tasks division

Another potential problem caused by a lack of role definition is the division of tasks between operators and the shift leader.

A shift manager should be responsible for: prioritizing tasks, assigning work to operators based on their skills, verifying that tickets are opened properly and that relevant personnel are notified when required, escalating problems, communicating with management during important NOC events, sending notifications to the entire organization, preparing reports, and making critical decisions that impact many services, such as shutting down the data center in case of an emergency.

Operators, on the other hand, are responsible for handling the technical aspect of incidents – either independently or by escalating to another team member with the required skills. Operators are also responsible for following up and keep tickets up to date.

While it might sound as if operators lack independence and responsibility, this is not the case. When faced with technical challenges, operators’ input and skills are probably the most critical for resolution and smooth NOC operation. Operators provide additional insights into problems, and can provide creative solutions when the standard procedures fail to work.

Invest in orientation program for new NOC employees

How often have you started a new job, without receiving any orientation, mentoring or guided training?

Failing to provide proper training to new NOC operators always has consequences. A new NOC operator may not know where to find a procedure or how to execute it; be confused about who should handle a task – the NOC, service desk, or higher level of support; or in a more severe case, take a decision that causes equipment damage or results in downtime of critical business services.

Therefore, an extensive training program should be put in place for new NOC employees. This is definitely a challenge, considering the lack of resources, particularly in small NOCs. Ideally, such a program would consist of one week of classroom training followed by three weeks of hands-on training under the supervision of a designated trainer.

A new employee should only be trained by an experienced member of the NOC, preferably a shift leader. The trainer should be released from all duties during the entire period of the training – in order to ensure that the training does not gradually fade between all the urgent shift tasks.

The training program should be updated on an ongoing basis, and should include topics such as required users and permissions, technical knowledge, known problems, troubleshooting, teams and important contacts.

Communication and collaboration

Within your Organization

Establishing a solid communication flow between NOC members and other IT teams has many advantages. It propels professional growth, provides opportunities for advancement in the organization, and makes it easier to approach other teams when requiring assistance. But most importantly – it allows NOC personnel to see the larger picture. NOC members that are aware of projects, services and customers’ needs, simply provide better service.

A designated member of the NOC should attend weekly change management meetings. That person should communicate any issues or upcoming activities, such as planned downtime, to the rest of the team.

Define NOC members as focal points for important IT areas, such as NT, UNIX, Network, or a specific project is another good practice. These members should attend the meetings of the relevant teams, deliver new information and knowledge to the rest of the NOC, and handle specific professional challenges.

Within NOC Team

Another important form of communication is within the NOC team itself. There are clear advantages to having a strong connection and collaboration between NOC team members. Members are more willing to help each other, information is shared more easily, and the general atmosphere encourages collaboration when addressing problems, as opposed to an individualized approach.
Team communication is a challenge when the NOC team is geographically spread out or located in different countries. Because cultural and language differences can cause confusion and misunderstandings, spending efforts on building team communication and collaboration are even more critical.