Mryntu

Hi ladies and gentlemen. I have been asked to come up with a plan for upgrading old 3750G switches to 3850Us. The manager asked me to present a plan of what I would do if I were the network engineer in charge of this change. This is a for a medical facility and I'm wondering what do I need to be certain to include in the plan. The old switches on the left side and the plan for the new ones on the right side. What is the specific configuration on the switch so that some devices would stay online as much as possible? Can I just reuse old configs from the old devices? what about the PortChannel? can I use the same ones?

Below are the circumstances that were presented to me by the manager:

1. The downtime is from 6:00p to 9:00p.




2. The current IOS standard for a 3850U stack is 03.06.06E.




3. All of the network gear is currently on site and still in boxes. Inventory has not been done.




4. The default VLAN in the closet is VL3. 80% of the devices are on VL3.




5. There are other devices on other VLANS, VL53, VL68 (PACS), VL42 (Utility), VL501 (APs), 7XX (wired phones).




6. There are some devices on VL68 that need to stay online as much as possible.




7. There will be a few other devices that you have to identify and make sure they stay online as much as possible.




8. There is one mission critical department that uses this closet, ICU

I will start by assuming that there is enough room and power in the IDFs to rack the new switches along with the old ones. If this is not the case, STOP RIGHT HERE. Three hours is waaaay too short a change window to remove and replace the switches. You will have to come up with a plan over multiple change windows to move one stack per window. Even then, you need to be sure you know exactly what is plugged in where.

You should configure the new switches "on the bench," meaning they should be fully configured and tested before you start moving cables.

To answer your specific questions:

What is the specific configuration on the switch so that some devices
would stay online as much as possible?

To minimize downtime, you need to have the old and new switches trunked together, so VLANs will exist on both old and new switches at the same time. Then, you can move cables one at a time from old to new.

Can I just reuse old configs from the old devices? what about the
PortChannel? can I use the same ones?

For the most part, the VLAN and interface configurations can probably stay the same, but I notice you have different number of switches in the old and new stacks, so some port numbers will change. There may be a few other things that may be hardware specific. If you post your configs, we can give you more detailed answer.

In addition to the excellent answer about the switches themselves ...

Testing

Consider:

• Exactly what tests will qualify the new setup as satisfactory?


• Can you run the tests continuously?


• Can you do a practice run? (Perhaps without actually replugging anything)


• How long will it take you to reverse to the old equipment?

As examples, perhaps you can automate ping tests to your entire set of devices, and have that running continuously. Then, as each device moves from old to new, a brief failure will show, followed by recovery. If seeing that happen is just looking over to a screen, that's much quicker than having to sit and manually perform some tests.

What happens if it fails?

You've said the network is a hospital with an ICU.

Suppose it fails half-way through, for any reason whatsoever, the consequences for an ICU are very possibly life-threatening.

Things I'd consider:

• Do I have backup staff who can reverse to a known good state?


• Do I have a clear cut-off time to commit to the new equipment?

The reasons for the failure can be as unpredictable as network staff family or medical emergency, equipment theft, terrorist event -- all of which have happened to projects of mine. Never mind surprise equipment bugs or mistaken parameters and project underestimation.

Ground control

• Do I have a "ground control" person authourised to abort the cutover if they see it going wrong?

Nobody ever plans to get into the situation where the networking staff are over-tired and over-stressed and thing "just a few more minutes" will fix it. But it happens with surprising frequency. One way to avoid the threat of overwhelmed technical staff is to have a "ground control" person who knows the schedule and knows how and when to order a reverse to the known-good state. I put in measures like this on high-stress projects as mental health protection after seeing over-committed staff work until they dropped -- thankfully only large commercial and never medical projects. If they do that and fall asleep without proven project completion on mission critical projects it's awful costly. The more important the project, the more likely people will push themselves hard if it goes wrong, and many are unaware of their own endurance limits.