I remember doing annual power shutdowns of the data center and IDFs where I worked years ago. I remember doing OS upgrades on critical network devices. I also remember the chaos and the amount of people that needed to be on a bridge validating “everything looked okay” when the devices came back online. Was everything always okay? Hardly, but it wasn’t until business started on the following Monday, those one off issues were uncovered and fixed. If there were tools verifying routing tables, interfaces, and general state of all IT systems (pre and post upgrade/reboot), it would have made everyone a little saner.
The interesting thing about automation --- it is in fact risky automating configuration changes. Some may call it scary and some call it giving a monkey a loaded gun. Do you REALLY want to do that? Is there a safety switch included?
Note: please always make sure you test and QA any new tool/product. Just figured I’d state the obvious here.
Rather than just writing about all of this, I felt it was best to show an example of what I’ve been describing here (see demo/video below). With more demos out there, the hope is there will be more “ah ha” moments to see the benefits that various forms of automation can offer organizations regardless of vertical, size, and scale.
The video below is a demo of some new custom Ansible modules I created that automate the validation of OSPF adjacencies. Data is automatically extracted from each router and then compared to determine if the routers are configured properly for OSPF. You’ll see towards the end of the video I modify one of the interface’s MTU and re-run the Ansible playbook and the playbook automatically re-configures the MTU to what it should be. So, this playbook actually auto-validates, but also auto-configures. The re-configuration part is purely optional, but thought it would be good to show what can be achieved with some of the DevOps tools out there such as Ansible.
While usually overlooked, validating and learning are SIGNIFICANT pieces to the puzzle to create the ultimate feedback loop in terms of IT optimization. As always, would love to hear feedback and general commentary in the comments section below.
P.S. Please ensure you don’t have any monkey’s operating your company’s automation platforms or IT systems.