#Leadership #Remote

I have been in a distrubted team – as a leader or an individual contributor – for nearly a decade of my career. But like most people the pandemic made me feel like a prisoner in my own home. The world hit a pause button on nearly every aspect of life. Many, including myself, dove headfirst into the grind of work and spent more hours in front of the keyboard even if it wasn't productive.

The only rule of Monday coffee time is that we try to stay away from work topics because we want everyone to step back from their obligations and have a little bit of genuine human interaction. Quite surprisingly, we have never actually talked about our morning beverages of choice. Some of the more recent topics that have come up:

• Pandemic parenting
• Kansas City Chiefs
• Homemade wedding cakes
• Methods of smoking turkey/chicken
• Hiking and bicycling
• Home improvements
• Past travels

For those of you that have been considering carving some time out of your week to bond with your team we have found this to be an especially good way to begin the work week. Those of us that have been in a distributed work environment for awhile may say that this isn't new or original, but I think we can all appreicate the stress and difficulty over the past seven months. And for some of us it may continue well into the next year.

So try it out. If you don't have a topic in mind there's always the weather.

#Leadership #Management

Time Management

If you are lucky enough to have an executive assistant they can work with you to ensure that you are using your time as effectively as possible. But not everyone is that lucky so here are a few tips that will set you up right. Guard your time wisely; the less you waste the less personal time you'll be spending.

1. Ask for and review agendas for your meetings. Otherwise cancel it.

2. Try to schedule the bulk of your 1:1 meetings on the same day each week.

3. Use color coded calendar categories to manage your day at a glance.

4. Set aside blocks of time for administrative chores. Yes, that includes reviewing and answering emails.

5. Use the “out of office” meeting designation to block out personal time.

Equipment

Most people will receive their work equipment on the first day that they start and, unless you're in an extremely progressive organization, will rarely have a say in the make or model. Remember: you're the head tech guy so you can basically get whatever you want but don't abuse the privilege.

1. That standing desk is most likely a waste of money but a good desk chair isn't.

2. Invest in a good pair of wireless headphones. You're going to be using them.

3. One big, high-resolution monitor goes a lot further than two or three.

4. A laptop with one USB-C connection for charging, video and peripherals is pretty much panacea.

5. Use a wired microphone for important meetings. It really does sound better.

People

This is where you'll end up spending most of your time because people are the lifeblood of any organization. It is the truth that culture will eat strategy for breakfast, lunch and dinner; so its important to work well with your peers. But it doesn't stop there: be ready to go in front of customers and prospects.

1. Most problems aren't technical and won't be solved with technology.

2. You shouldn't be making technical decisions. If you are, that's a problem.

3. It is your job to convince the business and product teams that agile isn't magic.

4. Talk to your exiting employees. You'll always learn something.

5. Don't play politics with your team.

I do not think that any of these are unique to the type or size of the organization. There is only so much energy and time that you can input into the system. So spend your time wisely.

#Technology #Leadership #Storytime

Disaster recovery (DR) is serious business. When a catastrophe strikes you’re never completely prepared. At Bloomberg, as I imagine most large companies, we take care in designing our systems to be reslient. For us, this means highly available, uninterrupted service for all of our clients. In order to be able to sustain a complete datacenter outage we must be fully confident in our automated solutions for failover. A disaster recovery plan is no good if it hasn’t been tested and a disaster is the wrong time to execute that test.

Wikipedia says that there are seven tiers to disaster recovery but this gives a thousand foot view of how to plan for the technical portion of business continuity. From experience there are a few practical actions you and your team can take to keep your cool during a disaster.

Reboot your machines on a regular basis.

Years ago, any system administrator worth their salt laughed at those who counted their uptime in years. If you never reboot your machine how can you be sure that it will actually come back online in a usable state? A few thousand dollars worth of hardware is no good if you are not sure how to QC it before putting it in front of customer traffic. Spend the time and make a checklist of the services that you want to come online automatically after booting.

Work towards making your scalable systems ephemeral.

Scaling your application out horizontally does not immediately mean that the individual nodes are capable of spinning up and down without any major problems. This often means designing all of your services with high availability in mind. This is much more than just sticking HAProxy in front of all your application endpoints. You need to be sure that your applications and services will fail gracefully, or better yet, swing to their alternate slave.

Don’t put all your eggs in one basket.

When deploying physical hardware in a datacenter you wouldn’t put it all on the same power circuit. Unfortunately there is often no longer physical access to hardware. All provisioning is performed over the wire. How much sense would it make for all of your company’s virtual machines all being tenants on the same physical hardware? Not much at all! If possible ask your service provider if their compute services are rack aware.

Package complex steps into shell scripts.

If you cannot automatically orchestrate a failover of a machine take the time and define a clear and concise shell script to perform the work. It is much easier in documentation to point to a single bash script which illustrates the more complex procedures. In the middle of the night your operations staff may not have any context in the problem that they’re trying to solve. Rather than calling an all hands on deck they can run through a list of procedures.

Clearly document manual procedures. The analog way.

This brings me to the final item on the checklist. There’s a lot of companies out there that offer digital solutions for organizing your team and their work. Did you know that the popular Heroku Platform as a Service is built on top of Amazon Web Services? If that fancy SaaS application which contains all of your documents goes offline you’re now unable to bring your systems back online manually. Spend the time and physically document procedures and distribute it to all of your team members. Especially your operations staff.

These are only but a few action points that you must cover while planning for business continuity. As I mentioned above this is serious business. You should begin first with hiring a proper operations staff, and remember that these should not be the same people that are your developers. Writing code and managing infrastructure are two completely separate beasts. But your engineers should start by designing their systems to sustain failure at any level in your stack.

I hope that these steps help you while planning for failure. If you find this useful give me a shout and let me know!