#Barbecue #Cooking #Summer

Last week, after spending weeks teetering back and forth about purchasing the Pit Barrel Junior Cooker/Smoker, I finally pulled the trigger on it. Over the course of the pandemic I have been poking and prodding some folks that are much more knowledgeable than me.

A whole chicken, smoked with applewood, rubbed with olive oil and Pit Barrel's All Purpose Rub.

A big motivation for me for purchasing this cooker/smoker was the fact that it was charcoal. At the new house, we decided to go with a propane grill out of ease, but there was a big part of me that wasn't happy with some of the results. You aren't getting an applewood smoked chicken on that bad boy.

The chicken hanging vertically in the smoker with the Turkey Hanger accessory.

My next experiment will be a smoked turkey and some ribs. If all goes well this may be the new Thanksgiving tradition because it is a helluva lot easier than cooking and basting the bird in the oven. It is 7am when I am writing this post and I am getting hungry just looking at that chicken. I am going to go raid the leftovers and have a few bites. After this pandemic is over, you're welcome to stop by if you provide the chicken and beer.

#Leadership #Remote

I have been in a distrubted team – as a leader or an individual contributor – for nearly a decade of my career. But like most people the pandemic made me feel like a prisoner in my own home. The world hit a pause button on nearly every aspect of life. Many, including myself, dove headfirst into the grind of work and spent more hours in front of the keyboard even if it wasn't productive.

The only rule of Monday coffee time is that we try to stay away from work topics because we want everyone to step back from their obligations and have a little bit of genuine human interaction. Quite surprisingly, we have never actually talked about our morning beverages of choice. Some of the more recent topics that have come up:

• Pandemic parenting
• Kansas City Chiefs
• Homemade wedding cakes
• Methods of smoking turkey/chicken
• Hiking and bicycling
• Home improvements
• Past travels

For those of you that have been considering carving some time out of your week to bond with your team we have found this to be an especially good way to begin the work week. Those of us that have been in a distributed work environment for awhile may say that this isn't new or original, but I think we can all appreicate the stress and difficulty over the past seven months. And for some of us it may continue well into the next year.

So try it out. If you don't have a topic in mind there's always the weather.

#Technology #Blogging #HowTo

Now, I am all for ensuring that content creators are paid, but after having a “taste” of the platform as it was I didn't really see myself wanting to contribute to it. That led me to take a look at the Ghost platform. This post isn't really about why I chose Ghost but a major reason was because of how dead simple it is to self-host.

Before we get into the nerdy technical bits it is important to note that rather than self-hosting on Digital Ocean (or some other cloud provider) you can pay a yearly fee to have a managed version with Ghost Pro. It was my decision to not go down that road because I want the flexibility to modify the source code myself. It also happens to be quite a bit cheaper with only minimal risk.

Domain Purchase

There are a countless number of vendors that offer the ability to purchase a domain name. In this example, I have used Google Domains, but you can use any service that fits your needs. In some cases, certain Top Level Domains (TLD) are only offered from specific vendors, e.g. the .dev TLD from Google, which obviously will force your purchase one way or another. But for the common extensions you can use any registrar.

The domain for this blog was purchased using Google Domains.

Most modern registrars offer an initial configuration for DNS nameservers and a user interface to manage records. We'll be adding an A record after we set up the Ghost instance in Digital Ocean.

Set Up Ghost with Digital Ocean

Digital Ocean's Marketplace offers 1-Click installation of Ghost and many other applications.

Digital Ocean offers an excellent Marketplace that has 1-click installations for many different types of software. The single click creation of a droplet running Ghost is foolproof. This will bring you to the instance creation page that has a whole lot of toggles, but my suggestions for setting up a simple installation of Ghost:

• Set the hostname to the domain name you purchased for your blog.
• $10/month instance with 2GB ram and 1 CPU.
• Select a region nearest to your core audience.
• Tag your instance with blog for quick searching.
• Add instance backups for $2/month.

After you click the Create Droplet button and the instance finishes booting you'll see the public IPV4 address. This is what you're going to use to configure the A record in your registrar's domain configuration.

Configure Domain

The data for the A record is the IPV4 address of the new Digital Ocean instance.

Now that your instance is available the final external step is to configure the DNS A record for the domain. Most people are going to want to add two records:

• The @ A record, e.g. https://jbellone.blog/
• The www CNAME record, e.g. https://www.jbellone.blog/

The data component on the A record is the IPV4 address of your Digital Ocean droplet and the data component on the CNAME record would be set to @. But if you're using Google Domains, you can leverage smart forwarding and forego the second record.

Google Domains gives you the option of setting up smart forwarding.

The domain forwarding option allows the most flexibility and if your registrar offers it you should definitely take advantage. After the records are set up it is time to follow the Getting Started Guide from the Digital Ocean marketplace. This will walk you through the initial set up of the instance.

Your First Post

If you've made it to this point you're ready to start configuring your Ghost installation. My suggestion would be to walk through the Settings and set up all of the parameters for your blog, e.g. title, description and theme. There are also a whole set of integrations that you can leverage with social accounts such as Twitter to automate the posting.

If you run into any trouble feel free to reach out to me on Blueskyor send me an e-mail.

#Leadership #Management

Time Management

If you are lucky enough to have an executive assistant they can work with you to ensure that you are using your time as effectively as possible. But not everyone is that lucky so here are a few tips that will set you up right. Guard your time wisely; the less you waste the less personal time you'll be spending.

1. Ask for and review agendas for your meetings. Otherwise cancel it.

2. Try to schedule the bulk of your 1:1 meetings on the same day each week.

3. Use color coded calendar categories to manage your day at a glance.

4. Set aside blocks of time for administrative chores. Yes, that includes reviewing and answering emails.

5. Use the “out of office” meeting designation to block out personal time.

Equipment

Most people will receive their work equipment on the first day that they start and, unless you're in an extremely progressive organization, will rarely have a say in the make or model. Remember: you're the head tech guy so you can basically get whatever you want but don't abuse the privilege.

1. That standing desk is most likely a waste of money but a good desk chair isn't.

2. Invest in a good pair of wireless headphones. You're going to be using them.

3. One big, high-resolution monitor goes a lot further than two or three.

4. A laptop with one USB-C connection for charging, video and peripherals is pretty much panacea.

5. Use a wired microphone for important meetings. It really does sound better.

People

This is where you'll end up spending most of your time because people are the lifeblood of any organization. It is the truth that culture will eat strategy for breakfast, lunch and dinner; so its important to work well with your peers. But it doesn't stop there: be ready to go in front of customers and prospects.

1. Most problems aren't technical and won't be solved with technology.

2. You shouldn't be making technical decisions. If you are, that's a problem.

3. It is your job to convince the business and product teams that agile isn't magic.

4. Talk to your exiting employees. You'll always learn something.

5. Don't play politics with your team.

I do not think that any of these are unique to the type or size of the organization. There is only so much energy and time that you can input into the system. So spend your time wisely.

#Technology #HowTo #DevOps

Requirements

Our immediate need was to provision a Chef Server for our build infrastructure that we would be distributing to about twenty teams. But in addition to this we have brought in Chef to train several classes of our developers. After the classes concluded there were many requests for individual Chef Server tenancies for testing.

Using Chef Server within our infrastructure was an afterthought because most of our virtual machines had a very short lifespan. Since we could have several dozen development teams wanting their own Chef Server deployments we needed a solution to not only easily deploy a Chef Server, but to hydrate it with the data necessary for each OpenStack tenancy. This included both user and client authentication information.

After a bit of back and forth I was able to isolate the requirements for our Chef Server infrastructure:

1. It must use the Chef Server community cookbook and the Omnibus packages provided by Chef;

2. It should be quick and easy to deploy a new instance because we may be doing so for several dozen tenancies in OpenStack;

3. All user and client keys should be generated locally and the server should be hydrated during the initial convergence run;

Playing With Metal

I was first introduced to Chef Metal at this past year’s ChefConf where John Keiser gave an excellent talk. My team has a lot of experience using Vagrant, but it made much more sense to me to include a metal recipe with our Chef infrastructure cookbook. It could serve as both a teaching aide, and would also make our internal cookbooks much more versatile. Throughout the discovery process I found the community Chef Server Populator cookbook, ended up completely rewriting it to use Chef resources to modify the Chef Server, and ultimately used Chef Zero to bootstrap everything.

Using Chef Metal I was able to easily distribute a recipe with our Chef infrastructure cookbook that would provision a Chef Server very easily using the machine resource. The recipe that I began with looked very similar to what is below. It first laid down changes to the base operating system using our base cookbook, and simply used the community cookbook to install the Chef Server from the Omnibus packages. It was very straight forward.

Unfortunately I still needed to hydrate the local data into the newly minted Chef Server. Luckily for me the cats over at Heavy Water Operations have solved this problem and provided the Chef Server Populator cookbook. There were only a few minor modifications that I needed to make to support client validation keys. But other than that everything seemed to be smooth sailing.

Chef Zero

At first, with only a few minor changes, the Chef Server Populator cookbook seemed to be the answer to my problems. I had a few cases that were not covered, but these were all pretty simple extensions. It was natural and made sense to convert our Chef infrastructure cookbook into a Chef repository and use Chef Zero as an in-memory Chef Server during the initial provisioning.

That worked out quite well at first. I was able to cobble together a simple Rake task to generate and write out the private and public keys to JSON data bag items. My metal recipe started to look a lot more complex than I had initially intended, and it didn’t seem as clean as I thought it could be. As I was reading through the code for the Chef Metal Fog driver a light bulb went off in my head: I could use the Cheffish gem Chef resources and build the data on the fly!

Chef within Chef

I ended up completely rewriting the Chef Server Populator cookbook. My fork now used the Chef resources provided by the Cheffish gem to read from data bag items and create the user and client records against the Chef Server node. The beauty of this approach is that I no longer had to execute clunky knife commands since in Chef 11 erchef exposed an API to pass both user and client public keys. The cookbook is now much more clean and simple to grok.

The code for creating the user and client keys was also able to use the same Chef resources from Cheffish. Instead of operating against our newly provisioned Chef Server it executed against the in-memory server provided by Chef Zero.

Using keys which are generated inside of the metal recipe is not idempotent. There were two ways that I went about solving this problem. This first was merely reading in the private keys off disk and delegating the generation to some external process. The second would be to use the Chef resource to generate private keys and allow it to manage if they needed to be updated or not. Ultimately it worked out that if the keys were generated outside of the metal recipe it was much more straightforward. Here is a full example where keys for clients were generated outside, but user keys are generated on the fly:

Tying The Bow

The final result was a complete cookbook refactor which uses Chef resources to hydrate a newly provisioned Chef Server. The default recipe uses node attributes and creates both client and user resources on the new server. A data bag recipe uses a Chef data bag search to find items and applies them to the node attributes prior to including the default recipe.

Since Chef Zero provides a fully functioning in-memory Chef Server we are able to use Chef resources to add data bag items with locally generated public keys. When using the Chef Metal recipe this data is now made accessible to the data bag search, and thus it gives us an elegant way to hydrate a Chef Server. I have provided an example of this inside of my fork of the cookbook which illustrates all of the steps above.

As I mentioned at the top of this post I originally started work on this solution to allow us to quickly deploy our build infrastructure to multiple tenancies within our private cloud. In a future post I will explain how my team was able to create an easily deployable continuous integration framework using the community Jenkins recipe.

If you are reading this post and are interested in joining our growing group of Chef technologists at Bloomberg feel free to reach out to me. We are hiring!

#Technology #Leadership #Storytime

Disaster recovery (DR) is serious business. When a catastrophe strikes you’re never completely prepared. At Bloomberg, as I imagine most large companies, we take care in designing our systems to be reslient. For us, this means highly available, uninterrupted service for all of our clients. In order to be able to sustain a complete datacenter outage we must be fully confident in our automated solutions for failover. A disaster recovery plan is no good if it hasn’t been tested and a disaster is the wrong time to execute that test.

Wikipedia says that there are seven tiers to disaster recovery but this gives a thousand foot view of how to plan for the technical portion of business continuity. From experience there are a few practical actions you and your team can take to keep your cool during a disaster.

Reboot your machines on a regular basis.

Years ago, any system administrator worth their salt laughed at those who counted their uptime in years. If you never reboot your machine how can you be sure that it will actually come back online in a usable state? A few thousand dollars worth of hardware is no good if you are not sure how to QC it before putting it in front of customer traffic. Spend the time and make a checklist of the services that you want to come online automatically after booting.

Work towards making your scalable systems ephemeral.

Scaling your application out horizontally does not immediately mean that the individual nodes are capable of spinning up and down without any major problems. This often means designing all of your services with high availability in mind. This is much more than just sticking HAProxy in front of all your application endpoints. You need to be sure that your applications and services will fail gracefully, or better yet, swing to their alternate slave.

Don’t put all your eggs in one basket.

When deploying physical hardware in a datacenter you wouldn’t put it all on the same power circuit. Unfortunately there is often no longer physical access to hardware. All provisioning is performed over the wire. How much sense would it make for all of your company’s virtual machines all being tenants on the same physical hardware? Not much at all! If possible ask your service provider if their compute services are rack aware.

Package complex steps into shell scripts.

If you cannot automatically orchestrate a failover of a machine take the time and define a clear and concise shell script to perform the work. It is much easier in documentation to point to a single bash script which illustrates the more complex procedures. In the middle of the night your operations staff may not have any context in the problem that they’re trying to solve. Rather than calling an all hands on deck they can run through a list of procedures.

Clearly document manual procedures. The analog way.

This brings me to the final item on the checklist. There’s a lot of companies out there that offer digital solutions for organizing your team and their work. Did you know that the popular Heroku Platform as a Service is built on top of Amazon Web Services? If that fancy SaaS application which contains all of your documents goes offline you’re now unable to bring your systems back online manually. Spend the time and physically document procedures and distribute it to all of your team members. Especially your operations staff.

These are only but a few action points that you must cover while planning for business continuity. As I mentioned above this is serious business. You should begin first with hiring a proper operations staff, and remember that these should not be the same people that are your developers. Writing code and managing infrastructure are two completely separate beasts. But your engineers should start by designing their systems to sustain failure at any level in your stack.

I hope that these steps help you while planning for failure. If you find this useful give me a shout and let me know!