This post is a creation of me wanting to know more about precisely how anchors work. There are a lot of rules of thumb on how much rode you should lay out but where did they come from? How do you know when to adjust it? What size anchor should I choose and why?

All these questions and more, I will answer based on what little I’ve been able to figure out. Let’s start with basics first.

What is an anchor, anchor rode, and scope?

An anchor is at its heart is something designed to secure a boat to the seabed so it doesn’t drift when pushed by wind, current, or tide. While it could be anything, like a big rock, in reality, its made of metal and has a shape that make it easy to secure itself to the bottom. They come in a wide variety of shapes but fundamentally, the composition of the seabed determines the most advantageous shape of anchor to use, most of which we won’t get into here.

Rode is whatever connects your boat to the anchor. This is typically rope, chain, or a combination of rope and chain. Both have different properties which gives them distinct pros and cons, some of which we’ll get into here.

Scope is the ratio of rode length to rode height, as measured from where it connects to the anchor (the anchor shank), to where it physically attaches to the boat. We’ll get into this in more detail in a moment.

Why does a boat drift?

This may seem simple but lets cover it anyway. A boat with no external forces acting on it doesn’t drift. It sits in place, motionless. In the real world, the water and wind are constantly acting on the boat which makes it drift.

Examples of these forces are waves, swells, currents, tide, and wind. All of these things will push a boat around to one degree or another. In calm situations, the only forces significant enough to worry about are currents, tide, and wind. If it’s not calm, waves and swells can impart significant force as well. If you’re interested in the difference between waves and swells here’s a simple, short explanation. If you don’t care, that’s fine, it’s not really relevant to the rest of the discussion. If you have trouble with his accent just mute the audio and read the notes.

The goal of an anchor, then, is to resist all of these forces and prevent the boat from moving.

The importance of scope

When anchoring, scope is one of the primary factors in determining how well your anchor will hold and thus keep your boat from drifting. The American Sailing Association (ASA), one of the largest organizations which promotes safe sailing standards and practices for non-professional sailing enthusiasts, recommends a 7:1 scope when anchoring overnight. As an example, if you’re anchoring in 7 feet of water and the rode is secured to your bow which is 3 feet above the water, you would lay out 70 feet of rode, (7+3) * 7 = 70.

The ASA also says that if you’re anchoring for a short time in calm weather, and aware of your surroundings, you can use a 5:1 scope. Say you’re anchoring to eat lunch for an hour or two and want to enjoy the calm water.

Including the deck height in the calculation is required. Lets say you anchored in 6 feet of water and put out 42 feet of rode believing that you were using a 7:1 scope. If your anchor attaches at your bow, which happens to be 4 feet higher than the waterline of the boat, your actual scope is 3.5:1.

The primary reason that scope is so important to anchoring is that even when an anchor is properly set, with the anchor flukes (the “arms” of the anchor that dig into the seabed) well secured, if you lift straight up from above the anchor, it’s easy to move it. This is exactly why the windlass doesn’t struggle lifting the anchor off the bottom unless something is wrong.

Paying out enough scope ensures that when the boat is pushed by the wind the force delivered to the anchor only pulls on the anchor horizontally, resulting in the anchor either not moving at all or (hopefully) digging the flukes deeper into the seabed (or lakebed or riverbed) to stop the movement.

This outlines one of the key pros of an all chain anchor rode. It’s heavy and that weight keeps the rode on the bottom contributing to only pulling against the anchor horizontally. More on this in a minute.

Why are 7:1 and 5:1 recommended scopes?

The answer to that question is what led me down the rabbit hole. What’s so special about 7:1? Will this hold in all kinds of weather? Is there a better scope in general? In specific conditions? Does the type or size of the anchor alter the scope?

The short answer is, I honestly couldn’t find a reason that 7:1 is the recommended scope other than it seems to work for most people most of the time. I couldn’t find any reason that 7:1 was picked vs 8:1 or 6.5:1.

This led me to wonder, how do I calculate the forces acting on the boat? If I know that, I at least know the minimum amount of force the anchor should hold against. Once I know that, maybe I can understand how different anchor sizes and types, and scopes, affect holding power.

How anchors work in slightly more detail

For the rest of this conversation I’m assuming the use of an all chain rode.

The weigh of an all chain anchor rode keeps the shank of the anchor on the bottom and ensures that the pull against the anchor is horizontal, but this is only true if there is enough weight to overcome the forces of wind, current, waves, etc., that are pulling it from above. This is why you need to pay attention scope, and thus the amount of rode you’ve played out.

It turns out that there’s a term for the way the rode hangs between your boat and the anchor shank. It’s called a catenary. This is a term well understood and used in many branches of physics, engineering, and architecture, and it’s used in the design of everything from bridges to offshore oil and gas rigs.

Mathematically, the catenary curve is the graph of the hyperbolic cosine function and they were first written about by Galileo in 1638 who recognized that the shape of a catenary was different from a parabola and worthy of study in its own right.

A catenary is a hyperbolic function, meaning that the curve is much steeper at the top (at the boat of the boat in this case) than the bottom. To calculate the shape of a catenary between a boat and the shank of an anchor, where the initial slope of the chain at the shank is flat and completely horizontal you would use this formula:

Where wind subscript f is the force of the wind, and C subscript w is the weight of the chain per meter.

We only care about the endpoints of the catenary, however, not the entire curve, so we can simplify this to:

where C subscript L is the length of the chain, H is the horizontal force acting on the boat (wind, wave, etc.), h is the depth of the water (plus height above the waterline of the rode attachment point), and w is the weight of the chain per meter.

What does all that mean?

Well, using high school algebra (if you can still remember that far back) you can notice a few relevant things. As H (forces acting on the boat) increases, length of chain will increase. If the weight of the chain increases, the length will decrease. If you increase h (height of the catenary or in our case depth + height above water), the length will again increase.

Now, all I need to know are the values of all of those variables and I can calculate the precise length of rode to pay out for those conditions.

So, let’s see, the force of the wind acting against my boat can be found with:

Where F is force in newtons, rho is air density, A is the projected area of the boat in square meters, C subscript d is the drag coefficient, and V is the windspeed in meters per second.

Calculating the force generated by waves became…well, really, really complicated. Instead I thought I’d just go with treating it as an additional load as a percentage of the force generated by the wind, which kind of works since for waves local wind is generating them. So maybe an additional 30-50% of the wind force for moderate and 100% of wind force for rough conditions. That’s probably too high but better safe than sorry.

The drag coefficient is a term in fluid dynamics that represents the resistance of a body moving through a fluid. It’s used in the drag equation to calculate the drag force on an object. The drag coefficient of the boat is stupidly difficult to calculate. Since I don’t have access to a computational fluid dynamics simulation, I just make some assumptions. Mast and rigging I assume to be 1.0 since they are not designed with aerodynamics in mind. For the rest I’ll assume a value between a sphere (0.5) and a streamlined body (0.2), let’s say around 0.35.

So the good news is, we actually can come up with an approximation of the forces acting against the boat which we can then plug into the catenary endpoint formula! I’m nerdy enough that I actually went through that exercise and did the math. It was pretty cool but ultimately, as you may have guessed, that technique is not practical for use in a daily situation. Just to see how the math works, I’ve used 12 ft as my cross-sectional width (2 hulls of 6 feet), and 20 feet as my cross-sectional height from waterline to top of salon roof. is my estimate of the cross-sectional area above the waterline for the hull and the 92.8 is my estimated cross-sectional area of the mast and rigging.
I’ve converted everything to meters instead of feet and 30 knots into meters per second just to make the math easier.
Let’s see how that math works with our assumptions:

This comes out to about 5157.5 Newtons of force, or about 526 kilograms force (kgf).

There were a lot of assumptions that went into the calculation but I feel like that’s actually a reasonable number. I expect that the fact that it’s a catamaran with 2 hulls and significantly higher coach roof make those assumptions more of a best guess than an reliable estimates.

Generalities to the rescue!

Fortunately, by pre-calculating the force applied on the boat using 30 knots of wind (34 knots is considered gale force so maybe I should have used that instead for safety sake), a formula that approximates the complicated math exists that works for 30-50 feet sailboats to maintain the catenary shape of the chain. This is also what we use on SV FearLess when we anchor:

Where C subscript L is the total chain length and d is depth plus height above the waterline. So if we anchor in 8 feet of water, we would lay out at a minimum 50 feet + (2 * 12) = 74 feet of chain. Don’t forget to add the height that the chain comes out of the water and attaches to your bow. For us, the rode comes out of the water about 4 feet, which is why we use 12 instead of just the 8 feet for the water depth.

If you compare this number to the typically recommended 7:1 scope (8*7 = 56), you’ll see that we lay out 22 more feet of chain. By contrast, if we anchor in 20 feet of water we use much less chain than the 7:1 scope. 24*7 = 168 feet of chain, 50+(2*24) = 98 feet of chain. Given that we have 200 feet of chain rode onboard, this gives us a lot of anchoring options that we wouldn’t have if we just used the 7:1 scope rule.

Details

As always, there are caveats.

First, this is the minimum amount of chain to lay out. If you have the swing room to accommodate more chain, do it. In this case more is almost always better, at least as long as you don’t swing into a rock, dock, or another boat.

Second, be smart; if you know weather is incoming and you’re going to stay on anchor, lay out more chain to help compensate against the higher winds and waves. Don’t just roll the dice and hope for the best.

Third, always take tides into account. If you anchor in 8 feet of water at low tide and the tidal range is 3 feet, at high tide your chain length hasn’t changed but the height of the catenary has. If you used the catenary formula of 50+(8*2) you laid out 66 feet of chain. If high tide raises you to 9.5 feet, you need to have 50 +(11*2) or 72 feet of chain. Always lay out enough rode to be safe at high tide if possible.

Resources:
The resources that I primarily relied on for this post were Chat-GPT, the excellent work done by the people at Trimaran San, the Cutter Hale Kai, and the NauticEd sailing blog. Also, I found an interesting article on the merits of anchoring from the stern in heavy weather from the man who tested the series drogue.

All of these references do a much better job of explaining things than I do, particularly the Cutter Hale Kai when discussing moorings, so give them a try.

Background

I love using command-line tools. I have no idea why I find it so much fun but there you have it. As a result, I like to use terminal-based editors. I settled on Vim a very long time ago. If you want to know why, ask me and I’ll write a short post about my reasoning. I switched to using NeoVim a while back and have never looked back, however, I’m going to be using the term vim to refer to both of them interchangeably.

This is because if you’re handed an out-of-the-box, vanilla instance of them both, very few people will be able to tell the difference between them. Which is good because while I’m virtually guaranteed that any flavor of *nix I might use will have vim installed, the same can’t be said of nvim.

One of the things I like about using vim is that despite using it for decades, I’m always learning new things about it, and I’m somehow always surprised that I never figured it out sooner.

Registers

If you’ve used vim for any amount of time, you’ll probably know that it has registers. A register is a place where vim stores text for later use, e.g., a copy-and-paste buffer. Vim has a lot of registers with a lot of different purposes.

There’s the unnamed register, the numbered registers, the named registers, and the read-only registers. Then there’s the expression register, the small delete register (yep, that’s a thing), and the selection and drop registers. But at the end of the day, they all store text.

I am not going to go into exhaustive detail about all of them. If you want me to do that, please ask, I’ll be happy to write about it. Today I just want to cover the ones that seem to get the most use.

Good Old Copy and Paste

Before we get going, I want to present the syntax of accessing a register. This is done by typing ” followed by the register name or symbol while in normal mode. You paste the contents of the register using the P or p command. The uppercase command pastes the text in front of the cursor location, the lowercase command pastes the text behind the cursor location. To copy text into a register using the yank command you type, imaginatively, y.

The Y command will yank text from the cursor position to the end of the line in Vim versions 8.2 and beyond. Versions older than 8.2 yank the entire line. In all NeoVim versions, the behavior of the Y command is to yank from the cursor to the end of the line.

The Unnamed Register

As an example, the most used register that I can think of is the unnamed register. The unnamed register is the register where the text from the last delete or yank command gets stored.
Although it’s called “unnamed”, you can access it by typing the double-quote character. So to insert text from the unnamed register you type, `””p`. The reason it’s called the “unnamed” register is that if you don’t explicitly provide a register, that’s the one that gets used. You can accomplish the same task simply by typing `p` without providing the register name. The same thing goes for the yank commands.

The Named Registers

Moving to the next most used registers, we come to the 26 named registers. These registers are, “named”, as the letters a-z, hence 26 of them. You might think that you could double that by using A-Z as well, but that isn’t what the creators went with. Instead when you yank text into a named register using the lowercase letter, you overwrite any existing text. When you use the uppercase letter, you append the yanked text to any existing register contents.

Now, I have to confess, I’m lazy, and while the idea of the named registers sounded great on paper when I first discovered them I went about using them in the most idiotic way possible. By trying to remember where I stored all of the different text snippets I had yanked. This was obviously a poor plan, and as a result, I would rarely use named registers.

In fact, the only time I can recall using named registers in the past was when I needed to swap two words or pieces of text which required that I use a named register because I would overwrite the first word if I used the unnamed register. This later developed into an even worse habit of only yanking and pasting one word at a time because it was just easier than using registers.

I made a fundamental mistake. I allowed my lack of familiarity with my tool dictate how I used it. This is invariably a bad idea.

The Numbered Registers

Eventually, I simply forgot about the named registers. I mean, I knew they were there, and I even knew how to use the commands, but I never developed the habit, so I just forgot about them. It was like I’d blocked them out even to the point of not using plugins for code snippet management because I just knew they used registers and registers were a pain to use.

Then came the moment that I saw someone use the registers command, or reg if you’re lazy like me. In this case, they were using it to show the numbered registers. The numbered registers are numbered 0-9 and are accessed using “0, “1, etc., where register 0 contains the text from the last yank command, and registers 1-9 contain a rolling list of the last 9 change or delete actions with 1 being the most recent and 9 being the oldest. The contents of 9 are replaced with the contents of 8 when a new change or delete action is taken and the contents of 9 are rolled off the end of the list and lost. 1

Now, while they were looking for the numbered register contents, the reg command will show the contents of all registers that have text in them, including the named registers.

How had I been using vim for decades and not known about this? I felt like suddenly everything I’d ever been missing about using registers made sense and I was filled with a sense of endless possibilities.

Always Learning

And I won’t lie, I also felt like a right moron. How had I never seen, heard of, or researched this obvious answer?? This also highlights a rule I try to live by and frequently fall short of; re-evaluate what you think you know on a regular basis. You never know which of your assumptions or habits may be hindering you.

Now we come to the moment I reveal the true depths of my awe-inspiring stupidity in all its majestic glory. I use a convenience plugin called which-key. It provides a pop-up showing the possible keybindings that can complete the command you started to type. Guess what that includes? Correct! If I type the “ command, that popup appears with the register list showing you exactly what text is in which register. I had been so averse to using registers that I’d never once typed in the command to access a register during the entire time I’d been using the plugin. That was at least a year.

After the angelic hosts stopped singing and I began playing with registers and the reg command, I discovered by accident that the which-key had already taken care of the inconvenience of managing registers this entire time if only I’d bothered to look. I was so averse to registers that even reading the homepage of the plugin when I first looked at it I didn’t even recognize that it solved my long-standing complaint against registers.

After having many people poke at me and my latest annoyance at having to rebuild my preferred environment in a container image that isn’t under my control, I decided to bite the bullet and get my dotfiles under useful management.

Up until now, I’ve put off real management of my dotfiles by just creating a git repo and storing my dotfiles there. It was a half-baked solution at best and required a lot of tweaking by hand when I used it to try and recreate my CLI environment.

As the years rolled on my personalizations grew. A lot. I now have a lot of configurations to track and I’ve now found myself in the position of having to rebuild my podman containers on a semi-regular basis.

This last rebuild has motivated me to try and find a simple, portable, and functional solution to exporting and importing my dotfiles to a fresh environment. There may be better solutions out there but this one seems to work pretty well for me, at least so far. It begins, as many things in software seem to do, with GNU.

The Solution I Chose

I decided to go with GNU stow. It’s really a symlink manager and its original purpose was to manage software versions. Since that point of origin, it has been widely adopted as a way to manage dotfiles. There are other solutions, such as dotbot, but the instructions say you need to install it as a git submodule, or if you have Python set up you can use PyPi. This seemed like I might head down another rabbit hole if I went that route and I didn’t want anything complex so I went with stow. It’s been around for over two decades so it’s a known quantity, there’s a lot of documentation for it (not that it needs a lot) and it’s available pretty much anywhere you can use a package manager.

Pre-requisites

You really only need to know a couple of commands to use Stow for managing dotfiles and the pre-requisites are pretty straightforward. You need to be familiar enough with your package manager to install packages, you need to be familiar enough with git to use it, and lastly, you need to install git and stow.

Implementation

I’m on a Mac so for me I just needed to execute a

brew install stow (I already have git installed) and I’m off to the races. If you’re using homebrew and you’re on a Linux machine, I understand and please use your package manager of choice, if you’re on a Mac and you’re not using brew, shame on you, you’re doing it wrong.

Segue

Before getting started, since a lot of the rebuilding I’m doing is being done on my work-issued laptop, and both my personal GitHub and my work GitHub use SSH keys to manage AuthN/AuthZ on GitHub, I need to juggle multiple SSH keys. Work also requires that I sign commits with GPG, which is a step beyond my personal paranoia and fear that someone really wants to tamper with my code. This means I need to take extra steps to make it convenient to commit code when I make changes to my dotfiles.

First I needed to make sure that the identities were set up properly. I did this in the SSH config file. This is typically located at ~/.ssh/config and mine looks something like this:

Host github-work
 HostName github.com
 ForwardAgent yes
 IdentityFile ~/.ssh/work_github
 IdentitiesOnly yes
Host github-personal
 HostName github.com
 User git
 ForwardAgent yes
 IdentityFile ~/.ssh/personal-gmail-github
 IdentitiesOnly yes

In this example I’m mapping my 2 SSH keys to separate hosts, named github-work and github-personal, which can be thought of as aliases/profiles for our purposes. The only tweak worth pointing out here is that I deliberately use the IdentitiesOnly configuration. By default the ssh client will still send the default key. This can be problematic when using multiple accounts and by including the IdentitiesOnly directive it only sends the key explicitly given in IdentityFile.

End Seque

Now that we have SSH configured and stow installed, we get to the meat of the article. Spoiler alert, the seques are probably more interesting than the stow commands.

Directory Structure

When using stow you need to create a directory to hold the managed files and the directory structure of the files under management must mirror the directory structure of the files as they would appear in your home directory, i.e., if your .tmux.conf file is in your home directory and the directory containing your managed files is called dotfiles then the managed tmux file would be located at ~/dotfiles/.tmux.conf and similarly if your tmux plugin files are located at ~/.tmux/plugins/my-plugin then the corresponding managed files would be located at ~/dotfiles/.tmux/plugins/my-plugin.

Another Seque

In my case I already had a repo for my dotfiles called imaginatively, dotfiles, on github that I decided to clean out and reuse. So I just needed to clone the repo to set up version control. Given I have multiple accounts to juggle my clone command looked like this:

git clone github-personal:Clay-Ratliff/dotfiles.git

Note that I used the profile name created in the Host directive to use the correct ssh key. Also, given that I did not need to sign commits on my personal repo I also executed git config commit.gpgsign false which disabled gpg signing for that local repo, not globally.

End Seque. Again.

I’m going to go with the safest way to get started migrating your dotfiles to a managed system rather than the quickest. I’m a fan of do as a say not as I do and I don’t want to encourage other people to adopt my bad habits.

Back. Up. Your. Dotfiles.

Before you put your files under management make a backup. Only then would I move them to the stow directory. My preferred method is the simple expedience of the following:

cp ~/.tmux.conf ~/.tmux.conf.bak
mv ~/.tmux.conf ~/dotfiles/

So simple even I couldn’t screw it up.

Magic Happens Here

From here you can put your tmux config under control. The magic happens like this:

stow .

Are you not impressed? OK, I admit, it’s more than a little anti-climactic. If you look in your home directory you can see that stow has created a symlink of your tmux config file in your home directory. That’s it. You have met the magic, and the magic is boring.

Commit your changes. Now, not later.

At this point, I immediately commit that change and then remove the backup file because I know that I can reproduce a working copy of my tmux conf from the repo and move it back by hand if something weird happens.

You’ll want this comfort because things have been so boring and easy up till now you’re going to want to start running with scissors which is never a good idea now matter how well you know the path. All the other files will be a rinse and repeat of this.

Yet Another Segue

This can be a rinse and repeat action. However, if you’re like me (or a lot of other people) you are almost certainly using plugins, for your shell, for tmux, for your editor, etc., and if so, they’re probably under source control. It’s a common practice for plugin managers to simply pull the plugin down from github and place it in a plugins directory as a git repo and update the repo and plugins get updated.

The delima here is that you’ve placed your dotfiles directory in a git repo. If you try to add it as a normal file you’ll get something like this:

[0] <git:(main 350f7bd✱✈) > ga .
warning: adding embedded git repository: .tmux/plugins/tmux-continuum
hint: You've added another git repository inside your current repository.
hint: Clones of the outer repository will not contain the contents of
hint: the embedded repository and will not know how to obtain it.
hint: If you meant to add a submodule, use:
hint:
hint:   git submodule add <url> .tmux/plugins/tmux-continuum
hint:
hint: If you added this path by mistake, you can remove it from the
hint: index with:
hint:
hint:   git rm --cached .tmux/plugins/tmux-continuum
hint:
hint: See "git help submodule" for more information.

In this case, you should take the common sense approach and follow the advice. You can take a peek inside the repo config to get the url, in the above example that looks something like this (ignore the formatting stuff, I use a lot of customizations for my CLI):

[2] <git:(main 350f7bd+) > cat ~/.tmux/plugins/tmux-continuum/.git/config
───────┬──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
       │ File: /Users/clay.ratliff/.tmux/plugins/tmux-continuum/.git/config
───────┼──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
   1   │ [core]
   2   │     repositoryformatversion = 0
   3   │     filemode = true
   4   │     bare = false
   5   │     logallrefupdates = true
   6   │     ignorecase = true
   7   │     precomposeunicode = true
   8   │ [submodule]
   9   │     active = .
  10   │ [remote "origin"]
  11   │     url = @github.com/tmux-plugins/tmux-continuum">https://git::@github.com/tmux-plugins/tmux-continuum
  12   │     fetch = +refs/heads/master:refs/remotes/origin/master
  13   │ [branch "master"]
  14   │     remote = origin
  15   │     merge = refs/heads/master

Then you can substitute the URL and follow the advice like so

g submodule add @github.com/tmux-plugins/tmux-continuum">https://git::@github.com/tmux-plugins/tmux-continuum .tmux/plugins/tmux-continuum

Now you’ve preserved all of the repo information and whatever plugin manager you use for whatever app should behave as expected.

An alternative approach

If you want to speed things up a bit, you can also use stow --adopt . as an alternative.

Normally, if a file exists and you attempt to import it into stow, then stow will throw an error because it will detect that the target file/directory already exists. In order words if I we had copied the tmux config file from the home directory into the dotfiles directory and run stow ., we would have gotten an error because the orginal .tmux.conf file would still have existed.

└─[0] <git:(main 16932a5✈) > stow .
WARNING! stowing . would cause conflicts:
  * existing target is neither a link nor a directory: .fzf.zsh
All operations aborted.

In the example above I simply copied the .fzf.zsh file into the dotfiles directory and attempted to add it.

You can change this default behavior by using the adopt flag. Adopt will shorten the process by one step by removing the need to delete the target file/directory before stowing. The trade-off for skipping that step is that the files that you’re stowing can be altered. This is not normally a big deal, unless the package is already owned by stow.

Side note, files and directories managed by stow are called ‘packages’ and any packages that stow knows about are considered ‘owned’ by stow.

Imagine that you copied the tmux file in the above example into the stow directory and placed it under version control but didn’t stow it. Then you made some changes to the original config file later and realized that you hadn’t stowed it. If you now run stow --adopt .tmux.conf then stow will replace the .tmux.conf file in the home directory with a symlink to the .tmux.conf package that it now owns, however, if you run git status against that directory you’ll see that the file has changed. The original file in stow was overwritten by the modifed version from the home directory. If it’s under version control you can decide to keep or discard those changes. If it’s not, you’ll never know the file had changed.

It’s a weird edge case, especially considering that since it’s a symlink if I do something like vim ~/.tmux.confto edit the file then I’m editing the file the symlink is pointing to (because that’s how symlinks work), and I honestly haven’t experienced any time where that’s an issue but I suppose it could happen?

Conclusion

This turned out to be a bit more of a confusing ramble than I was anticipating so if you found this more confusing than educational, please let me know by asking for clarification in the comments.
Also if you have another topic you’re interested in please let me know and I’ll see if it’s something I can answer!

Also if you have any interest in following along on the weird adventures of couple of newbie sailors living on a sailboat, you can subscribe to my newsletter at Sailing FearLess.

When we last left off we were using the primitive executor that is available with the futures crate, sometimes called futures-rs. The futures crate provides the common abstraction layer for async behavior in the Rust language. It is intended to be the basis for creating complex runtimes. As discussed in the previous articles, this is by design. Unless you intend to create a custom runtime, executors, etc., from scratch, you probably won’t be using futures directly.

I’ve chosen to use the async-std crate for this example simply because it’s easily read by anyone already familiar with the Rust std library. I’ll also be using timers to demonstrate that we’re actually running in a concurrent fashion as expected.

First, we’ll set up our cargo.toml file as follows.

[package]
name = "rust-async"
version = "0.1.0"
edition = "2021" # See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html[dependencies]
async-std = "1.12.0"
futures = "0.3"
rand = "0.8.5"

I simply reused the original cargo.toml file and added the async-std and the rand crates as dependencies. We add the rand crate so that we can generate random sleep times for our function.

Next, we’ll alter the main.rs file to look like this:

use std::time::{Instant, Duration};
use async_std::task;
use rand::Rng;

fn main() {
    let start_time = Instant::now();
    task::block_on(async_main());
    print!("Completed in {} ms", start_time.elapsed().as_millis());
}

async fn async_main() {
    let long_running_stuff = make_coffee_and_bacon();
    let other_stuff = make_eggs();
    futures::join!(long_running_stuff, other_stuff);
}

async fn make_coffee_and_bacon() {
    futures::join!(make_coffee(), make_bacon());
}

async fn make_coffee() {
    burning_time("Coffee".to_string()).await;
}

async fn make_eggs() {
    burning_time("Eggs".to_string()).await;
}

async fn make_bacon() {
    burning_time("Bacon".to_string()).await;
}

async fn burning_time(breakfast_item: String) {
    let mut rng = rand::thread_rng();
    println!("{} started!", breakfast_item);
    let sleep_duration =  Duration::from_millis(rng.gen_range(100..1500));    task::sleep(sleep_duration).await;
    println!("{} finished after {:?} ms", breakfast_item, sleep_duration.as_millis());
}

This looks very similar to the final version of the code that we had at the end of the previous article after all the modifications we made. The set of changes that we made start with the use statements. We import Instant and Duration from the Rust std library. We use those for timing the execution of the code so that we can prove the functions are running concurrently. Instead of importing the futures crate, we import the async-std crate. We also import rng::Rng so that we can generate some random timings.

The only change to main other than creating a timer and printing the total execution time out is that instead of using the block_on() executor from the futures crate, we instead use the task::block_on() interface from the async-std crate. It behaves similarly to the one we saw in the futures crate but the executor is more sophisticated because while the task blocks the current thread, it also runs all of the async code concurrently. If one async function can’t complete, e.g., it’s asleep, it will yield control to a task that can make progress. As an aside, if it seems weird that we use a blocking interface to run concurrent tasks, the reason is that we don’t want the program to exit prior to all of the tasks being driven to some form of completion.

If you compile this program and run it, you will see results similar to the ones shown here:

└─[0] <git:(master b7644d4✱) > cargo run
    Finished dev [unoptimized + debuginfo] target(s) in 0.03s
     Running `target/debug/rust-async`
Coffee started!
Bacon started!
Eggs started!
Bacon finished after 211 ms
Eggs finished after 430 ms
Coffee finished after 1318 ms
Completed in 1319 ms%
┌─[clayratliff@pop-os] - [~/PersonalGithub/rust-async] - [2022-07-01 05:59:36]
└─[0] <git:(master b7644d4✱) > cargo run
    Finished dev [unoptimized + debuginfo] target(s) in 0.03s
     Running `target/debug/rust-async`
Coffee started!
Bacon started!
Eggs started!
Coffee finished after 255 ms
Eggs finished after 1429 ms
Bacon finished after 1499 ms
Completed in 1500 ms%

We can see from the results that while the breakfast items are always started in the same order, they are clearly being run concurrently because the order in which they are finished varies based on their random sleep times. We can also see that the total runtime of the code is always the length of time taken by the longest sleep plus one or two milliseconds.

Lastly, let’s take things one step further and create something that looks more like a real-world application. We’ll run multiple concurrent tasks to grab some data from the web and return a Result<T, E>, an extremely common return value for I/O.

First, let's add some needed crates to our cargo.toml file:

[package]
name = "rust-async"
version = "0.1.0"
edition = "2021"# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html[dependencies]
async-std = "1.12.0"
futures = "0.3"
serde = "1.0.138"
surf = "2.3.2"

We’re already familiar with the async-std and futures crates from the previous article. The data we’ll be pulling for our example will be stock ticker data in JSON format. Serde is a framework for serializing and deserializing data structures in Rust. More details can be found in the link but we’ll be using it to deserialize the JSON data. We’ll be pulling our ticker data from Alpha Vantage. Mostly because the signup is quick and easy. It also doesn’t cost anything but does give me live data to work with. A sample of the JSON data looks like this:

{
    "Global Quote": {
        "01. symbol": "IBM",
        "02. open": "141.0000",
        "03. high": "141.6700",
        "04. low": "139.2600",
        "05. price": "141.1200",
        "06. volume": "4012106",
        "07. latest trading day": "2022-07-01",
        "08. previous close": "141.1900",
        "09. change": "-0.0700",
        "10. change percent": "-0.0496%"
    }
}

Surf is an HTTP client framework that works well with async-std, and has a convenient integration with serde for handling JSON documents.

Now let’s take a look at our simple stock ticker example in main.rs

use async_std::task;
use futures::join;
use serde::Deserialize;
use std::time::Instant;
use surf::Result;

#[derive(Deserialize)]
struct GlobalQuote {
    #[serde(rename = "01. symbol")]
    symbol: String,
    #[serde(rename = "02. open")]
    open: String,
    #[serde(rename = "03. high")]
    high: String,
    #[serde(rename = "04. low")]
    low: String,
    #[serde(rename = "05. price")]
    price: String,
}

#[derive(Deserialize)]
struct TickerData {
    #[serde(rename = "Global Quote")]
    full_ticker: GlobalQuote,
}

fn main() {
    let start_time = Instant::now();
    task::block_on(async_main());
    println!("Completed in {} ms", start_time.elapsed().as_millis());
}

async fn async_main() {
    let api_key = "API_KEY";
    let ibm_symbol = format!(
        "https://www.alphavantage.co/query?function=GLOBAL_QUOTE&symbol=IBM&apikey={}",
        api_key
    );
    let tesla_symbol = format!(
        "https://www.alphavantage.co/query?function=GLOBAL_QUOTE&symbol=TSLA&apikey={}",
        api_key
    );
    let apple_symbol = format!(
        "https://www.alphavantage.co/query?function=GLOBAL_QUOTE&symbol=AAPL&apikey={}",
        api_key
    );
    let first_symbol = get_ticker(ibm_symbol);
    let second_symbol = get_ticker(tesla_symbol);
    let third_symbol = get_ticker(apple_symbol);
    let (result1, result2, result3) = join!(first_symbol, second_symbol, third_symbol);
    
    println!("IBM ticker price: {}", result1.unwrap().price);
    println!("Tesla ticker price: {}", result2.unwrap().price);
    println!("Apple ticker price: {}", result3.unwrap().price);
}

async fn get_ticker(url: String) -> Result<GlobalQuote> {
    let start_time = Instant::now();
    let TickerData { full_ticker } = surf::get(&url).recv_json().await?;
    println!("Async completed in {} ms", start_time.elapsed().as_millis());
    Ok(full_ticker)
}

The structures GlobalQuote and TickerData will hold the deserialized data. The #[derive(Deserialize)] annotations handle implementing the Deserialize trait for serde. The #[serde(rename = "")] does what you probably expect it to do; it maps the JSON field name inside the quotes, to the field name in your struct.

The main function hasn’t changed at all. The first 4 lines in async_main just set up the URI for the stock data API. We then create a Future for our three stock symbols we want to get information on, creatively named first_symbol, second_symbol, and third_symbol. We then pass those Future's to the join! macro, which we’ve also seen in the previous article. And finally, we print out the resulting stock prices.

The real meat, if you can call it that, lies in the trivial get_ticker() function. The Result<> that's returned is from the surf framework and is a wrapper for the standard library Result. We’re returning GlobalQuote because we want to return the actual stock data from the request to the calling function. The let TickerData { full_ticker } syntax indicates that the JSON should be mapped to the specified structure. It’s not super useful in our case since the values are all strings but if the returned data had been typed as something other than strings, e.g., if the JSON document price were an actual number, we could have defined the field price as a f64 and serde would have converted it to the appropriate type.

If you create an account and substitute your generated API key for the placeholder in the code, this should run out of the box and return a result similar to what you see below.

Async completed in 226 ms
Async completed in 232 ms
Async completed in 278 ms
IBM ticker price: 141.1200
Tesla ticker price: 681.7900
Apple ticker price: 138.9300
Completed in 281 ms

Again, you can see by the times that the requests are running concurrently and the total time for completion is within a few milliseconds of the longest-running task.

I had not done any significant asynchronous programming in other languages before attempting to understand it in Rust and I struggled to understand how it worked. My biggest struggle was a misconception that there was a clear standard on how async/await worked in Rust built on top of the futures-rs crate and that all of the runtimes were simply a more sophisticated version of futures. This turns out to be not quite the case.

There are three primary ecosystems for asynchronous Rust. These eco-systems are Tokio, async-std, and smol. I’ll take a brief look at each of them, and point out a couple of interesting things. The links provided throughout this article will give you much more detailed information.

Tokio is the oldest, released in 2016, and is the most widely used runtime at the time of writing. It uses custom I/O traits built on top of mio. This means it needs a compatibility layer to integrate with futures. It’s very configurable with many utilities but comes as one large crate although you can use feature flags to make the crate lighter.

Async-std was released in 2019 with the goal of being a full runtime async version of the Rust standard library and is built on top of futures. It also uses the async-executor crate, the executor provided by smol.

Smol is the newest, created in early 2020. It’s also built on top of futures and was created by Stjepan Glavina, co-creator of async-std, and also uses the futures crate. It’s intended to be as small as possible and as such the functionality is split into many different crates. It also provides a crate for compatibility with tokio.

I wanted to list the references that I used in the research for this article and I’m grateful to the authors. Some of the materials in them are outdated but this is a fast-moving target and they got me close enough.

• RustFest Barcelona talk

• Phil’s Blog

• Rust community wiki

Conclusion

I hope that this helps you understand Rust async better, rather than making things more confusing. I appreciate your time in making it this far. If you find that I got something wrong, or have something to add please let me know in the comments.

If you liked this article or the series, please like, share, and restack it.

Subscribe to get more tech tidbits when I publish them!

First, let’s create a new project using Cargo with cargo new rustasync --bin

Now, open up the generated Cargo.toml file and add the futures-rs crate to the dependencies section. You can follow the link above to learn more and find the complete documentation but in summary, it’s a bare-bones implementation of an async runtime. Your toml file should look similar to the one below:

[package]
name = "rust-async"
version = "0.1.0"
edition = "2021"# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html[dependencies]
futures = "0.3"

The futures-rs crate implements some of Rust's core abstractions for asynchronous programming. We’ll go over those in more depth later but for now, the Future trait is of the most relevance to us.

Let’s imagine that we’re going to fix breakfast at home. We might choose to make some coffee, eggs, and because I’m trying to be healthy, I’ll use the oven to make some bacon. My process would be something like starting the coffee maker and getting the coffee brewing. While that’s happening I could pre-heat the oven. While the oven is pre-heating I can get the bacon out and ready it for the oven. Once the bacon is ready I can put it in the oven, pour myself some coffee, and cook the eggs while waiting for the bacon to finish. I’m sure you get the idea at this point. If we were to prepare breakfast synchronously, one task after the other, it would take significantly longer to complete preparing breakfast and most of it would be cold by the time we got to eat it.

Let’s write a simple application to try and mimic this approach by using asynchronous code. Open up the main.rs file under the src directory and add the following code.

// The block_on function simply blocks the current thread until the 
// Future has completed and returned.
use futures::executor::block_on;

fn main() {
    // The make_coffee, make_eggs, and make_bacon functions 
    // have all been annoted with the async keyword. 
    // This means that they will all return a Future.    
    // In fact, the following 2 functions are the same    
    // async fn foo() { // do stuff }
    // fn foo() -> impl Future<Outpout = ()> {
    //     async { // do stuff }
    // }    

let coffee_future = make_coffee();
    // The block_on executor accepts a Future as input and blocks
    // the current thread until that future has completed
    block_on(coffee_future);
    let bacon_future = make_bacon();
    block_on(bacon_future);
    let eggs_future = make_eggs();
    block_on(eggs_future);
}

async fn make_coffee() {
    println!("Coffee started.");
}

async fn make_eggs() {
    println!("Eggs started.");
}

async fn make_bacon() {
    println!("Bacon started.");
}

First, we import a very simple executor from the futures crate. In Rust futures are lazy. We’ll look at this in a bit more detail later, but creating a future does nothing on its own. It needs to be driven to completion by an executor. That’s the job of the on_block() executor. It accepts a Future as a parameter and executes that code while blocking the main thread.

The code above is not idiomatic because we create the Future that represents the async function, immediately call the block_on() executor, wait for the async code to complete, and repeat this process with the rest of our async functions. This effectively makes the execution synchronous. Let’s tweak the code a bit and make it look a bit more idiomatic.

So we know that we can start the coffee maker and let it work while we cook our bacon in the oven, so let's start those two tasks before starting our eggs.

First, we’ll add a new async function called make_coffee_and_bacon() shown below:

async fn make_coffee_and_bacon() {
    make_coffee().await;
    make_bacon().await;
}

Next, we’ll add another new async function called async_main():

async fn async_main() {
    let long_running_stuff = make_coffee_and_bacon();
    let other_stuff = make_eggs();
    futures::join!(long_running_stuff, other_stuff);
}

Finally, we’ll change the existing main() function to look like this:

fn main() {
    block_on(async_main());
}

Let’s go through these changes one at a time. First, we create a new async function for the coffee and bacon tasks. You’ll notice that we use await inside the functions. Using await inside an async function is similar to using on_block() to wait until the Future has resolved, but instead of blocking the thread, it allows other tasks to run if the future is blocked, e.g., waiting on input from a socket or stream.

Second, we’ve added the async_main() function. This is a pretty straightforward change. Since await can only be used inside async functions or async code blocks, and main() function cannot be annotated with the async keyword. While the thread that runs main() will block until the block_on() call resolves, everything from async_main(), all the way down the execution chain will run asynchronously. The third line is new. The futures::join!() macro is similar to the await keyword but it operates on multiple futures, returning only when all Future’s complete, regardless of the order of completion.

Earlier I mentioned that Futures are “lazy”, in that they do nothing on their own. They will never return a value unless driven to completion. This may seem strange, especially if you’re familiar with how other languages implement asynchronous code, but this is a deliberate choice by the language designers. By allowing Future's to be lazy, creating an async function becomes a zero-cost operation. By “zero-cost”, I mean that there is no runtime impact for creating a Future unless you actually use them. If you want to see an observable example of this laziness in action, just remove the .await from the call to make_coffee() and make_bacon() in the make_coffee_and_bacon() function, and run the code. You’ll notice that when you run it after removing the .await, it will never print any output.

I wanted to note, in this specific example, that the order of execution of our async functions does not change. You could stick a random sleep in any, or all, of the three functions to change how long the execution time takes, but it will not impact the order the print statements appear.

This may seem counter-intuitive, after all, this is supposed to be asynchronous execution. Let's take a look at why. If you recall from the first article in the series, we mentioned that the await keyword is used by the generated state machine as a marker designating atomic units of execution. For example, in the async function make_coffee_and_bacon(), we first call make_coffee().await, followed by make_bacon().await. In the generated state machine, everything from the start of the function until the first await keyword is considered an atomic unit. Similarly, everything after the call to make_coffee().await, until the next await keyword would be another atomic unit. This means that the function make_coffee() will complete before running make_bacon(). The await keyword by itself doesn’t actually drive a Future to completion. It’s the block_on() call that drives the async code to completion. I used the term executor earlier but didn’t really define it. An executor is responsible for driving all of the async code it’s responsible for to completion. In our case, we pass async_main() to the block_on() executor and that executor is responsible for driving all of the async code that async_main() contains to completion, but block_on() doesn’t implement any logic which would allow it to multiplex execution. In order to allow code using async/await to execute in an asynchronous fashion, we need to use a more sophisticated executor. I think of it like this; a future is a computation we want to perform, the task schedules that futures execution, and the executor ensures that all of the tasks are completed, and coordinates interdependencies.

Incidentally, the reason the join!() macro exists is to allow a more flexible composition of tasks. For example, if we were to swap out block_on() for a more complex executor, we wouldn't need to make any other changes to our async code.

This was part 2 in a 3 part series. I’ve tried to present this information in a simple way while still giving enough details to provide a basic understanding. It turned out to be a bit more difficult than I anticipated.

In the next section, we’ll take a look at a more realistic implementation of Rust async and talk about some of the different runtimes available, as well as some of the gotchas. If you made it this far, please give the article a like, share it, or better yet, comment, so I know if you found this useful.

These days, if you’re writing code in a reasonably modern language, it’s difficult to escape the idea of asynchronous programming. If you search for the most popular programming languages of 2022, every single entry in the list supports asynchronous programming. Most of them even use the same keywords in their syntax: async and await. This article will be the first in a series of articles that will present the general concepts, followed by a brief discussion of how async is implemented in Rust, and finally, some sample code to demonstrate the concepts.

Intro to Asynchronicity

Let’s start by clearing up some common misunderstandings. First Concurrency is the act of performing multiple tasks simultaneously. For example, I can both walk and chew bubble gum at the same time (most days). A different example would be making a pot of coffee while I read the morning paper. Both of these are examples of concurrency but the means by which they are achieved differ. Walking and chewing gum at the same time is an example of parallelism since I can do both things simultaneously. Making a pot of coffee while reading the paper is an example of synchronicity since I can’t physically do both tasks simultaneously. Instead, I prep the coffee maker and start it, then begin reading my paper. When the pot is done, I then stop reading, pour a cup, and go back to reading. I can work on both tasks at once, just not at the same time. In other words, I move on to another task, while waiting for the previous one to complete.

In languages that support async programming, in very simplified terms, async code is implemented using a compiler-generated state machine. This state machine uses the await keyword to break up the execution units into smaller chunks, creating a code block from the beginning of the method to the first await boundary as one task. The next execution chunk would be all the code between the first await, and the second await, if there is one. I mention this detail to point out that threads do not enter into the picture. They can, and certainly do, but the point is that asynchronous code has nothing to do with threads.

Why did anyone want to create this complicated-sounding mess? In short, it beats the alternatives. This technique allows us to create a large number of independent concurrent tasks without the expensive requirement of spinning up a large number of new threads, it’s easier to understand and reason about than the alternatives, and it’s performant.

How Rust Implements Async

When the designers of Rust implemented async/await they had some specific goals in mind, and as a result, there are several differences between Rust and most other languages.

In broad strokes, Rust async is built on two types; Futures's and Task's. AFuture represents a piece of code that will execute and return a concrete result at some point in the future.

A Task is an in-process Future that is being driven to completion. In other words, a Task is a Future that can schedule its execution.

Async in Rust does not use heap allocations or dynamic dispatching, which makes it extremely performant and keeps the footprint small, allowing async to be used in embedded environments or other resource-constrained apps.

Rust has no built-in runtime for implementing async. It relies on separate crates to provide a runtime. As a consequence, you can select a runtime that aligns most appropriately with your needs. For example, there are both multi-threaded and single-threaded runtimes. It also means that if you’re not using async, your code won’t be bloated by unused features.

While Rust gives you the components you need to create a custom runtime, the reality is that very few people have the knowledge, time, or desire to put forth that level of effort. Instead, they’ll select an existing runtime. We’ll cover the primary runtimes later. In the next article, we’ll look at examples of async in Rust starting with a basic functioning async example, and adding complexity from there.

This has been part 1 of a 3-part series. If you enjoyed this article, please like, share, restack, or comment. Subscribe to keep up with the rest of the articles in the series where we move beyond introducing the concepts, and into actual code!

You can find part 2 of the series here.

Why do we need to keep revisiting this?

The short answer is we don’t.

The long answer is a bit more personal. While I don’t work as a developer anymore, I do still have to work with code occasionally, and I still, for whatever reason, enjoy tinkering with my tools. I find that talking about and tinkering with CLI tools is a lot of fun, and one of my favorite tools is Neovim. Every now and then I’ll re-evaluate my setup and try to improve on my previous config by setting up a new one from scratch, keeping what I felt worked well in the previous setup, and adding new things that I think will either improve on tools I already use or which will streamline a workflow that I didn’t previously have. And yes, believe it or not, as your career advances your workflow will also continue to change. Yet another reason occasional re-evaluation is a good thing.

To potentially save people some time, I want to mention that this article is not intended for people new to Vim or NeoVim. I assume that you have some previous knowledge of vim/vim and either have experience with Lua or, at the least, no aversion to it.

Neovim vs Vim

I wanted to get this out of the way real quick because there will probably be some confusion if you’re not familiar with the history. Neovim is a fork of Vim that has several notable improvements. The core functionality is compatible with Vim, e.g., if you aliased a bare version of vim with a bare version nvim, a user opening the editor would never realize they were using NeoVim.

The notable improvements mentioned above are based on aggressive refactoring giving the following benefits:

• Overall performance improvements

• Using the XDG Base Directory standard for configuration instead of hardcoded directories

• A more sophisticated plugin API that allows the use of RPC calls and Lua for plugin creation and scripting (Vimscript v1 is also supported)

• Native support of LSP

• Support for asynchronous I/O

I switched between the two editors without much thought for years but decided to finally commit to Nvim in June of 2022 when the decision was made to essentially re-implement vimscript from scratch instead of simply embracing an already existing language like Lua or Python. Are you going to re-invent a wheel that is already notoriously quirky and only usable in a single tool? That was when I decided just to embrace NeoVim.

On with the Configuration

Before we start a quick configuration note; nvim supports init.vim or init.lua as the default configuration file, but not both. If you’re going to commit to Nvim, you probably want to use init.lua, and since I do, I will.

I’ve always been kind of scattered when it comes to plugin management but I’d like to find a single solution and stick to it. Since I’m all in for learning new things, I’ll be starting off with a clean slate and using Lazy.nvim as my plugin manager. It’s still relatively new (and I’m new to it) but I’ve read good things about it, it embraces lazy loading of plugins which is nice, although load time has never been a real issue for me to be honest, and the default plugins it installs make for a fascinating user experience. There was a lot of exploration there.

One of the things I truly appreciate is that the instructions for installation start off by telling you to make a backup of your existing configuration and then show you how to do it. Thanks, buddy!

After following the install instructions and opening nvim, you’ll be visually assaulted by a bunch of scrolling text as lazy.nvim downloads all of the default plugins and installs them. When that finally ends you’ll see something like this that shows you the total number of plugins, whats’ installed and what’s currently loaded:

I’ll admit, I was a bit startled at first. It clearly just left me in a buffer displaying the lazy.nvim install plugin management interface as indicated by the menu at the top. While it was not what I was expecting, it was a simple and effective menu. The ? action gives some useful context on what was going on.

When I hit :qI’d intuitively half expected to quit nvim, but instead, it dumped me to the default lazy.nvim popup interface.

This is what you would see if you exit nvim and start it again. Clearly, the intent is to have a single plugin manager that, by default, turns nvim into an IDE any user of VS Code or IntelliJ would feel comfortable with. I would also add that I’d prefer it didn't have the giant modal proclaiming that it was Lazyvim on startup but I can’t really blame it, the resulting product is pretty nice.

The documentation for lazy.nvim is pretty good and after some reading and research, I found a more simplistic approach to using lazy.nvim that felt more like what I was after; using the plugin manager as a plugin manager and ignoring its opinion of what my IDE should be.

Simplistic Approach

I deleted the nvim directory and everything in it (remember not to delete your backup of the original configuration) starting over from scratch. Overkill perhaps but I like a clean slate approach, it helps make things more reproducible.

As I said earlier, my usual approach tends to look like a Frankenstein hodgepodge of configuration files, so I’m going to try and be less haphazard about my config and try to impose some structure on it from the start. I eventually decided on a pretty standard-looking folder structure as shown below.

I think it’s reasonably clear. The user directory is what I’ve chosen to name the directory where I’ll store my user-specific Lua scripts, and the user/plugins directory is where the plugins will go. I could have named the directory after my OS username, which is what most people do. I chose this just to be different. I suppose for a multi-user scenario, it would make sense to have the individual user directories in there, e.g., user/user.name but I don’t think most people use that kind of setup and if I’m honest, I was just lazy and wanted to be different.

Below is my base levelinit.lua in the root luadirectory. The lazy.lua script is in the root plugin directory. It puts the fzf executable into the runtime path, loads my preferences for options (things like line numbers, tab stops, etc.), loads the lazy.vim settings (more on that later), and finally create an auto-command that lazy loads keymaps and user-defined auto-commands:

-- Set runtime path to include fzf
vim.opt.rtp:prepend("/opt/homebrew/opt/fzf")

-- Load options file
require("user.options")
require("user.lazy")

vim.api.nvim_create_autocmd("User", {
  pattern = "VeryLazy",
  callback = function()
    require "user.autocmds"
    require "user.keymaps"
  end,
})

A couple of notes here — I load the options ahead of everything else because I keep my leader key in there and I want that mapped so that any subsequent plugins, autocommands, etc., get mapped correctly; I also don’t know that the lazy loading of auto-commands or keymaps is all that useful, but it was keeping with the theme so I went with it.

The current lazy.lua file contains code that pulls the latest stable copy of lazy.nvim when needed, adds lazy.nvim to the runtime path, and lastly tells lazy where the import directories are located (it doesn’t import nested directories under lua before setting a couple of options on what to do regarding updates.

local lazypath = vim.fn.stdpath("data") .. "/lazy/lazy.nvim"
if not vim.loop.fs_stat(lazypath) then
  vim.fn.system({
    "git",
    "clone",
    "--filter=blob:none",
    "https://github.com/folke/lazy.nvim.git",
    "--branch=stable", -- latest stable release
    lazypath,
  })
end
vim.opt.rtp:prepend(lazypath)

require("lazy").setup({ {import = "user.plugins"}, {import = "user.plugins.lsp"} }, {
  checker = {
    enabled = true,
    notify = false,
  },
  change_detection = {
    notify = false,
  },
})

Now when I start nvim it will start as it normally would I can use the command :Lazy to bring up the new plugin manager interface when needed.

The Plugins

The plugins listed in the screenshot are the ones that I’m currently using and have configured. A quick note on lazy; I was pleasantly surprised that when I created a new plugin file and saved it, lazy picked up the change and loaded the configuration.

Now that I’m embracing a Lua-only setup with nvim, I needed to transfer all of my vimScript stuff to Lua. I’m lazy and don’t like typing so I shamelessly stole my coworkers options.lua and keymaps.lua as a starting point to minimize effort. I also added a autocmds.lua file for any autocommands I’ll want to set up.

I set up nvim-tree as my file explorer, which is new to me. I traditionally haven’t used a file explorer. I also configured a nerd font to provide widgets.

I’ve got an init.lua for the plugins directory to load plugins that I don’t need to provide any specific configuration for and I’ve moved the color scheme into its own file. The lazy.lua file handles the configuration of the plugin manager specifically, as shown above. In addition to the init.lua file for simple plugins, there are individual files for plugins that require configuration to meet your needs.

There was more work involved than I’d hoped in migrating to a pure Lua configuration, but there are a ton of examples out there and the Neovim documentation is pretty good.

You can also see that I moved the lsp configuration into its own directory. The lspconfig.lua belongs to the nvim-lspconfig plugin which contains configurations for the nvim lsp client. It can get quite large so I’m thinking about moving out the configuration for individual languages into their own files. The other plugin there is mason which is used to configure external tools like linters, formatters, etc., through a single interface.

Of the less specialized plugins themselves, there are only a few interesting items. I’ve never used dressing, tree-sitter, telescope, or which-key. I saw dressing looking around at nvim plugins and liked the way it looked so I thought I’d try it, nothing really functional was added there.

I’ve heard good things about tree-sitter and the plugins that wrap it, and other nvim users that I respect use it, so I thought I’d try that and I like the idea of some additional text objects.

People have been raving about telescope for years. I’ve used fzf for years and have been completely satisfied with it as a tool for finding files, but I’ve been working almost entirely with familiar code bases. It was postulated to me that for getting around large, unfamiliar code bases, fzf is as much hindrance as help. This seems reasonable so I thought I’d give it a shot and give me an excuse to start looking at new codebases. I still have fzf.vim configured though because I love using it to find buffers. I may change my mind later and go all in on telescope. We’ll see.

I’ve never used which-key either but I figured with all of the new plugins, something that gives me a menu for keystrokes will be helpful until I have them in muscle memory.

These were just my ramblings on re-configuring my nvim setup. Let me know if you have any questions, recommendations, or comments.

Kafka Tiered Storage for the Masses

If you’re a Kafka fan you may have heard that Apache Kafka 3.6 RC 0 implements tiered storage for Kafka. This means a preview will be available in the opensource Kafka.

Until now tiered storage was not supported natively in Kafka and has only been available using third-party tools or modules, most of them proprietary, such as the Confluent tiered storage module, and Amazon Managed Streaming for Apache Kafka. This means that if you wanted tiered storage, you were almost certainly locked into a vendor paying for it.

With the release of 3.6 RC 0, tiered storage is now available to the masses. This may sound small but in reality, it’s a pretty big deal, and I believe that we’re on the precipice of a true paradigm shift in how we approach streaming data.

Why This is Important

Kafka has become a huge part of the streaming data ecosystem and is arguably one of, if not the, most common entry point of all data into a distributed system. Data is commonly consumed in real-time, but there’s also the capability to fetch historic data as well, as long as it’s within the retention policy parameters. Here is where Kafka has traditionally shown its weakness. To understand this weakness we need a little background.

How Kafka Works (High Level)

Kafka stores messages in append-only log segments on the brokers' local disk. The retention policy for these logs, configured by topic or system-wide, determines the oldest message that can be fetched. This mechanism is used to guarantee that consumers won’t miss messages even if the application fails or otherwise loses connectivity.

The Storage Issue

The total storage needed for the cluster is proportional to the number of topics/partitions, message rate, and the retention period. As a grossly over-simplified example without getting into topics, partitions, replication factors, index size, etc., the approximate storage size per node of a hypothetical cluster with 12 nodes, a 3 day retention policy, 10k messages per second, and a message size of 1k, can be determined using the following method:
Storage (in bytes) = (Messages per second) * (Message size in bytes) * (Retention time in seconds)

This gives us a storage size per node of over 2.59TB. Granted, this isn’t a huge amount of storage, but that example is not a large Kafka cluster. As an example, LinkedIn claims to run over 100 Kafka clusters with more than 4,000 brokers, which serve more than 100,000 topics and 7 million partitions, resulting in over 7 trillion messages processed daily. Using the grossly oversimplified approach we used earlier and making some naive assumptions we can get a better idea of what that kind of scale looks like.

We’ll assume a 1k messages size, that the 7T messages are evenly distributed between clusters over a 24 hour period and a 3 day retention policy, and that it’s something like 100 clusters of 40 nodes each. Using the previous formula that would be around 190.3TB per node. The naive storage size for one of those clusters is over 7.6PB and we’re assuming a 3 day retention period.

Compounding Storage Issues

We’ve seen that the retention period has a huge effect on the storage needs of the cluster but that’s just the start. Kafka cluster storage is typically scaled by adding new nodes to the cluster but this lowers the cost efficiency because we’re adding an entirely new machine instead of just adding the storage by itself.

In an ideal world, we would keep the data indefinitely but due to inherent constraints around storage, it’s just too impractical to do. To add more complexity to this scenario, when a broker fails, the failed node is replaced by a new node. The new node must then copy all of the data that was on the failed broker from other replicas.

Let’s add a little more scaling complexity; when a new node is added to scale the cluster storage, cluster rebalancing assigns partitions to the new node which also requires copying of data. The time for recovery and rebalancing is proportional to the amount of data stored locally on a Kafka broker. When you have a lot of clusters running 100s of brokers, a node failure is a common occurrence, and a lot of time is spent in recovery adding stress to the system.

Full Circle

Tiered Storage is the Solution

Kafka messages are typically consumed in a streaming fashion using tail reads. That is to say that it’s leveraging the OS-level page cache to serve the data, not the disk. Older data is typically only read from the disk infrequently for backfill (lagging consumer) or node recovery purposes.

Tiered storage in Kafka configures two tiers for storage; local and remote. The remote tier uses S3, GCS, or some other arbitrary external, long-term storage service. The two tiers configure separate retention policies allowing the local storage policy to be reduced significantly, while the remote storage can be configured to retain data for as long as desired.

Storage has now effectively been decoupled from the broker, and by reducing the local retention policy we’ve reduced the amount of local data to be copied for node replacement and rebalancing.

This results in longer data retention, lower costs, faster scaling, and faster rebalancing.

Implications of Tiered Storage

Once a new feature gets released into the wild, people start playing with it in unexpected ways. Sometimes, if the thing changes a fundamental assumption, it grows and changes, and gets used in ways that are obvious only in hindsight. I feel that tiered storage in Kafka is one of those features. There has been a fundamental assumption by Kafka users that you never ever use Kafka for anything but recent data. It’s too expensive, it’s too difficult to manage. So they make Kafka the backbone of their streaming data and offload new messages as quickly as possible into other systems. They will transform the data and then push that data into other systems more appropriate for long-term storage, relational DBs for transactions, OLAP DBs for analytics, etc., creating the complex event-driven big data pipelines we all know and love. Tiered storage gives the middle finger to that assumption.

I’m not saying that existing ETL pipelines are going to go away. What I am saying is that people are going to begin experimenting with what’s possible. Can I have a Kafka cluster with 1PB of remote storage which can be accessed as needed? Can Kafka now become my single source of truth and distribute messages on demand to anyone who wants to query it?

Here’s a scenario to think about; Let’s assume that we have a Kafka as the entry point of all data into our data pipelines and we’ve enabled tiered storage, so we can scale extremely efficiently. What do we do with all of that data that we can now store cheaply and efficiently?

Let’s take our row-based data from Kafka and transform it into a columnar-based format like Parquet, using Flink or Spark, and dump that data into the Iceberg Open Table format which can then be queried directly.

We’re talking about merging the ingestion and storage layers and with some simple transformations, we’ve got data we can query by column or row. How many steps in the data pipeline can we eliminate or simplify?

What if the remote tier were directly transformed into Iceberg tables? I do think there is a paradigm shift coming in the streaming data ecosystem. It’s like seeing the outlines of a person through the fog, but you know it’s there.

Then again, I could be completely wrong. Who knows?

I am not a Kafka expert, but I do work with it on an almost daily basis. Recently, one of my clients told me they were considering implementing a feature that would double their Kafka workload and asked if I could support them by helping right-sizing a new cluster. The ensuing conversation uncovered some questions I didn’t have answers for and I thought I’d share my search for one of those. Namely, had the client already saturated their consumer groups.

What is a Consumer Group?

In Kafka, a consumer reads messages from the broker, for example, temperature data published to a topic on the broker (by a producer), is read by an application that analyzes the data for trends (by a consumer). Simple enough.

However, if there are 5,000 sensors publishing data to that topic every minute, my single app may not be able to read the messages fast enough and a backlog starts to build up. In this case, one solution might be creating multiple consumers to process the data in parallel. Enter, consumer groups. Consumer groups allow multiple consumers to read from a single topic without stepping on each other's toes. It lets the processing of messages be parallelized across multiple consumers to provide horizontal scalability.

What is Consumer Group Saturation?

Consumer group saturation is pretty much what it sounds like. Just like a single consumer can get overwhelmed by the volume of messages it has to process, so too can a consumer group. Basically, messages are coming in faster than they can be processed and the message backlog grows faster than they can be consumed. This is referred to as saturation.

Saturation can occur for a bunch of different reasons but the most common ones can be grouped into

• Not enough consumers in the group

• Network latency

• Consumer group rebalancing

• Consumers are slow/inefficient

The not-enough-consumers issue tends to sneak up when you’re not paying attention and don’t have good monitoring in place. When you’re message volume increases over a long enough time period that the trend goes unnoticed, you’ll find that your backlog will grow sporadically, and eventually consistently until eventually you’re never able to catch up.

Network latency tends to be transient but can become a more persistent issue if you’re already on the edge of saturation as it will create a larger backlog and if you’re already on the edge of saturation you may not be able to catch up before something else happens to create a saturation event. These types of network events can exacerbate the situation outlined above.

Kafka topics are divided into partitions. Partitions are used to distribute the load across the cluster by assigning a different partition to each broker in the cluster. When a consumer group adds a consumer that consumer is assigned a subset of partitions that it will read from. When a consumer is added or removed from a consumer group, Kafka will rebalance the consumer group to evenly spread the load, i.e., it will reassign the partitions to each consumer. And while the rebalancing process does its best to cause as little disruption to consumer processing as possible, in reality, a small storm of controlled chaos results. This means that while the rebalancing is in process some consumers may be sitting idle while others are working on overload. Adding and removing consumers frequently will create havoc on topics with high message throughput.

Much like not having enough consumers in the group, slow/inefficient processing of messages tends to sneak up on you over time, only in this case, it's probably due to application/code changes in the consumers that are processing the messages, rather than in the number of messages to process. Of course, you could turn that on its head and say that instead, it’s an issue with code optimization rather than consumer group saturation, but from Kafka’s point of view (and likely your DevOps/SREs perspective), “po-ta-to”, “po-TAH-to”.

How Do I Know It’s Saturated?

Before I throw out some key indicators, I want to take a moment to point out the obvious, but you need to establish a baseline for your performance metrics because as is the case with almost everything, each use case has its own peculiar needs. So make sure you know what it looks like when behaving well.

Assuming that you know what these metrics look like under normal circumstances, here are some things you can monitor to determine if your consumer groups are saturated, or headed that way.

Consumer Group Lag:
What does your consumer group lag look like? The difference between the current consumer group message offset and the offset in the topic represents the number of unprocessed messages for a consumer. Automated monitoring implementation is beyond the scope of this article but the most convenient way to determine the consumer group lag in an ad-hoc fashion is by using the Kafka Consumer Offset Checker tool included with your Kafka installation. 
If the number of unprocessed messages is growing or sits at a high threshold without shrinking, you’re probably saturated.
Be sure to monitor the lag rate, i.e., is the consumer group lag growing or shrinking over time? You can use this to determine trends and see if you’re approaching saturation before you start suffering from it.

Consumer-based Metrics:
How fast are you consumers processing messages? If the time is constant but consumer lag is increasing, you’re saturated.

Resource Usage:
High CPU and memory usage may indicate that the consumers are struggling to keep up with the load.

How Do I Fix It?

Monitoring:
The best medicine is preventative. Avoid getting into the issue in the first place with proper monitoring. Make sure that you are monitoring trends that can alert you if, for example, you see that over the past month your message throughput has grown by more than 10 percent.

Avoid Consumer Group Rebalances:
Are consumers rebalancing frequently? Rebalances are typically caused by adding or removing consumers from a consumer group. Avoid doing this on a frequent basis as it can wreak havoc with your message processing.

Slight Tangent
Remember that partitions are how Kafka spreads the load over the cluster allowing for scalability and parallelism. This leads to some potential gotchas.

• If you have more consumers than partitions in a consumer group, some of the consumers will sit idle.

• If there are more partitions than consumers, some partitions will sit idle and some consumers will be assigned multiple partitions.

Consumer groups and partitions need to be deliberately balanced. On an efficiency note, you can never remove partitions, you can only add, so if you’re unsure about the future, starting smaller is better.

Add More Consumers:
If you’ve determined that a lack of consumers is the source of saturation, one of the easiest ways to deal with this is to simply add more consumers to the group. Just be sure that you balance the number of partitions with the number of consumers.

Slow or Inefficient Consumers:
If you’ve determined that the consumers are too inefficient, then it’s time to look at optimization or re-architecting how your messages are consumed and/or processed. Go figure.

Network Latency
On the one hand, there’s very little most of us can do about this one other than change our provider. The good news is that as far as businesses are concerned, it’s one you’re least likely to encounter as a persistent issue.

If you found this quick overview of saturation useful please let me know by clapping or subscribing so I know what you like. If you hated it, or have any comments, corrections, or additions, please leave me a comment and I’ll try to use that feedback in the future.

Quick and Dirty

I do a lot of work in the cloud so moving my development environment there makes sense for some things. In addition, it makes my development environment very consistent which is important for things like benchmarking as I can reproduce the exact hardware, control bandwidth, and in general, have more precise control of my test box.

This article is mostly just to show how I create a quick and dirty development box in GCP since I was asked this question. I like the command line so there’s nothing fancy about this; no IDE installs, just a simple how-to on creating a VM that runs a container with the tools you choose. If you want to make it a bit more complex, feel free to add as many bells and whistles as you like.

First..a Dockerfile

As a starting point, for various reasons, I use podman instead of Docker, so I won’t be using Docker as the authenticating client for GCP, but more on that later. First I need to build a docker image so I have a Dockerfile that looks like this:

FROM rust

ARG USERNAME
ARG UID
ARG UGROUP
ARG GECOS

RUN apt-get update -y && apt-get install -y --no-install-recommends \
zsh \
git \
curl \
neovim 

RUN groupadd -g ${UGROUP} ${USERNAME}
RUN adduser --force-badname --shell /usr/bin/zsh --uid ${UID} --gecos "${GECOS}" --gid ${UGROUP} ${USERNAME} --disabled-password
RUN chown -R ${USERNAME}: /home/${USERNAME}

USER ${USERNAME}
RUN cd ~/
RUN sh -c "$(curl -fsSL https://raw.github.com/robbyrussell/oh-my-zsh/master/tools/install.sh)"

EXPOSE 28578

This is pretty straightforward. In this case, I’m using the official Rust image on Dockerhub but this can be replaced with whatever base image is appropriate, e.g., python, OpenJDK, etc.

I pass in some arguments to the build command to create the user and group based as well, as you will see shortly.

I then update apt-get and install my base preferred tools for the CLI, configure a group and my user, and switch to that user before installing OMZ. I expose port 28578 because I happened to be working with Redis when I created this one.

Prep for pushing

Using podman I build the file and tag it like so

podman build \
--build-arg USERNAME="$(id -un)" \
--build-arg UID="$(id -u)" \
--build-arg UGROUP="1000" \
--build-arg GECOS="$(id -F)"  -t us-east1-docker.pkg.dev/aiven-sa-demo/clays-docker-registry/cr-rust:1.0 .

I pass in the build arguments based on my user account to the build command using the id command. I could have passed in the UGROUP value as well but chose to do this as a hard-coded value.

After that, I authenticate against the GCP Artifact Registry using this command: gcloud auth print-access-token | podman login -u oauth2accesstoken — password-stdin us-east1-docker.pkg.dev/bogus-project/bogus-registry where you would paste your registry information in place of the bogus one listed in the command clip. The tag needs to be in the form LOCATION-docker.pkg.dev/PROJECT-ID/REPOSITORY/IMAGE_NAME on your local image.

Push It Real Good

Pushing the image feels sort of anti-climatic if you were expecting anything clever. It works just as you’d expect it to; podman push us-east1-docker.pkg.dev/aiven-sa-demo/clays-docker-registry/cr-rust:1.0 just like its docker counterpart.

Actual deployment

The most anit-climactic part has come. Now we need to deploy the container in a VM. To do that, you use the create-with-container command like so:

gcloud compute instances create-with-container redis-experiments \
 — container-image us-east1-docker.pkg.dev/aiven-sa-demo/clays-docker-registry/cr-rust:1.0

That’s really all there is to it. You now have a development environment deployed to the cloud and containerized. There are clearly a lot of improvements that can be made, like adding configuration files for your tooling or putting all of this into a source code repository and automating the deployments for example, but that’s the gist of it.

Final Note

Some of you may have wondered why I passed a hard-coded value for the group ID to my build command. The steps I outlined above work great in most situations. However, the machine I wrote this article on happens to be a Mac. If I’d used the “$(id -g)” to get the group ID of my account, it would have returned 20. The low numbers that are used on Mac for user accounts would have conflicted with the group IDs used by Linux so I passed a semi-arbitrary high number.

Just to make it more fun, if you’re on a newer Mac using ARM architecture, you won’t be able to create a container from an image built on your Mac because of the architecture differences. Unless you force the VM type to be compatible you’ll need to move the build process itself to the cloud or find a non-ARM machine to create the image.

If you made it this far, give me a clap or a follow to let me know if you found the article useful, and comment below if you have any questions, observations, or find any issues!

Windows Loves Rust

At BlueHat IL 2023, in Tel Aviv on March 29-30, David Weston, the director of OS Security for Windows, announced that Windows would be booting with Rust in the kernel soon, “in probably the next several weeks or months”. That’s pretty exciting to Rust fans out there.

It’s no secret that Windows has been experimenting with Rust. The love affair started years ago. For those of you who don’t know, Rust is a programming language designed with memory safety as one of its primary goals.

It achieves this through a combination of approaches. In simplified terms:

1. Architecturally — the language uses a concept of memory ownership where any allocated memory is owned by a single variable (but can be borrowed as read-only).

2. Borrow checker — a component that enforces the ownership model outlined above, ensuring that no memory is accessed after deallocation while also ensuring that there are no data races where multiple threads attempt to access the same memory concurrently (not general race conditions).

3. Deterministic lifetimes — the lifetime of a variable is known at compile-time. This allows the borrow checker to enforce ownership and borrowing rules and ensures that as soon as allocated memory is no longer needed, it can be automatically freed.

4. Immutable — by default, all variables are immutable helping to prevent inadvertent issues with memory safety.

According to MS memory safety accounted for more than 70% of the security-related updates released over the last decade plus. While MS is never going to completely rewrite 40-plus years of one of the most complex code bases in the world, it does make sense that it would begin experimenting with replacing especially vulnerable code given that Rust was created in part to address this very issue.

It’s already begun

The first port they’ve introduced is to deprecate DWrite (DirectWrite), part of Windows font parsing code, in favor of DWiteCore. DWriteCore is a cross-platform version of DWrite written in Rust that is now the recipient of all new feature development this is currently in use. They’ve also implemented a portion of Win32k GDI by implementing the Region data type, a core component of the Windows kernel responsible for window management. This component is currently shipping but disabled behind a feature flag, which is due to be removed for insider previews “shortly”.

So what’s the problem then?

All of the above is absolutely fantastic news for fans of security, Rust, and Windows OS. The potential issue is the Rust Foundation itself.

On April 6th, just a week after the announcements at the conference in Israel, the Rust Foundation released a new draft of its trademark policy and invited comments via a Google form giving a deadline of April 16th. Because it's in a Google form the comments aren’t publicly visible, but developers took to any and all social media to make their concerns known. And holy smoke, were those concerns voiced. People came out of every corner of the internet to comment, YouTube, SubStack, Reddit, Twitter, and basically every other forum you can think of. It even reached the point where the Rust community created a fork of the language.

For the sake of brevity(-ish), the highlight is this; after a year of deliberation, the Foundation published a final draft of a new trademark policy which essentially states that you cannot use the name “Rust” or use the Rust Logo, for any kind of profit or gain, register a domain name without explicit permission from the Rust foundation.

That’s right, you can’t use the name or logo at all without explicit permission in a domain name (for profit or not), or in anything that might make a profit or gain. Anywhere. For any reason. Have a domain name that uses the word “rust” in reference to the language? Denied. Your YouTube channel about the Rust language has the word Rust in it? Denied. You want to make a t-shirt promoting rust by using its logo? Nope. Have a non-profit website to teach Rust that has a domain name with the word rust in it? Not anymore.

And for more fun, this was also part of the new policy:
“Using the [word “Rust”] in the name of a tool for use in the Rust toolchain, a software program written in the Rust language, or a software program compatible with Rust software, will most likely require a license.”

If you’re wondering what that means, it means that any software that’s any part of the Rust ecosystem is banned from using the word rust. That’s IDE plugins, libraries like openssl-rust, rust-mysql, and every other tool, crate, or library you can think of.

What does this mean?

Maybe nothing. On the one hand, MS made the announcement at BlueHat IL 2023 prior to the release of the draft trademark policy and I can see this kind of insanely restrictive policy making them extremely nervous about using the language, certainly about promoting their use of it.

On the other hand, MS is one of the founding members of the Rust Foundation, along with Amazon (who also is promoting the adoption of Rust), among others. It seems like a big pill to swallow that MS wasn’t aware of this insanely restrictive policy but since they’re one of the founding members one would assume that they can probably get explicit permission to do whatever they want with the name and logo.

The day after the close of comments the Foundation issued an apology for its lack of transparency in the process, and that they will carefully consider the feedback.

So what’s my point?

I don’t think MS is going to have any issue with the trademark policy because they are a founding member and I can’t imagine them not being able to get permission for whatever they want to do regarding the name and logo. What I do see are three possible positions that MS could occupy regarding the trademark policy.

1. They had no idea what was going on because they didn’t care or were ignorant. We can throw the ignorant option out immediately by looking at the board of directors.

2. They actively support the new policy

3. It’s a genuine mistake that the wording was so insanely restrictive and it will be a nothing burger when it's all said and done.

In the first case, they don’t care, then it's just a community problem and MS may or may not have to deal with some bad press for not pushing back.

In the second case, you really should start asking yourself, “Why would MS support this policy?” But let’s not stop there, that’s a valid question to ask of Amazon, Google, Meta, and the other founding members. What’s in it for them if this trademark policy stands? Who benefits from restricting people from using the word “Rust”, or the Rust logo?

In the third case, this will be something we all laugh about someday. This option I find about as likely as the board members being ignorant of the new policy. First, there was nothing to fix with the original trademark policy and even assuming it need to be refined, there was definitely no reason to suddenly roast everyone over a roaring fire. Second, the corporate member directors are far from being idiots or oblivious, and they are well equipped to understand the community's reaction. The fact that all of them missed out on what the reaction was going to be seems another tough pill to swallow.

This brings me back to the second case. Who benefits and how? I don’t see any clear answer to that but I’ll be very interested to see what the updated policy looks like when it comes out.

If you made it this far please clap, or subscribe so I’ll know you found the article interesting. If you want to read more like this or are interested in anything in particular, please comment and let me know. As always, thanks for reading!

Intro

Working with the CLI tends to be a very opinionated environment. Everyone has their own preferred tools. Everyone’s workflow is slightly different depending on personal preferences, the tasks they regularly perform, and even workplace culture. But regardless of preference, culture, or daily tasks, some things remain common.

Overview of CLI Usage

When using the CLI you’re always working with text and you always want it to be as convenient as possible. There are also common scenarios that you’re dealing with on a regular basis. Broadly you can lump these scenarios into working with formatted documents (code, LaTeX, JSON, etc.), working with logs, and interacting with the shell. There is admittedly overlap between the first two scenarios, i.e., logs are certainly formatted, and some are written in JSON, but we interact with logs differently than we interact with other formatted documents; with logs, we’re extracting information and they’re the last thing we want to edit, with formatted documents, editing is almost always involved.

When we interact with the CLI we’re dealing with a serial execution environment, which involves blocking actions. If you execute something like tail then unless you do something to interfere, that shell interaction is now blocked. The same is basically true when you run any executable. Until that executable completes, you can’t interact with that shell session. This means that to work effectively you really need a way to work with multiple interactive shells simultaneously, be that through opening multiple terminals, using an emulator that provides a tabs feature, tmux/screen, etc.

Working with a Blocking Interface

My preferred tool for managing multiple interactive shells is tmux. This is something that gets pretty opinionated pretty quickly so let me outline my reasons for using it.

First, while tmux is a pretty new tool in terms of the *nix tooling, its popularity has grown enough over the last 15 years that if it doesn’t come pre-installed, then it’s easy to install with whatever package manager is available. Second, while it’s extremely customizable, it’s also usable out of the box. A vanilla install gives me the ability to manage multiple shell sessions, group screens together, pick up previous shell sessions where I left off, and much more.

I prefer this over using tab-capable emulators because I’ll almost always have access to it, whereas customizing iTerm2 to meet my needs means my workflow will be awkward on anything other than a Mac.

I also don’t rely on creating a new terminal for each session I need because it’s annoying to organize, and if I don’t have a GUI, I can’t create another terminal anyway. Think, logging into a Linux VM on a cloud provider like GCP or AWS. You can either use a browser to ssh into a terminal session, or you can just ssh from your local machine into a terminal session. If you don’t have some kind of multiplexor you’re stuck creating a new emulator for each session you need.

Regardless of how you solve the problem of working with a blocking UI, be it tmux, iTerm2, screen, or something else entirely, it’s something you’re going to contend with.

Interacting with the Shell

For the purposes of this article, I’m not going into which shell you should select as, again, this is a very opinionated topic. My shell of choice has changed over the years as my workflow, needs, and wants have evolved. Also, a good argument can be made for using any shell I can think of with the exceptions of the Thompson or Mashey shells, both called sh. Unless you’re working with Ancient Unix, you’ll never encounter them.

The most common and useful tools used while interacting with the command line follow the UNIX philosophy. Love it or hate it, what’s relevant to this discussion is that any tooling worth its salt will assume that its output will be the input to another tool that it knows nothing about. In other words, the vast majority of tools will use pipes to string multiple commands together to do new things.

For example, if I want to see all of the lines containing the word error, I can do tail file.log | grep errorand if the logfile is live, I can do tail -f file.log | grep error which will only display log lines if they contain the word error.

The examples above are clearly simplistic and maybe a tad contrived, but the point is that you will have a plethora of tools at your disposal and they can almost all be strung together to accomplish far more than any single one could. They’re legos, learn how to stack them together and you can save yourself a lot of time and pain. Let’s see an example that's a little more “real-world”.

If you work with REST APIs at all, you’re probably familiar with cURL which is used to send an HTTP request and show the response. A common pattern for a PUT command is to return a JSON payload. Unformatted JSON is ugly and difficult to read. We also want to play with OpenAI because that’s all the rage at the moment. First let's send the curl command so that we can verify the response is what we expect:

curl https://api.openai.com/v1/completions \
-H "Content-Type: application/json" \
-H "Authorization: Bearer YOUR_API_KEY" \
-d '{"model": "text-davinci-003", "prompt": "Say this is a test", "temperature": 0, "max_tokens": 7}'

This returned this ugly piece of JSON:

{"id":"cmpl-6SXlLQAk9smbBI2TuNjNCEdKH6FHW","object":"text_completion","created":1672260987,"model":"text-davinci-003","choices":[{"text":"\n\nThis is indeed a test","index":0,"logprobs":null,"finish_reason":"length"}],"usage":{"prompt_tokens":5,"completion_tokens":7,"total_tokens":12}}

Let’s pipe this to jq to pretty-print it:

curl https://api.openai.com/v1/completions \
-H "Content-Type: application/json" \
-H "Authorization: Bearer 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" \
-d '{"model": "text-davinci-003", "prompt": "Say this is a test", "temperature": 0, "max_tokens": 7}' | jq '.'

Which yields this much more readable version seen below:

% Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   383  100   287  100    96    318    106 --:--:-- --:--:-- --:--:--   425
{
  "id": "cmpl-6SXtWoZY29vj2hIK2PqdsvIe1jUoc",
  "object": "text_completion",
  "created": 1672261494,
  "model": "text-davinci-003",
  "choices": [
    {
      "text": "\n\nThis is indeed a test",
      "index": 0,
      "logprobs": null,
      "finish_reason": "length"
    }
  ],
  "usage": {
    "prompt_tokens": 5,
    "completion_tokens": 7,
    "total_tokens": 12
  }
}

By adding one simple tool, we’ve made it possible to read the returned JSON. In addition to pipes, redirections are another tool that makes your life easier. Redirection allows you to send output from STDOUT, which by default is the screen, to a file. A single redirection symbol ( >) will create a new file if one doesn’t exist, or overwrite an existing file if it does. Using two redirection symbols, >>will append to an existing file rather than overwrite. Now I can simply add >> sample_response.json after the jq command and the JSON results are in a file ready to use in an editor.

There are more tools out there than you can count, all of which can be used as building blocks to create results that are only limited in usefulness and power by your willingness to understand and use them.

Working with Logs

When it comes to working with logs, we’re usually trying to locate a specific part(s) of the log in some manner, e.g., time a period, a specific message type, a specific identifier (customer number, service ID, etc.), or some combination factors like the ones listed above.

The other common possibility is that we may be aggregating logs to discover trends. For example, maybe we’re looking for unauthorized login attempts over the past hour or we want to know how many connection timeouts have occurred on a particular service over the past two weeks.

Now, analytics and monitoring can obviously be done far more efficiently in a production environment if we use tools and services specifically built for those purposes such as Grafana, Prometheus, Datadog, etc., but there are occasions when it’s just easier to do investigations with the raw logs themselves with ad-hoc tools. Sometimes, raw logs are the only source of the data you need for your investigation.

Logs (at least good ones) will have a consistent format. Many logs are modeled off of the Linux log format which looks like this by default:

priority, timestamp, hostname, service name, message

Below is a snippet of the log output by sshd when a login attempt occurs and the user doesn’t exist. Note that in this case, the priority is not logged. As an aside, you’ll find that many logs omit the priority from the log messages. You can find more detailed information here.

Jul 7 10:51:24 chaves sshd[19537]: Invalid user admin from spongebob.lab.ossec.net

There are a number of common CLI tools that we can use to make studying logs easy.

Grep:

I think the first tool that comes to mind for most people is grep. Grep provides flexible and powerful search capabilities. The key required to unlock that power is Regular Expressions or RegEx. Grep's power comes from its ability to apply regex patterns to its search. I would also recommend using the -E flag instead of the -e to prevent a lot of extra backslash escapes which will be more error-prone as well as more difficult to read. As an example, the command below will search the syslog file for any entry that contains the text system76 regardless of what comes before or after it, as long as the text is followed by a colon and white space (“: “). It will also print the line number where it found the reference.

grep -n -E ".*system76.*: .*" /var/log/syslog 

Full details on the regex it understands and all command flags can be found here.

Head and Tail:

If you just need a quick peek at the start or end of the log, say, to make sure that it started or stopped execution without errors, then you’ll find head and tail useful commands. They simply display the first N or last N lines of a file. There’s a convenient flag that instructs tail to continue outputting any data appended to the file as it grows. This is extremely useful in a development environment. You can take a look at the man pages for them to get the full details.

Awk and Cut:

Since all useful logs follow some kind of consistent format we can take advantage of awk to parse them for specific information. While Awk is its own text-processing language, and far beyond the scope of this article, it can be a very useful tool for working with any document that has consistent formatting, not just logs. Awk is another tool that requires at least a minimal working knowledge of regex to be used effectively. Going back to our previous log example:

Jul 7 10:51:24 chaves sshd[19537]: Invalid user admin from spongebob.lab.ossec.net

The following command will locate the line using regex, and print the username and hostname that originated the failed request. Note that the comma between the 2 fields is required to separate the output:

awk "/.*Invalid user.*/ { print $8, $10}" /var/log/auth.log

The cut utility is used to select delimited text and print it to standard output. Note that despite its name, it does not alter the source file's text.

Using cut you can select text by character or byte position (or range), and by field using an arbitrary delimiter. As an example, we want a list of all accounts currently on the machine for a security audit. We know that we can find a definitive list of accounts for the machine in /etc/passwd. Let’s say our file looks like this:

root:x:0:0:root:/root:/bin/bash
daemon:x:1:1:daemon:/usr/sbin:/usr/sbin/nologin
bin:x:2:2:bin:/bin:/usr/sbin/nologin
sys:x:3:3:sys:/dev:/usr/sbin/nologin
sync:x:4:65534:sync:/bin:/bin/sync
games:x:5:60:games:/usr/games:/usr/sbin/nologin
man:x:6:12:man:/var/cache/man:/usr/sbin/nologin
lp:x:7:7:lp:/var/spool/lpd:/usr/sbin/nologin
mail:x:8:8:mail:/var/mail:/usr/sbin/nologin
news:x:9:9:news:/var/spool/news:/usr/sbin/nologin
uucp:x:10:10:uucp:/var/spool/uucp:/usr/sbin/nologin
proxy:x:13:13:proxy:/bin:/usr/sbin/nologin

If we want to create a file with a list of all of the accounts created on the machine we can do something like this which selects the first field from the file using the colon (:) as the delimiter, and writes the result into the file called account-list.txt:

cut -d ':' -f 1 /etc/passwd > account-list.txt

The result would be a file containing this:

root
daemon
bin
sys
sync
games
man
lp
mail
news
uucp
proxy

There’s a lot of information out there for both of these commands but here’s the man page for both awk and cut to get you started.

Working with Formatted Documents

I’ve saved this for last because it’s both a very simple and very complex topic. While all of the tools previously mentioned can certainly be used to work with formatted documents, there are tools designed specifically for editing text.

Sed:

Sed is a stream editor that allows you to perform insert, delete, and substitution operations based on selected text. A quick example using the input from /etc/passwd used in the cut command above might be to flag all service accounts since we know that all accounts that have /user/sbin/nologin instead of a startup shell are going to be service accounts.

A word of caution here; we’re now starting to get into editing, which can be dangerous if you’re talking about system files like /etc/passwd so I recommend always using the -i flag when using sed on system documents. the -i flag allows you to specify the creation of a backup before any editing occurs. Even better, don’t edit system files directly, just create a duplicate and perform your edits on that. Then make a backup of the original anyway in case something breaks when you start using the new one.

The example below will create a backup of the original file called passwd.backup, read the contents of the original file, replace everything between the first and last delimiters (:) with a space, and replace the /usr/sbin/nologin with the text Service account.

sed -i'.backup' -e 's/:.*:/ /g' -e 's/\/usr\/sbin\/nologin/ Service account/g' cut-test.txt

After this command executes the file will look like this:

root /bin/bash
daemon  Service account
bin  Service account
sys  Service account
sync /bin/sync
games  Service account
man  Service account
lp  Service account
mail  Service account
news  Service account
uucp  Service account
proxy  Service account

As you can see, while I was expecting to see the login shell of any non-service account users, I did miss the sync account, which..well, it’s an artifact of an earlier era (pre-1993) that was used to safely shutdown systems without needing to know the root password. I don’t know of any modern systems that still use it. If I’m wrong, please let me know in the comments. Always happy to learn something new!

Also, note that if you want to take full advantage of sed you’ll also want to be comfortable with basic regular expressions.

JQ:

We already briefly met jq already in the, Interacting with the Shell, section. If you work with JSON chances are you’re already familiar with jq. It’s a utility for parsing and formatting JSON documents. The . operator by itself represents the root of the document. You can use the . operator to reference the children of the root document. Given a JSON document called pets.json which looks like this:

{
  "pets": {
    "cat": {
      "name": "Mr Fuzzy",
      "color": "calico",
      "age": 8
    },
    "dog": {
      "name": "Sir Barky",
      "color": "blonde",
      "age": 12
    }
  }
}

Then this command, jq . pets.json, will render the entire document in a prettyprinted format which would look identical to the raw document above. This command, jq .pets pets.json, will render the pets field and the child fields of cat and dog. This command, jq .pets.dog, will render just the dog field as seen below:

{
  "name": "Sir Barky",
  "color": "blonde",
  "age": 12
}

It also allows you to work with JSON arrays (including array slices), built-in functions for filtering, mapping values, transforming documents, and, you guessed it, regular expressions can be used to match with the test function to determine if an input matches the expressions criteria.

Again, the entire topic has is too large for this article and there are already tons of articles that dive into jq usage. Here’s a link to get you started.

And since this could continue to go on forever, I’m going to end it with both the easiest and most difficult tools to recommend. Easy because it’s a no-brainer that you’re going to need it, and difficult because opinions are so strong as to be toxic on occasion. Terminal based editors

Terminal Editors:

Terminal editors a very opinionated topic. Everyone has a go-to favorite. When it comes to editors for the terminal almost everyone thinks of either vim or emacs. While those 2 dominate the terminal editor space, there are a surprising number of editors available for the terminal. Most of them I have never used, at least not in a very long time, but outside of vim and emacs, there are a few worth noting.

Nano — Nano has been part of the GNU ecosystem since 2001. It was created as a free replacement for the Pico editor, which was part of Pine, which was THE email client for UNIX systems at the time. Pico was the editor embedded in Pine used to create and edit emails. Its biggest advantage in my view is that it’s common to find it on many systems, it’s small, and it’s simple. Like a kinder vi.

Mcedit- Midnight Command is a fairly popular terminal file manager (well, it used to be, but I really don’t know anymore tbh). Mcedit is the built-in editor. The only reason to bother with this one is if you use MC. I definitely don’t think it’s worth installing MC to be able to use Mcedit but if you’re already a user of MC I think people sometimes forget it’s there.

There are many more alternatives out there, but I place a heavy weight on ubiquity and skill transference. If I put my time into learning a tool well, I want to make sure that the tool is available in as many environments as possible and that if possible I can transfer those skills to other tools.

Vim and Emacs:

The holy wars between vim and emacs are well documented as are the editors themselves, with countless tutorials available so I won’t go over them here.

The only comments I have to add about them are that they both meet my criteria of ubiquity and skill transference. They can both be found on virtually any platform by default, and the skills from both have some transference to other tools.

Regarding skill transference, all shells that I’ve used in the past 20 years have keybindings for both emacs and vi. This means that I can edit on the interactive shell using the same keystrokes I use for my editor. I can also use emacs or vim key bindings on my favorite GUI editors, e.g., all JetBrains editors, Eclipse, VS Code, etc. If only Word or Google Docs would do that.

With regards to vi/vim/nvim specifically, if you can use vim or neovim, you can use vi despite its age. You might miss the fancy bells and whistles the others give you but you’ll be fine. This is why people refer to them as if they were a singular item. I am starting to separate out neovim in my head from vim. With the announcement that vim will be re-writing its vimscript engine from scratch, rather than follow in neovim’s footsteps and use an already existing language to create plugins and custom scripts, I think it’s warranted. While we’ll always see backward compatibility between nvim and vim, at least for the foreseeable future, I think vim made a mistake and that it will suffer for it. Why reinvent the wheel and spend all the time, effort, and tears creating an entirely new engine from scratch, debugging it, etc., instead of just providing a means for an already existing language that is well-known, well-documented, and well-loved already?

Personally, I use neovim. My choice comes from two simple facts.

• It’s lighter-weight

• My first experiences working with emacs were frustrating and the keybindings hurt my hands after extended use.

• Bonus fact: When I first started trying emacs there were a lot of different versions, and compatibility between them was not guaranteed. That may have contributed to some of my frustrations.

As always, whatever tools you choose to devote your time to learning, do it deliberately and do it well.

There’s a reason the stereotype exists

There’s a reason that the image of the programmer sitting in front of a terminal emulator using arcane commands is an enduring stereotype. I’ve written before how the command line is inescapable if you’re a software developer. In almost any role that requires you to deal with the technical end of the software industry knowing how to use the command line is a big help.

There is nothing as nerdy-chic as being able to use command-line tools effectively. The bonus is that not only are CLI tools easily portable between environments, from cloud VMs to OS, but they’re also easily automatable. The biggest barrier to the efficient use of these tools usually comes down to people being intimidated or frustrated by them, both of which can be solved by taking the time to learn them. I won’t get into a big list of specific tools, there are all kinds of articles that already do that. Instead, I’d like to focus on what types of tools you really want to have under your belt. Most of these are probably self-evident but I try not to make assumptions.

The Terminal

The most fundamental part of using the CLI is the terminal emulator, usually just called a terminal. This is your gateway to the CLI. Regardless of what terminal you decide to use, you’ll likely be using it for a very long time, so try to be deliberate about your choice. Terminals can be as simple or complex as you like. They can offer things like tabs to keep track of multiple sessions, transparency to look nice, and built-in functions like search. I personally prefer a terminal that looks clean. I don’t want tabs or most of the other fancy stuff. All of those things can be achieved inside the CLI itself.

The Shell

The shell is the next most basic item that everything else is built on top of. Again, which shell you use is fine as long as you know why you’re using it. Be deliberate. The shell is going to be the means by which you interact with your tools, filesystem, and everything else. If you don’t have the time or desire to dig into all of the various shells with their specific pros and cons, my advice is to stick with bash. Bash is ubiquitous being the default shell for most Linux distros, and it’s available for practically every OS out there.

The Package Manager

The package manager is an often overlooked part of the CLI ecosystem. You’re going to have less flexibility around package manager selection because your primary package manager will be dictated by the OS in most cases. While you don’t get to choose the primary package manager used by the OS, you do get more flexibility in choosing the one that you will use for installing your own tools and software. This is a good idea because it separates the package management of your OS from the package management of your personal tools. If you’re anything like most developers you’ll be regularly, if not constantly, installing, updating, removing, downgrading, and in general, creating a big mess in the filesystem. This way you aren’t accidentally screwing with the libraries that your OS depends on. Maybe it’s me but I’ve been bitten too often by adding or removing something that it turns out the OS has a dependency on and things get annoying. I will make this instance the exception to my rule about not recommending any specific tool because there aren’t a lot of OS-independent, portable options out there outside of homebrew. The good news is that it’s available for Mac, Linux, or Windows(using the Windows Subsystem for Linux).

The Editor

Now that you have a terminal, a shell, and you can install tools, you’re going to need to start editing things, e.g., config files. I highly recommend finding a terminal-based editor because once you start seriously using the CLI you’re going to need it. Is it possible to edit locally with a GUI editor, load it onto my EC2 VM, and use it? Of course! That absolutely works. On the other hand, I would much rather simply open the file in situ, make my change, and move on without ever leaving the VM. I’ll leave the flame wars to someone else, I don’t care what editor you use, but I do recommend that it’s something commonly found on most systems, is powerful enough to meet your needs, and doesn’t take you out of the terminal. I also recommend that you learn it well as it makes your life much easier in the long run.

Everything Else

I spent a while trying to come up with more fundamental tools and finally came to a couple of conclusions.

1. Everything after the editor depends on your personal workflow

2. Your personal workflow won’t develop without deliberate focus from you.

Everything after the editor depends on your personal workflow

Any other tools that I would consider fundamental will only really apply to me. The tools you use and how you use them change depending on your experience, comfort level, and personal preferences. They will change over the course of your career, what you’re working on, and your evolving standards. At one point in my career, I would have declared a terminal multiplexer an indispensable fundamental tool. Then my specific work changed, my habits changed, and while I still use one, it’s not the must-have tool it once was. This can be said about many of the CLI tools that I’ve used over my career. Formatting tools, file managers, network tools, what you use and how you use it, all of it evolves and changes over time, driven by you and your own growth. The only thing I can say for certain is that no matter what I chose to learn, I never regretted investing the time to learn it and it’s always paid good dividends across my career.

Your personal workflow won’t develop without deliberate focus from you

That’s a misleading statement. It will grow and develop without deliberate focus from you. The difference is how it will develop. Without deliberate focus from you what will happen is that you’re leaving your professional growth to the roll of the dice. If you get lucky, you’ll stumble around and step into a pot of gold. If you aren’t, you’ll learn habits that will hamper you for years. The most likely scenario is that you’ll be a mediocre developer. And yes, this is true without the CLI focus. In my experience, the CLI is an amplifier for intentional learning. Six months of deliberate learning while using the CLI vs six months of flailing around on the CLI is the difference between taking thirty seconds to locate a file, and three seconds to locate a file.

Closing Thoughts

This may have been a little rambling but if you’ve made it this far, I appreciate your persistence. While writing this article, I realized that the biggest difference for e personally was deliberate focus and intent. Beyond using the CLI, or professional growth, being intentional in your pursuit of whatever it is you’re chasing is key. There have been times in my life when I lost that deliberate intent, in my work and my personal goals, and flailed about not really accomplishing much but not really losing ground. Just..stagnating. When I would wake up and notice I realized that I’d wasted a lot of my time getting nowhere. After regaining my focus I would accomplish more in a month than I had in the past six. Don’t be me. Keep your focus, be deliberate, and whatever you do, do it with intention.

If you made it this far, give me a clap or a follow so I know if you found this interesting or useful.

This post assumes that you already have the knowledge to get a working Kafka environment, e.g., you have at least a working Kafka cluster with Kafka Connect, or you have a working service hosted somewhere. For simplicities sake, you can sign up for a free trial account at Aiven. Setting up a Kafka service there is as trivial as a few mouse clicks.

I have recently begun working with data pipelines using Kafka to move data between various sinks and sources. For example, you might use a MySQL DB as a source of data and use Kafka to propagate that data into S3 for long-term storage, sending Kafka metrics to Influx for use by operations in tracking the server's health. I’m sure you can come up with scenarios that would make sense to you personally without much help. All of these sources and sinks are most conveniently tied together using Kafka Connect.

Kafka Connect relies on a REST API for the configuration of connectors. I love REST API’s. They can be convenient to use, allow a clean separation of architectural boundaries, and are conveniently agnostic when it comes to interfacing. However, when we’re talking about ease of development, I personally don’t like them, primarily because I either have to construct URIs by hand for every call or come up with some way of avoiding that task like pulling up the closest command from shell command history and tweaking it. Nothing insanely painful but it would be much easier if I had a CLI tool.

This is where kcctl comes in. This tool provides a CLI wrapper around the REST API which allows you to interact with Kafka Connect more conveniently. It takes the responses and prints them in an easily readable format to the terminal. The installation instructions for kcctl can be found at the GitHub repo link above. I will caveat the installation instructions with one item; as of the time of this writing, the brew install method was not working on my Mac M1 laptop. I suspect this is another M1 issue because installing with homebrew works fine on my Linux laptop. My suggestion for anyone having difficulty on a Mac M1 machine would be to either build from the source or try the precompiled pre-release version. You can find a link to the precompiled release in the repo README file.

Once the kcctl is installed, you set the server context for the tool. The context handles the server URL and credentials for your cluster. There are two ways to do this shown below.

kcctl config set-context my-cluster --cluster http://localhost:8083 --username myusername --password mypassword

kcctl config set-context 
--cluster=https://user-name:password@localhost:443 my-context

Both the commands above will set the URL and credentials of my cluster, as well as a name for the context, my-context in this case. The context name is required because kcctl remembers multiple contexts and needs a way to reference them. The sharp-eyed will notice that the context name is not a positional argument. To address the potential security concerns of storing passwords in the shell history, you can instead set them via a configuration file. The contexts you provide are stored in a dotfile in JSON format. You can find it in your home directory, in ~/.kcctl, which you can use to set sensitive information outside of the shell. One caveat on contexts, the tool is still young so there’s no convenient way of removing a context at the moment, you just have to remove them from the dotfile but the JSON format makes it fairly intuitive to do.

We can verify the cluster and any credentials using the kcctl info command which will return some basic information about the Kafka Connect cluster. The version number is the version of Connect worker serving the API (and its git commit) and the cluster ID is the ID of the Kafka cluster its connected to.

URL:               https://<USERNAME>:<PASSWORD>@localhost:443
Version:           3.2.0
Commit:            54454035f8b22b67
Kafka Cluster ID:  tArO_G21TQyIPS4A3BE1nw

Now that I’ve confirmed I’m connected, we can get a list of available plugins with the get plugins command.

kcctl get plugins

TYPE   CLASS                                                 VERSION
source com.couchbase.connect.kafka.CouchbaseSourceConnector  4.1.6
sink   com.couchbase.connect.kafka.CouchbaseSinkConnector    4.1.6
...

Let’s say I have a Postrgres DB created that I’d like to tie to Kafka as a data source. First, I’ll need to set up my configuration information.

{
  "name": "debezium-connector",
  "config": {
    "database.server.name": "demo-pg",
    "database.hostname": "demo-pg.host.com",
    "connector.class": "io.debezium.connector.postgresql.PostgresConnector",
    "name": "debezium-connector",
    "database.port": "28577",
    "database.user": "USER",
    "database.password": "PASSWORD",
    "database.dbname": "DB-NAME",     "database.history.kafka.topic": "DB-NAME-schema",
    "database.history.kafka.bootstrap.servers": "demo-kafka.host.com:28579"
  }
}

We then use the apply command to apply the configuration, but before we do that, we should check the validity of the configuration just to be safe. We can do this by using the --dry-run flag with the command. If all goes well, we should see the following:

kcctl apply --dry-run --file register_debezium_connector.json
The configuration is valid!

Now we can apply the configuration to register our new connector. We literally just remove the dry-run flag.

kcctl apply --file register_debezium_connector.json
Created connector debezium-connector

We can check on the connector by executing kkcctl get connectors which will return a response that looks like the following.

NAME                 TYPE     STATE     TASKS
 test-connector       source   RUNNING   0: RUNNING
 debezium-connector   source   RUNNING   0: RUNNING

If the connecter were failing the command would report that as well. We can simulate a failure by simply shutting off Postgres.

kcctl get connectors

NAME                 TYPE     STATE     TASKS
 test-connector       source   RUNNING   0: RUNNING
 debezium-connector   source   RUNNING   0: FAILED

If we use the describe command we can get a lot of details about the connector as well as the failure.

Name:       debezium-connector
Type:       source
State:      RUNNING
Worker ID:  demo-kafka-2.aiven.local:3000
Config:
  connector.class:                                    io.debezium.connector.postgresql.PostgresConnector
  incrementing.column.name:                           id
...
Tasks:
  0:
    State:      FAILED
    Worker ID:  demo-kafka-2.aiven.local:3000
    Trace:      io.debezium.DebeziumException: Couldn't obtain encoding for database star_trek
        at io.debezium.connector.postgresql.connection.PostgresConnection.getDatabaseCharset(PostgresConnection.java:476)
        at io.debezium.connector.postgresql.PostgresConnectorTask.start(PostgresConnectorTask.java:75)
        at io.debezium.connector.common.BaseSourceTask.start(BaseSourceTask.java:130)
        at io.debezium.connector.common.BaseSourceTask.startIfNeededAndPossible(BaseSourceTask.java:207)
        at io.debezium.connector.common.BaseSourceTask.poll(BaseSourceTask.java:148)
        at org.apache.kafka.connect.runtime.WorkerSourceTask.poll(WorkerSourceTask.java:305)
        at org.apache.kafka.connect.runtime.WorkerSourceTask.execute(WorkerSourceTask.java:249)
        at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:188)
        at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:243)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:829)
      Caused by: org.postgresql.util.PSQLException: Connection to demo-pg-cratliff1-demo.aivencloud.com:28577 refused. Check that the hostname and port are correct and that the postm
aster is accepting TCP/IP connections.
...

There was a great deal of detail left out but you get the picture. I much prefer working with Kafka Connect in this fashion. If you have a need or desire to work with Kafka Connect and your primary interface is the CLI, I would urge you to take a few moments and look over the repo.

Now that Docker has decided to begin charging a license fee for the use of Docker Desktop in large companies and/or commercial use, many people are scrambling to find open source alternatives.

This article isn’t about the available alternatives or my opinions on them (although I’ll do that as well). It's about clearing up some docker misconceptions that I’ve run into lately.

The qualifier “many people” refers to those who use Docker Desktop regularly. Anyone that falls into that category will also fall into one of two other categories; Windows users or Mac users. This is because Linux doesn’t have a Docker Desktop application. This fact brings us to what Docker Desktop has been hiding from you. Docker requires a Linux kernel to run. That means that to run a docker container in Windows or macOS you need a VM running a Linux kernel, which in turn runs docker.

The primary reason Docker Desktop (referred to from here on as DD) was created was to make your life easy by removing the need for you to care about managing a VM. If the default VM configuration is good enough for your needs it’s entirely possible that you may not even know that you’ve been running a VM this entire time. Given the original release date was about 8 years ago, it predates many developers' careers and they’ve never known life without it.

Now that people are reaching for alternatives the above situation has led to some confusion and a lot of taking reevaluation of how docker works. I am not going into detail on WSL+Windows and if you know enough to have opinions on that, you probably don’t need this article. So let's get the basics cleared up.

1. Docker requires a Linux kernel to run, version 3.1 or higher, specifically

2. Windows and macOS require a VM running Linux to run docker

3. DD manages the VM and the mapping of host machine resources to the container for you so that the VM to docker mapping is invisible to the user.

It’s a lack of awareness of this VM intermediary that has caused the most confusion that I’ve seen. After uninstalling DD then becomes a rabbit hole of finding an alternative. Once we’ve decided on one, we go down the rabbit hole of installing said alternative as well as a VM to run the alternative in. And while it may come as a surprise that you need to install a VM, at least with Windows this is made fairly clear in the installation instructions that I’ve seen. With macOS, most people are going to turn to a tool called homebrew to install their chosen alternative. Homebrew installs typically install all of their dependencies as well, so you still may not realize that it installed a VM when you installed colima, for example.

Once you’ve got everything installed and set up you are now going to have to set up the VM as appropriate for your particular needs. This also includes understanding that if I need to mount a volume in a docker container to the host filesystem, I will first need to configure file sharing between the host machine and the VM, and then configure sharing between the VM and the docker container.

My rambling point is simply this; be aware that DD is no longer hiding how docker works from you and understand that you’ll need to take a few extra steps to use docker without DD. Because of all of this, my advice is to continue to use DD unless there is a compelling reason not to (or you just want to play with things under the hood), but if you feel a need to ditch DD then be aware of the extra effort you’ll need to put in.

I use colima on my MacBook because it provides me the closest to a drop-in replacement for the magic that DD performs. I can quickly and easily change my VM configuration directly from the CLI when needed, while not forcing me to install kubernetes if I don’t need it (I’m looking at you minikube), it provides decent performance, and makes dealing with port mapping and volume mounting easy.

TL;DR

If you just want to see my take on what I like, jump down to the last 5 paragraphs.

The story of my distro hopping

In 1997 I had just exited from the military and started working at my first job as a professional developer. There I met a systems administrator who introduced me to Linux for the first time. The distro was Slackware, distributed at the time by Walnut Creek CDROM as, if memory serves, a set of 5 or 6 CDs.

Since that time, my mentality has shift from, “Linux is a really cool toy I can tinker with on weekends and evenings” to, “Linux is an inextricable part of my job and I can’t be spending hours or days puzzling out why some setting isn’t working as expected, or why my tool chain isn’t working right out of the box”.

In other words, my needs and requirements have evolved over the years. The same thing happened with hardware. I used to love to build all my hardware on my own, but the more valuable that my time became, the less patience I had with trying to correctly match hardware and assemble machines.

Fortunately, Linux distributions were evolving as well. Slackware was truly great as a toy or to test out ideas but it took so much tinkering for me to get it to work the way I wanted that I never considered using it as a work tool. A couple of years after i was introduced to Slackware and was wondering how I could use it as a working tool, another friend of mine introduced me to Red Hat. This thing was light years beyond Slackware, it even had a graphical installer! Although it still couldn’t replace Windows it was a major step forward for me.

This cycle repeated over the years, sticking with a distro that allowed me to do more and more work in Linux until something so definitively superior would come onto the scene that I had to switch. This switch would improve my corresponding user experience, workflow, and personal satisfaction each time.

For the past 5 years or so, Ubuntu has been my go-to Linux distribution. It was the best combination of user friendly without getting in the way of work. Up until a few months ago.

I was in the market for a new laptop and a colleague of mine told me about his preferred vendor for buying laptops. They make higher end Linux laptops and workstations. This vendor also supports a version of Linux that it tailors to it’s machines. He personally did not use this OS but highly recommended their machines (as do I).

I read a little bit about Pop!_OS on their website and was intrigued by what I saw (file system encryption by default for example) but didn’t really want to deal with an unknown distro, so I just went with Ubuntu (which you may choose as an alternative distribution) and forgot about it.

The laptop came, and I set about customizing it to me needs. I’d gotten a model with a dedicated Nvidia graphics card so that I could take advantage of CUDA for ML projects. If you’ve ever had to set up Linux with an Nvidia you’ve almost certainly had to deal with setting up proprietary drivers and tweaking the X configuration to get things working. This is where I started hitting my first frustration. There were a couple of other tiny things that were irritating me but the point here is that I was experiencing my normal frustrations at dealing with the hardware tweaks that Linux occasionally needs for anything that isn’t out of the box.

Eventually I threw up my hands and told myself, I’m already moving slower than I wanted, I might as well play with a new distribution anyway. So I contact support and asked if there was an upgrade path from Ubuntu to Pop which didn’t require losing all of the work that I’d already put into customizing my tools. The answer was yes, they gave me the instructions, and I was off to the races. It took all of about 4 days to realize I should have just had them ship it with Pop from the start.

Out of the box, you’re going to get a few things, like a version optimized for Nvidia or Intel/AMD. The respective versions are optimized for the specific GPU you are using. I didn’t need to make one tweak to have my Nvidia card set up correctly. Even getting the CUDA toolkit installed was a simple apt install which worked without a hitch.

Setup is straightforward and easy. File system encryption is just a choice during install (the default is yes). The install itself isn’t bloated. The interface is fairly clean. And while not a driving force, it is hands down the easiest Linux OS I’ve ever tried to play a game on. They release updates every 6 months in sync with Ubuntu on which it is based.

In a very pleasant surprise, customer support has been truly outstanding. They will answer every question no matter how detailed or simple. Interactions with them were a dialog with someone who actually understands Linux and the issues you’re likely to experience. When the laptop shipped they had even taken the time to correctly set up the SSD read/write configuration optimized. And it’s all open source so you can create a bootable USB to experiment with and see how it works out for you.

There were a small list of things that miraculously fixed themselves after upgrading to Pop! OS without intervention on my part but just as an example, if I left Chrome open overnight and the laptop went to sleep, it wouldn’t wake up. I tried disabling all plugins, ensuring I had the latest version, downgrading to previous versions, you name it, the only solution I found was simply killing the browser before the laptop went to sleep. This just vanished when I went to Pop. I still have no idea what the issue was.

Pop just stays out of my way when I don’t need it, it’s clean, things just work out of the box, it’s not bloated with things I don’t care about, it’s still super-configurable, and while default encryption may not seem like much it’s definitely important to me. Also, I really like the giant robot mascots.

I have heard anecdotal stories from the a number of people who experiment with creating art in multiple mediums that aside from physical vs digital, the most difficult switch to make is between watercolor, and other paints. Accomplished oil and acrylic painters initially have a hard time using watercolor, and vice-versa. On the surface these two mediums don’t seem so different, so let’s take a quick look at how these mediums are different.

There are 5 types of paints generally used in art. Oil, acrylic, encaustic, gouache, and watercolor. Of these 5 paint types, 4 share a common physical property; they are all opaque. This means that the colors that you see when looking at a painting done with those four are created because those paints will absorb the entire spectrum of light except the color that you see, which gets reflected back to your eyes. You can mix the paints themselves to come up with different colors to apply directly to the canvas.

Because of this opacity, paintings typically start with dark colors, and lighter colors are applied as the painting gets closer to completion. This makes opaque paints more forgiving, since, you can generally paint over mistakes. In particular, oil paints also dry incredibly slowly. We’re talking about 6 months to a year before completely dry. This can make oil very forgiving because you can work on a project for months without losing the capacity to make subtle changes to the paint that’s already been applied. Contrary to what many people may intuitively think, this makes them a good medium to use while learning.

Watercolor paint, on the other hand, is a different beast altogether. Watercolor is translucent, meaning that the color you see when viewing a watercolor painting, is the light reflected from the surface beneath the watercolor after its been refracted through the pigment of the paint. This means that the surface the paint is applied to is going to be an influence in what the finished piece looks like so selection of paper, and paper color, has an impact. This also means that darker colors are created by applying layers of paint until the desired color is reached. This also means that blending paint colors is typically done directly on the painting. If you blend the wrong colors, recovery can be difficult because each additional application of paint darkens the color, while simply applying more water to wash the paint runs the risk of saturating the paper and making it difficult to reapply paint in that area. This also means that the lighter colors need to be considered first and gotten correct the first time because each new layer of paint applied will darken the final product.

Ironically, this means that watercolor can be much less forgiving than the other paints. I personally think that the reason most peoples intuitive reaction is that watercolor is “easy”, “childish”, or “not real art”, is based on the fact that watercolor is what people have the most exposure. They get that exposure in art class when they’re just starting school. Schools choose that medium not because it’s easy to teach but because watercolor is the cheapest of the 5 types of paint, by definition it’s water soluble meaning it’s easy to clean up any mess, and lastly the kids typically aren’t going to get frustrated that their art isn’t exactly the way they want it. Children are much less concerned with “getting it right” than adults are. They’re just enjoying the act of creative expression.

So we can see that there is definitely a reason for the rumors that watercolor is a difficult medium to learn. I also mentioned briefly what I’ve personally observed as the mainstream reaction to watercolor as an art medium. I’d like to take one last moment to point out that watercolor is the oldest paint that we know of. Natural pigments were mixed with water, or even saliva, to create large cave murals by prehistoric man. The Lascaux Cave paintings date between 15,000 to 9,000 BCE.

Just because something is simple in concept, doesn’t mean that you should think it’s irrelevant.

Over the years I have noticed an interesting phenomenon when it comes to developers and the use of the command-line interface (CLI). In most mainstream media, the depictions of “hardcore” developers involve the subject sitting in front of one or more screens, surrounded by, and working with, massive amounts of text. There can be graphs, images, and all sorts of interesting eye candy on the screens but the meat of what the developer is doing is clearly some insane amount of text, usually scrolling by far faster than most people could process.

In the real world of development, at least in the mainstream, this is not the case. I know and have worked with (and still do), many very capable developers who avoid using the terminal at all costs. They’ll engage with it only when it’s unavoidable and get out as soon as possible. I feel confident in saying that a large number of developers today feel uncomfortable, and in some cases, intimidated, frustrated, lost, or all of the above.

To be sure, there are always exceptions. The outliers, that actively avoid graphical interfaces, eschewing modern IDEs for a terminal text editor. But that’s exactly what they are; outliers. The reality is that every person who works as a developer for any length of time has to deal with the command line to some degree. It’s simply unavoidable.

There’s a social aspect that’s often overlooked as well. Developers are still people, regardless of stereotypes or personal inclinations. Age discrimination in the software industry is a serious topic, and it does exist. I’ve been on the receiving end of it once or twice. But judgment based on perceived skill with the CLI is a thing. Now, to be clear, I’m not saying that age discrimination is a trivial subject or that judgment over CLI skills is on par with that kind of social issue. One of the reasons for that is because, in mainstream conversation with anyone not involved with software development, it rarely comes up, and when it does I’ve usually seen it handled like this: “That’s only a thing in the movies, real devs don’t work that way. In reality it’s not that exciting”, and the conversation moves on. However, I have seen developers in conversation go from exuberant to silent when topics involving knowledge of the CLI come up. It’s almost as if they have a sudden bout of impostor syndrome because they aren’t the CLI genius that pop culture demands superstar developers must be.

Having said all of that, the CLI is just another tool in your kit as a developer. The only reason that GUI-based IDEs aren’t intimidating is that it’s easy to poke around menu options, and icons representing certain tasks tend to be the same regardless of what the IDE is, e.g., “Run program” is almost universally represented by some variation of an arrow. Most people don’t worry too much about being judged for not knowing a particular tool, be it an IDE, a web application, or an office app, if they’ve never used it extensively. CLIs should be regarded in the same light. Actually, it’s much more difficult to master than an IDE or spreadsheet. It’s more like learning a new language. It’s so incredibly expressive, and has so many little bits, that no one really “masters” it, not completely. The same thing can also be said for GUIs but the big difference here, again, is the perception, or maybe expectation, is a better word. While people understand that there is a difference between the skill levels of someone that opens IntelliJ the first time, and someone who has used IntelliJ for years, no one judges them for that difference.

This brings me back to the point of this article. Using the CLI is unavoidable, and frankly, it’s becoming less avoidable as the year's pass. Since sometime roughly around 2010, the use of the terminal has become more commonplace. This interestingly coincides with the rise of cloud computing. At that time, AWS had launched EC2 a few years earlier and it was starting to become extremely popular. Microsoft launched Azure, Rackspace and NASA launched OpenStack, and Google introduced Google Compute Engine. Over the past decade, the mainstream use of all of these cloud entities has continued to grow at a breakneck pace. The thing that all of them have in common? You guessed it, command-line use.

The CLI has undergone a resurgence over the past decade. There are simply things that can’t be done in a convenient or timely fashion on those platforms without the use of the CLI, and the existence of tools like gcloud, AWS CLI, and Azure CLI, make it clear that the providers understand the need. Even before all of this, containers like docker require at least some use of the CLI to be productive. There are countless articles extolling the virtues of the CLI ad nauseam, I’m not going to preach to you that the command line is better. Go find one of them if that’s what you’re interested in. I’m simply saying unless your career as a developer is extremely short, or extremely specialized, you’re going to deal with the command line. Having said all of that, let me be clear; I’m speaking in the context of greenfield development. If I’m talking about automating a CI/CD pipeline, I’m probably using API calls.

Like articles espousing the superiority of ad nauseam the CLI, there are articles ad nauseam that tell you how to learn your way around a terminal interface. I don’t know you, your learning preferences, and IMO that’s a topic better suited to a book anyway, so I’m not going to speak to that. What I do want to tell you is that regardless of your personal level of knowledge, the tools available through the terminal are virtually infinite, and much like the universe, they continue expanding all the time. From the shell that you’re using to the commands you have access to, it’s all larger than you think. There are certain things that I personally feel are relevant regardless of what I’m trying to learn, or the medium I’m using to learn it. One of those things is, that truly learning something new, or improving on what you already know, requires a deliberate effort. For me, one of the things that means is, if I don’t know how to accomplish it with the tool I’m currently using, do I know that tool well enough? I’ll give you a concrete example from something I did this weekend that most developers have probably experienced in some form.

I had a set of personal goals I wanted to complete this weekend because I wanted to play with some technology I’d never worked with before, specifically Redis. I had two GCP compute engine instances set up that I wanted to use because I don’t have the hardware to run meaningful performance testing on my laptop. I wanted to install Redis on both instances, populate instance 1 DB with some data from memtier-benchmark, and have that information replicated on instance 2. I also wanted to write a rudimentary client that I could use to insert data into instance 1, and read that data from instance 2, simply to demonstrate the replication worked remotely and get a feel for the libraries. How did I access these instances you ask? I used SSH to connect. Now I’m firmly rooted in the CLI universe, running bash as my shell. I downloaded and installed Redis OSS on each instance. How did I download it, you may ask? I used wget. Compiling Redis OSS on the first instance took more than a few seconds so I wanted to start compiling it on the second GCP instance while the first instance was still compiling, but if I dropped my connection to the first instance, I would have killed the job and aborted the compile. So just killing the connection and logging into the second instance to start the compile would have been pointless. My solution? I used tmux to create a second session in my terminal window and used the new session to connect to the second instance. There are a number of ways to accomplish that same task, I was simply already using tmux so that was the laziest way and I could easily keep an eye on progress. Once they had compiled, the redis.conf configuration file had to be configured. I used vim for that. Now I had to run memtier-benchmark, the command-line tool for Redis perf testing, to load some data in and get a basic sense of expected performance. At this point, I needed to inspect the contents of both DBs to make sure that the contents had replicated as expected across instances. The way to do this was using the redis-cli command-line tool. Only one problem, I had run the servers from the command line. They were both sitting there throwing out logs to my terminal window. I could have just opened 2 more tmux sessions and had 4 ssh sessions going, 2 running a server, and 2 available for interacting with the command line. I had two problems with this; one, I have a peeve about too many windows open, it bugs me, and having 4 panes in a tmux session is a bit extreme for me to care about for more than a few minutes, and two, I knew linux had job control commands but for the life of me, the only thing I could remember was that I should have started the server using `./redis-server &` to start it as a background task. After a minute of googling to refresh my memory, I used the bash command `Ctrl+z` to suspend the process and get the command line back, then used the bash shell command `bg` to start the suspended process as a background job. This freed up my terminal to use start redis-cli to query the DB, and did not require me to open a new terminal.

If I had been uncomfortable with the CLI, at best it would have taken me significantly more than the 5-ish hours to complete all the tasks I had set out for myself. At worst, I may not have gotten any of my goals completed at all. I get that the terminal can be intimidating, there’s a lot of stuff to get familiar with, but if you look back at the afternoon I just described, you might see that terms “terminal”, or “command line”, hide a lot. The terminal is just the text interface I’m using to interact with the shell. The shell is just the interface that I’m using to interact with the OS. Tools like ssh, and tmux, are just the interface I’m using to interact with both the shell and operating system. We’re programmers, and to make good software we break things down into parts that can be easily composed. So to learn it break it down. I like to do it by deciding what my current issue is and asking if I can’t do it, is that because I need another tool or don’t know this one well enough? Is it my knowledge (or lack thereof) that makes this painful, or am I using the wrong tool for the job? What tool do I want to learn, what’s most beneficial? The shell? A terminal editor? I obviously can’t answer those questions for you. What I can say is that I think you should ask those questions of yourself and think about a good answer.