Managing command line history across multiple computers

This is a situation that I first began to create tools to improve on what looks about a good 7 years ago.

The problem is a simple one. As a command line junkie I've got a lot of commands that I run on my computer and we've got a number of systems we can use to collect this information (the shell provides a mechanism by default), but things become increasingly more fraught the more requirements we start to pile onto the problem.

• We might start to routinely connect to servers where we're spending a lot of time working. So all of a sudden our shell's history is missing a chunk of the work we're doing.
• bash's default behavior stinks! It easily and regularly overwrites its own history file.
  • First, history is totally lost on any unclean exits.
  • Also, the way bash history works is when it starts, it reads its history file in to memory, and starts appending commands you run to the list. Then, when bash exits, it will write this new list out to the file. You might see this as sensible and it may even be elegant, but this poses massive problems.
    • If you open shell A and then open shell B in a second terminal session, run 50 commands in shell B, close shell B, then close shell A, shell A will overwrite all of the history that shell B has saved. Note that none of the session closures had to be done uncleanly. The logic is simply lacking.
• Even if you configure sane history settings for your shell, it will periodically clear out the oldest entries. That seems like a good idea at first, since it's nice to eliminate this as a reason for your disk to be filling up, but, if you care about keeping everything, and why wouldn't you, it does not use up much space... This auto truncation makes it much more difficult to cleanly preserve the data far into the future, because you now have to track pointers inside your history file to know where to start syncing to back it up somewhere else.
• Dealing with this problem across multiple computers is pretty painful. Previously I had a system I made to handle this, and it worked well for a short period of time (a few years actually) but one of the things that I had to do on each system I wanted to track this way was install a cronjob. That in itself does not sound too onerous, but I also resolved to eliminate this requirement.

In July 2013 I came up with a nice solution that lives on to this day. The principles continue to be relevant:

1. I was using oh-my-zsh and i added a preexec() based alternative so I could append the commands I am running in the shell into a special new command history file. I also added additional metadata to this storage format. I used a silly text delimiter $@$ to separate fields, so it was like CSV except more like a DADSV (Dollar-At-Dollar separated values). That's because commas can clearly be easily found in shell commands. Pretty much anything. So I wanted to make something reasonably short but also weird enough not to see commonly. In terms of metadata I recorded stuff like:
   • working directory
   • timestamp
   • tty identifier
   • tmux session/window/pane ids I also made some escapement scheme for newlines in the commands since I was already at this time prone to experimenting with large shell incantations which often benefit from a little bit of formatting.
2. A Python script, which I named hist, that would consume this special ~/.zsh_enhanced_new_history file that I began using and renders it into a better human readable format. It would nicely render the command back out in a central column and keep the metadata from interfering with it. A semi-tabular way to review the data:

This runs fast enough that it doesn't get bogged down by hundreds of thousands of history entries, so I continue to use this thing.

Now that in itself helped a lot, but I needed to go further for multiple machines. I wanted to have a similar solution for bash also but I figured if I care enough to improve bash, I could just set up zsh. Now that turned out not really a great assumption either.

Originally my big hammer solution to multiple machines and trying to work with logs that self-truncate was the following system:

• I would use a git repo. I also called it hist.
• I would have a text file that lists all the shell command history files that I care about, and I had a shell script that would copy those local files into the git repo.
• Then that script would just automatically pull and commit. The idea is if e.g. .bash_history got hundreds of lines of history removed from the top of it, or if some stuff there got overwritten, then I may be able to have some snapshots in git of the intermediate state so I could use git pickaxe search to maybe find that content in the future somehow.

In practice this works, but is very inefficient. The git repo became impossibly huge.

So I finally hunkered down and made a proper attempt to do this the right way. I have only really the one North Star that I wanted to guide this one: Simplicity as an utmost priority. This means no databases, not even SQLite. I need text files, no more and no less.

So, the approach I'm taking is to enforce and rely on only ever appending. I will also double down on the preexec() hack I used for zsh by leveraging PROMPT_COMMAND for the same under bash.

I am making a small bootstrapper system that I can use to deploy this configuration into ~/.bashrc the moment i'm able to SSH into the machine.

The sync will be a hub and spokes model. The central computer is my main laptop machine and the process is the laptop has the cronjob that will cause it to go through and transfer the latest log content from configured machines via SSH.

This means I do a one time initialization on any environment that minimally interferes with its configuration or operation (just one line added to bottom of .bashrc) and from then on my laptop will be able to periodically fetch the latest command history content at any time that my laptop is able to reach it via SSH.