Input stream dumping

Finally I have enough groundwork laid to explain why this thing I just wrote is cool. Recently I've realized I want to power up the terminal sessions I use so I can use more than just tmux to view and interact with them. There are many aspects of tmux that are sort of limiting. One is that it's somewhat limiting that all my command history lives only in the tmux pane scrollback buffers. It is a little cumbersome to navigate and the performance is lackluster compared to a native terminal emulator scrollback buffer. And both approaches always end up with a fixed limited size of buffer, so if you want to grab some output that scrolled off the line limit at the top, well, that's just too bad.

I know that having a fixed buffer size for scrollback in a terminal is a very conscious design decision because, and this applies especially if you hack on shell scripts and do a lot of data manipulation on the command line, it's pretty easy to end up running a command that might dump out megabytes or gigabytes of data to the console, and trying to store all of it by default instead of letting it roll off the cliff may be undesirable.

I want to take a bit of a different stance on that, though, given the state of modern computing hardware. More on that later.

In the course of creating a minimal program to explore how I might get what I want, I also went ahead to see how to make a little helper program to help view the terminal program input stream.

It's likely the code will undergo more changes, but since I don't have it checked into its own repo yet for sharing with github I will just share code snippets in its simple as-is state.

I've always wanted to have a clean simple tool to help me troubleshoot more clearly what characters are showing up where. Typically I will just run cat or xxd inside the shell and call it a day as that is often enough to get the job done. It wasn't until now that I decided to get a program written that works a bit more cleanly.

hexflow.cpp

#include <iostream>
#include <iomanip>
#include <cctype>
#include <unistd.h>

static bool last_was_nonprint = false;

void print_byte(unsigned char c) {
    bool is_print = isprint(c);
    
    // Add space when transitioning from non-printable to printable
    if (is_print && last_was_nonprint) {
        std::cout << ' ';
    }
    
    if (is_print) {
        std::cout << c;
    } else if (c == '\n') {
        std::cout << " \\n";
    } else if (c == '\r') {
        std::cout << " \\r";
    } else if (c == '\t') {
        std::cout << " \\t";
    } else {
        std::cout << ' ' << std::hex << std::setw(2) << std::setfill('0') 
                  << static_cast<int>(c);
    }
    
    last_was_nonprint = !is_print;
    std::cout << std::flush;
}

int main() {
    unsigned char buf;
    while (read(STDIN_FILENO, &buf, 1) > 0) {
        print_byte(buf);
    }
    return 0;
}

It's far from perfect but the usage of C++ standard library is fairly idiomatic. I did not write this, this was all AI-written.

Now running it by itself from the shell won't really make it very useful because the default behavior is to buffer a line and send it to the program on newlines.

Terminal Capture

Let's talk about the centerpiece of this mini-project which I call terminal capture. The specific functionality I'm working toward is to liberate my workflow a bit from tmux in a terminal. One of the motivating factors is the probably-ill-conceived notion of wanting to be able to review the content I have in all my terminals from all my devices and in particular my phone, which I have with me even when I leave my computers behind.

For 15 years I have put up with mobile terminal emulator apps. There's nothing wrong with them, but it seems clear by now that the keyboard centric interfaces natural for terminal and command line computing are very ill suited to the virtual keyboards on tablets and phones, which also eat up precious screen real estate. I won't be able to work around having lots of buttons and probably a virtual keyboard when entering stuff, but the concept of having a gesture driven terminal emulator seems lost on the (still fairly deep) niche of terminal emulator mobile apps.

So this new concept of mine is that I can keep tmux and friends around and I don't have to immediately change anything about what I'm doing, but I want to surgically insert a new layer of abstraction into the stack so that I can take better control over how I am able to work. One of the big motivators is to be able to collect data and metadata across all the terminal sessions I launch and make that content (including terminal command output!) browsable from one single place. Taking that one step further, then, I could collect all of these streams across all of my computers and eliminate the time it would take to try to remember where I did something. All I would need to look anything up will be the timeframe of it or something to look up the content with.

Also fully AI-written, take a look at this beauty:

#include <iostream>
#include <fstream>
#include <csignal>
#include <cstdlib>
#include <cstring>
#include <unistd.h>
#include <fcntl.h>
#include <termios.h>
#include <sys/ioctl.h>
#include <sys/select.h>
#include <sys/types.h>
#include <sys/wait.h>

static volatile bool should_exit = false;
static struct termios orig_termios;
static int masterFd = -1;
static pid_t child_pid = -1;

// Restore parent terminal to original settings
void restore_terminal() {
  tcsetattr(STDIN_FILENO, TCSANOW, &orig_termios);
}

void cleanup_and_exit() {
  restore_terminal();
  if (masterFd >= 0) {
    close(masterFd);
  }
  if (child_pid > 0) {
    kill(child_pid, SIGTERM);
    waitpid(child_pid, nullptr, 0);
  }
  std::cerr << "\nTerminal capture completed. Logs have been saved.\n";
  exit(0);
}

// Handle Ctrl+C, SIGTERM, etc.
void signal_handler(int sig) {
  should_exit = true;
  if (sig == SIGCHLD) {
    int status;
    pid_t pid = waitpid(-1, &status, WNOHANG);
    if (pid == child_pid) {
      cleanup_and_exit();
    }
  }
}

// Propagate window size changes to the child PTY
void handle_winch(int) {
  struct winsize ws;
  if (ioctl(STDIN_FILENO, TIOCGWINSZ, &ws) == 0 && masterFd >= 0) {
    ioctl(masterFd, TIOCSWINSZ, &ws);
  }
}

int main(int argc, char* argv[]) {
  if (argc != 2) {
    std::cerr << "Terminal Capture - Records all terminal input and output to separate log files\n\n"
              << "Usage: " << argv[0] << " <prefix>\n"
              << "  <prefix>  Prefix for the log files. Will create <prefix>.input and <prefix>.output\n";
    return 1;
  }
  std::string log_path = argv[1];

  // 1) Open master PTY
  masterFd = posix_openpt(O_RDWR | O_NOCTTY);
  if (masterFd < 0) {
    std::cerr << "Error: posix_openpt failed.\n";
    return 1;
  }
  grantpt(masterFd);
  unlockpt(masterFd);

  // 2) Get the slave PTY name
  char* slaveName = ptsname(masterFd);
  if (!slaveName) {
    std::cerr << "Error: ptsname failed.\n";
    close(masterFd);
    return 1;
  }

  // 3) Fork to create child
  child_pid = fork();
  if (child_pid < 0) {
    std::cerr << "Error: fork failed.\n";
    close(masterFd);
    return 1;
  }

  if (child_pid == 0) {
    // Child: set up slave side
    setsid(); // new session
    int slaveFd = open(slaveName, O_RDWR);
    if (slaveFd < 0) {
      std::cerr << "Child: failed to open slave PTY.\n";
      _exit(1);
    }
    ioctl(slaveFd, TIOCSCTTY, 0);

    // Duplicate slaveFd to stdin, stdout, stderr
    dup2(slaveFd, STDIN_FILENO);
    dup2(slaveFd, STDOUT_FILENO);
    dup2(slaveFd, STDERR_FILENO);
    close(slaveFd);
    close(masterFd);

    // Optionally set TERM for interactive programs
    setenv("TERM", "xterm-256color", 1);

    // Exec a shell
    execlp("zsh", "zsh", (char*)nullptr);
    _exit(1); // Exec failed
  }

  // Parent: open separate log files for input and output
  std::string input_path = log_path + ".input";
  std::string output_path = log_path + ".output";
  
  std::ofstream inputFile(input_path, std::ios::app | std::ios::binary);
  std::ofstream outputFile(output_path, std::ios::app | std::ios::binary);
  
  if (!inputFile.is_open() || !outputFile.is_open()) {
    std::cerr << "Failed to open log files\n";
    return 1;
  }

  // Put parent terminal in raw mode so keys flow properly
  struct termios raw;
  tcgetattr(STDIN_FILENO, &orig_termios);
  raw = orig_termios;
  cfmakeraw(&raw);
  tcsetattr(STDIN_FILENO, TCSANOW, &raw);

  // Restore terminal on exit
  atexit(restore_terminal);

  // Handle signals
  signal(SIGINT, signal_handler);
  signal(SIGTERM, signal_handler);
  signal(SIGQUIT, signal_handler);
  signal(SIGCHLD, signal_handler);
  // Forward window size changes to the child
  signal(SIGWINCH, handle_winch);

  // Initialize child PTY with correct window size
  handle_winch(0);

  std::cerr << "Started capturing shell (PID " << child_pid << ")\n"
            << "Logging input to: " << input_path << "\n"
            << "Logging output to: " << output_path << "\n";

  // 4) Relay data between real terminal and child PTY
  while (!should_exit) {
    fd_set fds;
    FD_ZERO(&fds);
    FD_SET(STDIN_FILENO, &fds);
    FD_SET(masterFd, &fds);

    int maxFd = (masterFd > STDIN_FILENO) ? masterFd : STDIN_FILENO;
    int ret = select(maxFd + 1, &fds, NULL, NULL, NULL);
    if (ret < 0 && errno != EINTR) {
      break;
    }

    // Data from real terminal -> child
    if (FD_ISSET(STDIN_FILENO, &fds)) {
      char buf[1024];
      ssize_t n = read(STDIN_FILENO, buf, sizeof(buf));
      if (n > 0) {
        write(masterFd, buf, n);
        // Log user input
        inputFile.write(buf, n);
        inputFile.flush();
      }
    }

    // Data from child -> real terminal
    if (FD_ISSET(masterFd, &fds)) {
      char buf[1024];
      ssize_t n = read(masterFd, buf, sizeof(buf));
      if (n > 0) {
        // Print to screen
        write(STDOUT_FILENO, buf, n);
        // Log shell output
        outputFile.write(buf, n);
        outputFile.flush();
      }
    }
  }

  inputFile.close();
  outputFile.close();
  cleanup_and_exit();
  return 0; // Never reached
}

This ended up way easier to build and refine into a fully working state. I will perform a screen capture to illustrate:

There is a good bit of extra noise in here (weird neovim error and reshuffling tmux panes) but you are able to clearly see that the entirety of the terminal input and output are being captured streaming out to a pair of files.

I'm already able to get a perfect reproduction of the terminal output by using tail -f and I am able to get a really good debug experience for the input character stream as well by using the hexflow program.

This means that I am already able to go to the next step and create a web tech prototype to play with workflows, and then turn it soon toward 3d tech to enhance usability (much like the same is done in terminal emulators these days with GPU acceleration). I will need to scale to a GPU accelerated implementation sooner rather than later given the tons of data I fully intend on shoveling through these systems.